Image processing, image recognition network training and image recognition method and device

Abstract

The present invention relates to a method and device for image processing, image recognition network training and image recognition. The image processing method comprises: obtaining multiple original images in an original image set, wherein at least one original image has a marked area; splicing at least one first image according to the multiple original images; adjusting the shape and / or size of the first image according to the original image to form a second image; and converting the position of the marked area in the second image according to the position of the marked area in the original image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and device for image processing, image recognition network training and image recognition. Background Art

[0002] With the development of artificial intelligence technology, neural networks can perform image recognition, realize automatic classification, reduce labor costs, and improve efficiency and accuracy. On the production line in the workshop, products at all stages are photographed and input into the image recognition network. Through the recognition of the image, it is determined whether the corresponding product has defects and the specific type of defects. The image recognition network is trained by a large number of annotated images. Once the image acquisition equipment on the production line changes, it is necessary to collect annotated images again to train the image recognition network. Summary of the invention

[0003] The present invention provides a method and device for image processing, image recognition network training and image recognition to solve the deficiencies in the related art.

[0004] According to a first aspect of an embodiment of the present invention, there is provided an image processing method, comprising:

[0005] Acquire multiple original images in the original image set, wherein at least one original image has a marked area;

[0006] splicing at least one first image according to the multiple original images;

[0007] Adjusting the shape and / or size of the first image according to the original image to form a second image;

[0008] The position of the marked area in the second image is obtained by converting the position of the marked area in the original image.

[0009] In one embodiment, the original image set includes a plurality of process subsets, the process subsets include a plurality of type nodes, the type nodes include a plurality of original images, and the annotated areas of the original images in the type nodes are annotated with labels corresponding to the types;

[0010] The obtaining of multiple original images in the original image set includes:

[0011] Multiple original images are obtained from the same type of nodes in the same process subset, multiple original images are obtained from different types of nodes in the same process subset, or multiple original images are obtained from the same type of nodes in different process subsets.

[0012] In one embodiment, the original image in a type node of the process subset is an unlabeled image.

[0013] In one embodiment, acquiring multiple original images in the original image set includes:

[0014] Get N 2 original images, wherein N is an integer greater than or equal to 2.

[0015] In one embodiment, the step of stitching at least one first image according to the plurality of original images includes:

[0016] In response to the spaces corresponding to the multiple original images not overlapping, arranging the multiple original images into N rows and N columns to form a first image;

[0017] In response to the overlap of spaces corresponding to the multiple original images, overlapping areas of the multiple original images are superimposed to form a first image.

[0018] In one embodiment, adjusting the shape and / or size of the first image according to the original image to form a second image includes:

[0019] Determining a scaling ratio of the annotated area of ​​the original image to the annotated area of ​​the second image according to parameters of an image acquisition device corresponding to the original image and parameters of an image acquisition device corresponding to the second image;

[0020] scaling the first image according to the scaling ratio to form a third image;

[0021] The shape and size of the third image are adjusted to be consistent with the shape and size of the original image to form a second image.

[0022] In one embodiment, the shape and size of the third image are adjusted to be consistent with the shape and size of the original image to form the second image, comprising:

[0023] The third image is filled and / or cropped to form a second image having the same shape and size as the original image.

[0024] In one embodiment, converting the position of the annotated area in the second image according to the position of the annotated area in the original image includes:

[0025] Obtaining coordinates of the marked area in the first image according to coordinate conversion of the marked area in the original image;

[0026] Obtaining coordinates of the marked area in the third image according to coordinate conversion of the marked area in the first image;

[0027] The coordinates of the marked area in the second image are obtained according to the coordinate conversion of the marked area in the third image.

[0028] In one embodiment, the step of converting the coordinates of the marked area in the original image to obtain the coordinates of the marked area in the first image includes:

[0029] In response to the spaces corresponding to the multiple original images not overlapping, obtaining coordinates of the annotated area in a first coordinate system of the first image according to coordinate transformation of the annotated area in the original image, wherein the first coordinate system takes the center of the first image as an origin;

[0030] In response to the overlap of the spaces corresponding to the multiple original images, coordinates of the annotated area in a second coordinate system of the first image are obtained according to coordinate transformation of the annotated area in the original image, wherein the second coordinate system takes the center of the overlapped area as an origin.

[0031] In one embodiment, the step of converting the coordinates of the marked area in the third image to obtain the coordinates of the marked area in the second image includes:

[0032] The coordinates of the marked area in a third coordinate system are obtained according to the coordinate transformation of the marked area in the scaled first coordinate system or the second coordinate system, wherein the third coordinate system takes the upper left corner of the second image as an origin.

[0033] According to a second aspect of an embodiment of the present invention, a method for training an image recognition network is provided, wherein an image training set is used to train the image recognition network, wherein the images in the image training set are processed using the processing method described in the first aspect.

[0034] According to a third aspect of an embodiment of the present invention, there is provided an image recognition method for recognizing an image using an image recognition network, wherein the image recognition network is trained using the training method described in the second aspect.

[0035] According to a fourth aspect of an embodiment of the present invention, there is provided an image processing apparatus, including:

[0036] An acquisition module, used for acquiring a plurality of original images in an original image set, wherein at least one original image has a marked area;

[0037] An image module, configured to stitch at least one first image according to the plurality of original images;

[0038] an adjustment module, configured to adjust the shape and / or size of the first image according to the original image to form a second image;

[0039] A position module is used to convert the position of the marked area in the second image according to the position of the marked area in the original image.

[0040] In one embodiment, the original image set includes a plurality of process subsets, the process subsets include a plurality of type nodes, the type nodes include a plurality of original images, and the annotated areas of the original images in the type nodes are annotated with labels corresponding to the types;

[0041] The acquisition module is specifically used for:

[0042] Multiple original images are obtained from the same type of nodes in the same process subset, multiple original images are obtained from different types of nodes in the same process subset, or multiple original images are obtained from the same type of nodes in different process subsets.

[0043] In one embodiment, the original image in a type node of the process subset is an unlabeled image.

[0044] In one embodiment, the acquisition module is specifically used for:

[0045] Get N 2 original images, wherein N is an integer greater than or equal to 2.

[0046] In one embodiment, the image module is specifically used for:

[0047] In response to the spaces corresponding to the multiple original images not overlapping, arranging the multiple original images into N rows and N columns to form a first image;

[0048] In response to the overlap of spaces corresponding to the multiple original images, overlapping areas of the multiple original images are superimposed to form a first image.

[0049] In one embodiment, the adjustment module is specifically used to:

[0050] Determining a scaling ratio of the annotated area of ​​the original image to the annotated area of ​​the second image according to parameters of an image acquisition device corresponding to the original image and parameters of an image acquisition device corresponding to the second image;

[0051] scaling the first image according to the scaling ratio to form a third image;

[0052] The shape and size of the third image are adjusted to be consistent with the shape and size of the original image to form a second image.

[0053] In one embodiment, when the adjustment module is used to adjust the shape and size of the third image to a shape and size consistent with the original image to form the second image, it is specifically used to:

[0054] The third image is filled and / or cropped to form a second image having the same shape and size as the original image.

[0055] In one embodiment, the location module is specifically used to:

[0056] Obtaining coordinates of the marked area in the first image according to coordinate conversion of the marked area in the original image;

[0057] Obtaining coordinates of the marked area in the third image according to coordinate conversion of the marked area in the first image;

[0058] The coordinates of the marked area in the second image are obtained according to the coordinate conversion of the marked area in the third image.

[0059] In one embodiment, when the position module is used to obtain the coordinates of the marked area in the first image according to the coordinate conversion of the marked area in the original image, it is specifically used to:

[0060] In response to the spaces corresponding to the multiple original images not overlapping, obtaining coordinates of the annotated area in a first coordinate system of the first image according to coordinate transformation of the annotated area in the original image, wherein the first coordinate system takes the center of the first image as an origin;

[0061] In response to the overlap of the spaces corresponding to the multiple original images, coordinates of the annotated area in a second coordinate system of the first image are obtained according to coordinate transformation of the annotated area in the original image, wherein the second coordinate system takes the center of the overlapped area as an origin.

[0062] In one embodiment, when the position module is used to obtain the coordinates of the marked area in the second image according to the coordinate conversion of the marked area in the third image, it is specifically used to:

[0063] The coordinates of the marked area in a third coordinate system are obtained according to the coordinate transformation of the marked area in the scaled first coordinate system or the second coordinate system, wherein the third coordinate system takes the upper left corner of the second image as an origin.

[0064] According to a fifth aspect of an embodiment of the present invention, there is provided a training device for an image recognition network, comprising:

[0065] A training module is used to train the image recognition network using an image training set, wherein the images in the image training set are processed using the processing method described in the first aspect.

[0066] According to a sixth aspect of an embodiment of the present invention, there is provided an image recognition device, including:

[0067] A recognition module is used to recognize an image using an image recognition network, wherein the image recognition network is trained using the training method described in the second aspect.

[0068] According to a seventh aspect of an embodiment of the present invention, a device is provided, comprising a memory and a processor, wherein the memory is used to store computer instructions executable on the processor, and the processor is used to process an image based on the method described in the first aspect, train an image recognition network based on the method described in the second aspect, or recognize an image based on the method described in the third aspect when executing the computer instructions.

[0069] According to an eighth aspect of an embodiment of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in the first aspect, the second aspect or the third aspect is implemented.

[0070] According to the above embodiments, at least one first image can be spliced ​​through multiple original images in the original image set, and the shape and / or size of the first image can be further adjusted according to the original image to form a second image. The second image not only integrates the content and information contained in the multiple original images, but also matches the shape and / or size of the original image. Furthermore, by determining the position of the annotated area in the second image, the annotated area can be mapped to the second image, so that the second image also has an annotated area, that is, there is a certain area that is labeled, so the second image can also be used for training the image recognition network, that is, when the image acquisition device that collects the original image is replaced with an image acquisition device corresponding to the second image, there is no need to re-capture the image, but the original image set can be directly used to generate a new second graphic set, thereby improving the generation efficiency and quality of the new image set, and further improving the efficiency and quality of network training, image recognition and production line production.

[0071] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0073] Figure 1 is a flow chart of an image processing method shown in an embodiment of the present invention;

[0074] Figure 2is a schematic diagram of the storage form of the original image set shown in an embodiment of the present invention;

[0075] Figure 3 is a schematic diagram of a form of splicing first images shown in an embodiment of the present invention;

[0076] Figure 4 is a schematic diagram of another form of splicing first images shown in an embodiment of the present invention;

[0077] Figure 5 is a flow chart showing a method of determining a second image according to an embodiment of the present invention;

[0078] Figure 6 is a schematic structural diagram of an optical system corresponding to an image acquisition device shown in an embodiment of the present invention;

[0079] Figure 7 is a flow chart of converting the coordinates of the marked area shown in an embodiment of the present invention;

[0080] Figure 8 is a schematic diagram of a second coordinate system shown in an embodiment of the present invention;

[0081] Fig. 9 is a schematic diagram of a third coordinate system shown in an embodiment of the present invention;

[0082] Fig.10 is a schematic structural diagram of an image processing device shown in an embodiment of the present invention;

[0083] Fig.11 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present invention. DETAILED DESCRIPTION

[0084] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0085] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0086] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0087] With the development of artificial intelligence technology, neural networks can perform image recognition, realize automatic classification, reduce labor costs, and improve efficiency and accuracy. On the production line in the workshop, products at all stages are photographed and input into the image recognition network. Through the recognition of the image, it is determined whether the corresponding product has defects and the specific type of defects. The image recognition network is trained by a large number of annotated images. Once the image acquisition equipment on the production line changes, it is necessary to collect annotated images again to train the image recognition network.

[0088] Based on this, in the first aspect, at least one embodiment of the present invention provides an image processing method, please refer to Figure 1 , which shows the process of the processing method, including steps S101 to S104.

[0089] The image targeted by the image processing method can be used to train an image recognition network, and the image processed by the image processing method can also be used to train an image recognition network. The image recognition networks trained by the images before and after the processing can recognize the corresponding images respectively. The image processing method can process a single image (i.e., generate a second image), or batch process the images (i.e., generate multiple second images at the same time); a large number of processed second images constitute an image training set, which can be used to train the image recognition network later.

[0090] In step S101 , a plurality of original images in an original image set are obtained, wherein at least one original image has a marked area.

[0091] Among them, the original image set can be used to train the image recognition network. The trained image recognition network can identify the annotated area in the original image. Similarly, the new image collected by the image acquisition device corresponding to the original image can also be recognized by the image recognition network. That is to say, the image acquisition device collects a large number of images and annotates the annotated area of ​​each image to form an original image. These original images constitute the original image set; the image recognition network is trained using the original image set, and then the new image is collected using the same image acquisition device as the image to be recognized. The image to be recognized can be recognized by the image recognition network. However, once a new image acquisition device is used to collect the image to be recognized, the image recognition network cannot accurately recognize the image to be recognized. Therefore, it is necessary to use the image training set corresponding to the new image to be recognized to retrain the image recognition network. The original image obtained in this step is used to generate a new image training set later.

[0092] Among them, the annotated areas of the original image are marked with different labels, and the labels are matched with the contents of the annotated areas, that is, the corresponding labels are marked according to the contents of the annotated areas. Therefore, the image recognition network trained using the original image set can identify whether the image to be recognized includes the annotated area, and classify the annotated areas according to the labels.

[0093] In one example, the image recognition network can be an Auto Detection Classification (ADC) system based on deep learning applied in the screen production process, which requires a large number of images as training data sets, and the number of training data sets has a great influence on the overall effect of the model; accordingly, the image acquisition device is an Automated Optical Inspection (AOI) device on the production line; accordingly, the annotated area on the image is the defect area in the captured screen image, that is, the unqualified area on the screen, and different defects have corresponding labels; in addition, the annotated area on the image can also be other areas for classification in the captured screen image, such as areas for determining screen categories, batches, and quality, and different categories have corresponding labels, different batches have corresponding labels, and different qualities have corresponding labels. When the equipment on the production line is replaced or the production process is improved, the image captured by the automatic optical inspection device will change greatly compared with the previous image. For example, replacing the optical lens of the AOI device will shorten the focal length of the shooting device, resulting in the defect size in the captured image being reduced exponentially, and the existing ADC system is difficult to detect defects in the newly captured image, which is easy to cause missed detection, thereby affecting the subsequent process flow and affecting the product yield.

[0094] In step S102, at least one first image is obtained according to the multiple original images.

[0095] In this step, the first image is obtained by stitching together multiple original images, so the first image integrates the content and information contained in the multiple original images. For example, the first image integrates at least one marked area contained in the multiple original images.

[0096] In an example, among the multiple original images, only one original image has a marked area, and the marked area is marked with a corresponding label, then the first image also has a marked area, and the marked area is marked with a corresponding label.

[0097] In step S103, the shape and / or size of the first image is adjusted according to the original image to form a second image.

[0098] Among them, the image captured by the new image acquisition device has changed a lot compared with the original image, but it is mainly due to the change of the field of view caused by the change of parameters such as focal length and the change of the proportion of the content such as the annotated area in the image. For example, the annotated area in the original image occupies 20% of the entire image area, and after the field of view becomes larger, the captured space becomes larger, and then the annotated area occupies 10% of the entire image area. Compared with the original image, the image captured by the new image acquisition device can remain unchanged in size and resolution.

[0099] In one example, on a screen production line, equipment replacement and production process improvements can cause the field of view captured by the AOI equipment to become larger, and the proportion of the area of ​​the captured defects to the entire image area to become smaller. When the size and resolution of the image do not change, the size of the defect becomes smaller.

[0100] In step S104, the position of the marked area in the second image is converted according to the position of the marked area in the original image.

[0101] Among them, the label of the annotated area in the original image can be directly mapped to the second image; the position of the annotated area in the original image can be converted in combination with step S102 of splicing the first image and step S103 of determining the second image, so that the position of the annotated area in the second image and the corresponding label are confirmed, thereby making the second image an image that can be directly used to train the image recognition network.

[0102] In an embodiment of the present disclosure, at least one first image can be spliced ​​through multiple original images in the original image set, and the shape and / or size of the first image can be further adjusted according to the original image to form a second image. The second image not only integrates the content and information contained in the multiple original images, but also matches the shape and / or size of the original image. Furthermore, by determining the position of the annotated area in the second image, the annotated area can be mapped to the second image, so that the second image also has an annotated area, that is, there is a certain area that is labeled, so the second image can also be used for training the image recognition network, that is, when the image acquisition device that acquires the original image is replaced with an image acquisition device corresponding to the second image, there is no need to re-acquire the image, but the original image set can be directly used to generate a new second graphic set, thereby improving the generation efficiency and quality of the new image set, and further improving the efficiency and quality of network training, image recognition and production line production.

[0103] For example, when the images captured by the AOI (Automated Optical Inspection) equipment change due to equipment replacement or production process improvement on the screen production line, in order to retrain the ADC system, the data must be collected, labeled, and cleaned again. This process is very time-consuming, usually taking several weeks or even months, and the factory needs to provide a large number of personnel for manual labeling, which is time-consuming and inefficient. Using the image processing method provided in the embodiment of the present disclosure, it is possible to quickly create a new data set with the help of an existing data set, which not only greatly shortens the time for manual image collection, but also allows for rapid training and online launch of new models, reduces the risk of missed detection, and maintains product yield. This method is of great significance for further improving the factory's screen production capacity.

[0104] In some embodiments of the present disclosure, the original images in the original image set are stored in a hierarchical manner, for example, Figure 2 The secondary storage method of the original images in the original image set shown is that the original image set includes multiple process subsets, the process subset includes multiple type nodes, the type node includes multiple original images, and the marked area of ​​the original image in the type node is marked with a label corresponding to the type. For example, each original image stored in the type node "code-2" of the process subset "process 1" is marked with the position coordinates of the marked area and the label corresponding to the node, that is, "process 1, code-2"; for another example, the original images in the original image set can be stored in three levels: process - product model - type, and the present disclosure does not intend to make specific restrictions on the specific number of levels of multi-level storage.

[0105] Each of the process subsets has a type node in which the original image is an unlabeled image. That is, each process subset includes a special type node in which the original image has no labeled area. Therefore, the image is only labeled with the corresponding label, and the position of the labeled area is not marked. Figure 2 The images in Code-8 in process 2 are all unlabeled images, so each image in Code-8 is only labeled with the label "Process 2, Code-8".

[0106] In one example, in the original image set used to train the ADC system on a screen production line, each process subset is used to store images taken at a production site on the production line, and images of different defects taken at each production site are stored in different type nodes, and defect-free images taken at each production site are stored in the special unlabeled type nodes mentioned above.

[0107] Based on the storage method of the original image set described above, multiple original images in the original image set can be obtained by any of the following methods:

[0108] The first method is to obtain multiple original images from the same type of nodes in the same process subset.

[0109] Among them, the original images obtained in this way all contain the same defect, so that the images generated using these multiple original images can have multiple defects of the same kind. Therefore, the generated images can be used to train the image recognition network to identify the defect, and the generated image has multiple defects, which can improve the efficiency and accuracy of training. Since the original images come from the actual data of the production line, this method not only retains the actual data form of the production line, but also meets the new data requirements.

[0110] The second method is to obtain multiple original images from different types of nodes in the same process subset.

[0111] Among them, the original images obtained in this way can contain multiple defects, so that the images generated using these multiple original images can have multiple defects. Therefore, the generated images can be used to train the image recognition network to identify multiple defects, which can improve the efficiency and accuracy of training. Since the original images come from the actual data of the production line, this method not only retains the actual data form of the production line, but also meets the new data requirements.

[0112] In one example, an original image with an annotated area is taken from a type node in a process subset corresponding to the same site, and then multiple original images are taken from another type node in the process subset corresponding to the same site, and none of the multiple original images are annotated. Therefore, the multiple images taken in this example only include one defect.

[0113] The third method is to obtain multiple original images from the same type of type nodes of different process subsets.

[0114] Among them, the original image obtained in this way can contain the same defect of multiple processes, which can greatly improve the anti-interference ability of the image recognition network and reduce the misjudgment rate of network recognition.

[0115] In addition, the above methods for obtaining multiple original images can obtain N 2 original images, wherein N is an integer greater than or equal to 2, so as to facilitate the generation of the first image.

[0116] In the disclosed embodiments, the management accuracy of the original image is increased through the storage method of processes and types, thereby facilitating the acquisition of the original image as required, and improving the distribution accuracy and control accuracy of the multiple images acquired, so that the subsequently generated first image can be accurately controlled, so that the information contained in the generated first image is balanced, and the abnormal concentration of images contained in the first image is avoided, thereby improving the quality of the subsequent training image recognition network.

[0117] In some embodiments of the present disclosure, at least one first image can be spliced ​​based on the multiple original images in the following manner: in response to the fact that the spaces corresponding to the multiple original images have no overlap, the multiple original images are arranged into N rows and N columns to form a first image; in response to the fact that the spaces corresponding to the multiple original images overlap, the overlapping areas of the multiple original images are superimposed to form a first target image.

[0118] Among them, when the spaces corresponding to multiple original images do not overlap, the images can be directly spliced. The splicing method with the same number of rows and columns can make the long side and the wide side of the first image generated after splicing expand in the same proportion compared with the original image, so that the shape of the first image can be guaranteed to be the same as that of the original image, which can simplify the subsequent operation of adjusting the shape and size of the first image. When splicing, the positions of the original images can be arranged and combined to present a variety of situations, that is, multiple first images can be spliced, which can be presented specifically In this case, you can splice For example, by splicing the first image with four original images A, B, C and D, 24 arrangements can be presented, that is, 24 first images can be spliced.

[0119] Among them, when the spaces corresponding to multiple original images overlap, that is, when multiple original images have overlapping parts, the images cannot be directly spliced, because direct splicing will repeatedly display the overlapping areas in the first image, so de-duplication and alignment operations are required, specifically, the overlapping areas of multiple images are superimposed to form the first image. The overlapping area can exist on each of the multiple original images, or on at least two original images. All original images with overlapping areas can be superimposed. For example, four original images A, B, C and D are used to splice the first image, where A and B have overlapping areas, and C and D have overlapping areas. Therefore, it is necessary to superimpose the overlapping areas of A and B, superimpose the overlapping areas of C and D, and then splice the AB combined image and the CD combined image. There are two cases of multiple original images with overlapping areas. The first case is that the overlapping area is at the edge of the image, and the second case is that the overlapping area is inside the image (i.e., the non-edge part). The operation of superimposing images in the two cases is described in detail below.

[0120] In the first case, please refer to the attached Figure 3 , the four original images 301, 302, 303 and 304 are used to stitch the first image 305, wherein the original image 301 and the original image 302 have overlapping areas, the original image 303 and the original image 304 have overlapping areas, and the overlapping areas of the original image 301 and the original image 302 are respectively at the right edge of the original image 301 and the left edge of the original image 302, and the overlapping areas of the original image 303 and the original image 304 are respectively at the right edge of the original image 303 and the left edge of the original image 304, so the overlapping areas of the original image 301 and the original image 302 are superimposed, the overlapping areas of the original image 303 and the original image 304 are superimposed, and the two superimposed images are combined and stitched to form the first image 305, and there is a superimposed area 306 in the middle of the first image. Although the first image stitched by this superposition method is still a neat image, specifically still a rectangle, the aspect ratio of the image is changed, so the shape and size need to be adjusted in the subsequent steps.

[0121] In the second case, please refer to the attached Figure 4, four original images 401, 402, 403 and 404 are used to stitch the first image 405, wherein the original images 401, 402, 403 and 404 have overlapping areas 406, and the overlapping areas are located in the middle of the original images 401, 402, 403 and 404, respectively. Therefore, the overlapping areas 406 of the four original images are superimposed to form the first image. Since the four images will cover each other during the superposition process, they need to be superimposed according to the layer priority of the four images. The layer priority is based on the principle of not covering the marked area. In addition, various situations can be presented in the form of permutations and combinations. The first image spliced ​​by this superposition method is not neat, and the proportional relationship of each side is chaotic, so it is necessary to adjust the shape and size in subsequent steps.

[0122] In the embodiments of the present disclosure, different methods are used to splice the first image for multiple images with or without overlap and multiple images with different overlap situations, thereby ensuring the similarity between the first image and the original image as much as possible, retaining the marked area of ​​the original image, and simplifying the subsequent adjustment steps.

[0123] In some embodiments of the present disclosure, the shape and / or size of the first image may be adjusted according to the original image to form a second image in the following manner. Figure 5 , which shows the process of the above adjustment method, including steps S501 to S503.

[0124] In step S501, a scaling ratio of the annotated area of ​​the original image to the annotated area of ​​the second image is determined according to parameters of an image acquisition device corresponding to the original image and parameters of an image acquisition device corresponding to the second image.

[0125] Among them, the image acquisition device corresponding to the original image can be an image acquisition device used to acquire the original image, and the image acquisition device corresponding to the second image can be an image acquisition device used to acquire the second image, for example, they can be AOI devices before and after equipment replacement or production process improvement.

[0126] The zoom ratio of the marked area is the zoom ratio of the space corresponding to the image. It is more accurate to use the marked area as a ruler to measure the zoom ratio of the space. When calculating the zoom ratio, the imaging process of the image acquisition device can be simplified as follows: Figure 6 The optical system shown in the figure, where l represents the object distance, l' represents the image distance, for the image acquisition device corresponding to the original image, let the object distance be l1, the image distance be l1', and the focal length f be approximately equal to half of the curvature radius r1 of the lens (convex lens), that is, f = r / 2. The above optical system is approximately regarded as an ideal optical system, and the following formula can be obtained according to Gauss's formula:

[0127]

[0128] Therefore, the image acquisition devices corresponding to the original image have the following relationship:

[0129]

[0130] There is a linear relationship between the curvature radius r2 of the image acquisition device corresponding to the second image and the curvature radius r1 of the image acquisition device corresponding to the original image, so it is assumed that:

[0131]

[0132] When replacing imaging equipment in a screen production line, the curvature radius changes due to the equipment, but the object distance does not change, that is, l1 = l1'. Therefore, according to the above four equations, we can get:

[0133]

[0134] According to the vertical axis magnification formula of the optical system, we can get:

[0135]

[0136]

[0137] In the actual imaging process, n = n', so the original product after imaging with the new equipment, the zoom factor k is:

[0138]

[0139] That is, the corresponding spatial range is k times the spatial range corresponding to the original image, that is, the annotated area of ​​the second image is k times the annotated area of ​​the first image. For example, the transformation of the factory's new process and new equipment causes the size of the captured defects to be scaled k times.

[0140] In step S502, the first image is scaled according to the scaling ratio to form a third image.

[0141] In this step, the first image is scaled using the scaling ratio of the annotated area, so that the annotated area in the obtained third image meets the size requirements of the image acquisition device corresponding to the second image (such as the replaced image acquisition device AOI, etc.). For example, the pixels occupied by the annotated area in the original image are 200*200, while the pixels occupied by the annotated area in the second image are 100*100, so the scaling ratio k is 0.5. Therefore, the obtained first image is reduced by half to obtain the third image, and the pixels occupied by the annotated area in the third image are 100*100; however, the size and shape of the third image may not necessarily meet the requirements of the second graphics acquisition device, so further adjustment is required.

[0142] In step S503, the shape and size of the third image are adjusted to be consistent with the shape and size of the original image to form a second image.

[0143] In this step, the third image may be filled and / or cropped to form a second image having the same shape and size as the original image. There are corresponding adjustment methods for the several methods of splicing the first image, and the specific adjustment methods are described in detail below.

[0144] In the first method of splicing the first image, there is no overlap between the multiple original images, and the multiple original images are directly arranged in a matrix form with equal rows and columns, and then spliced ​​into the first image, so the shape of the first image is the same as the original image. Therefore, if the reciprocal of the scaling ratio of the above-mentioned marked area is an integer N, then N can be obtained when obtaining the original image. 2 Therefore, the first image is formed by splicing N rows and N columns of original images, and the first image is reduced to 1 / N according to the scaling ratio, and the size and shape of the third image formed are the same as the original image; however, if N rows and N columns are not obtained when the original image is obtained 2 If the size of the third image is larger or smaller than the original image, cropping is performed when the size is larger than the original image, and padding is performed when the size is smaller than the original image, that is, adding pixels with a brightness of 0 or 1 around the third image. Therefore, if the reciprocal of the scaling ratio of the marked area is not an integer, but the nearest integer to the reciprocal of the scaling ratio is N, then N can be obtained when obtaining the original image. 2 After obtaining the third image, if its size is larger than the original image, it is cropped; if its size is smaller than the original image, it is filled with white.

[0145] In the second method of stitching the first image, there are overlapping areas in the multiple original images, and the overlapping areas are located at the edges of the original images, so the first image obtained is as follows Figure 3 As shown, although it is a rectangle, its aspect ratio is different from that of the original image. Therefore, when the length and width of the third image are both larger than those of the original image, cropping can be performed; when the length and width of the third image are both smaller than those of the original image, white-filling can be performed; when the length of the third image is greater than the length of the original image and the width is smaller than the width of the original image, cropping can be performed in the length direction and white-filling can be performed in the width direction; when the width of the third image is greater than the width of the original image and the length is smaller than the length of the original image, cropping can be performed in the length direction and white-filling can be performed in the length direction.

[0146] In the third method of stitching the first image, there are overlapping areas in the multiple original images, and the overlapping areas are located in the middle of the original images, so the first image obtained is as follows Figure 4As shown, the shape is irregular. Therefore, the shape of the third image can be regularized by cropping or filling. When cropping, the innermost edge of each direction of the third image is used as the cropping boundary. For example Figure 4 The third image can be cut into a regular image 407. When filling, the outermost edge in each direction of the third image is used as the filling boundary. After the third image is adjusted to a regular rectangle, the size is adjusted in the second way to make it equal to the size of the original image.

[0147] In some embodiments of the present disclosure, the position of the annotated area in the second image can be obtained by converting the position of the annotated area in the original image in the following manner. Figure 7 , which shows a flowchart of the above method, including steps S701 to S703.

[0148] In step S701, coordinates of the marked area in the first image are obtained according to the coordinate conversion of the marked area in the original image.

[0149] In one example, in response to the fact that the spaces corresponding to the multiple original images do not overlap, the coordinates of the annotated area in a first coordinate system of the first image are obtained according to the coordinate transformation of the annotated area in the original image, wherein the first coordinate system takes the center of the first image as the origin; in response to the fact that the spaces corresponding to the multiple original images overlap, the coordinates of the annotated area in a second coordinate system of the first image are obtained according to the coordinate transformation of the annotated area in the original image, wherein please refer to the attached Figure 8 , the second coordinate system takes the center of the overlapping area as its origin.

[0150] Among them, when there are no overlapping areas in the multiple original images, the coordinates of the marked areas in the multiple original images arranged in an equal row and column matrix can be first unified into the coordinate system of one of the original images, and then the coordinates of the marked areas in the unified multiple original images can be uniformly mapped into the first coordinate system. For example, four original images A, B, C and D are used to splice the first image, and A, B, C and D are arranged in the order of upper left, upper right, lower left and lower right, and the coordinates of the marked areas in the three images B, C and D are unified into the coordinate system of A (the coordinate system of each original image takes its upper left corner as the origin), and the length of each original image is w and the width is h, then the coordinates in A (x A ,y A ) is still (x A ,y A ), then the coordinates (x B ,y B ) is converted to (x B +w,y B ), then the coordinates (x C,y C ) is converted to (x C ,y C +h), then the coordinates in D (x D ,y D ) is converted to (x D +w,y D +h).

[0151] Among them, when there are overlapping areas in multiple original images and the overlapping areas are located at the edge, the coordinates of the marked areas in the superimposed multiple original images can be unified into the coordinate system of one of the original images, and then the coordinates of the marked areas in the unified multiple original images can be uniformly mapped into the first coordinate system. That is, based on the above-mentioned method of converting coordinates of non-overlapping areas, the coordinates of areas B, C, and D are translated respectively with the upper left corner vertex of area A as the coordinate center, and the translated pixel value is the pixel value of the overlapping area.

[0152] In step S702, coordinates of the marked area in the third image are obtained according to the coordinate conversion of the marked area in the first image.

[0153] Among them, the process of scaling the first image to the third image according to the scaling ratio is based on the center of the first image or the center of the overlapping area. Therefore, the first coordinate system and the second coordinate system are also scaled synchronously during the scaling process of the first image, and the coordinate points in the coordinate system are also scaled synchronously. Therefore, it is only necessary to scale the coordinates in the first image according to the scaling ratio to obtain the coordinates in the third image.

[0154] In step S703, coordinates of the marked area in the second image are obtained according to the coordinate conversion of the marked area in the third image.

[0155] In the process of cropping or filling the third image, it is performed in the circumferential direction of the third image, which does not affect the coordinate positions of the scaled first coordinate system and the second coordinate system in the third image. Therefore, in an example, the coordinates of the marked area in the third coordinate system are obtained according to the coordinate conversion of the marked area in the scaled first coordinate system or the second coordinate system. Fig. 9 , the third coordinate system takes the upper left corner of the second image as the origin. That is, after the cropping or white-filling operation is completed, the first coordinate system or the second coordinate system with the center point as the origin is transformed into a coordinate system with the upper left corner as the origin, that is, unified into the same coordinate system as the original image.

[0156] In the embodiment of the present disclosure, the annotated area in the original image is transformed into the second image through coordinate transformation, that is, the second image not only annotates the label of the annotated area, but also annotates the position of the annotated area, further improving the annotation quality of the second image.

[0157] According to a second aspect of an embodiment of the present invention, a method for training an image recognition network is provided, wherein an image training set is used to train the image recognition network, wherein the images in the image training set are processed using the processing method described in the first aspect.

[0158] According to a third aspect of an embodiment of the present invention, there is provided an image recognition method for recognizing an image using an image recognition network, wherein the image recognition network is trained using the training method described in the second aspect.

[0159] Please refer to the attached Fig.10 According to a fourth aspect of an embodiment of the present invention, there is provided an image processing apparatus, comprising:

[0160] An acquisition module 1001 is used to acquire multiple original images in an original image set, wherein at least one original image has a marked area;

[0161] An image module 1002 is configured to stitch at least one first image according to the plurality of original images;

[0162] An adjustment module 1003, configured to adjust the shape and / or size of the first image according to the original image to form a second image;

[0163] The position module 1004 is used to convert the position of the marked area in the second image according to the position of the marked area in the original image.

[0164] In some embodiments of the present disclosure, the original image set includes a plurality of process subsets, the process subsets include a plurality of type nodes, the type nodes include a plurality of original images, and the annotated areas of the original images in the type nodes are annotated with labels corresponding to the types;

[0165] The acquisition module is specifically used for:

[0166] Multiple original images are obtained from the same type of nodes in the same process subset, multiple original images are obtained from different types of nodes in the same process subset, or multiple original images are obtained from the same type of nodes in different process subsets.

[0167] In some embodiments of the present disclosure, the original image in a type node of the process subset is an unlabeled image.

[0168] In some embodiments of the present disclosure, the acquisition module is specifically used to:

[0169] Get N 2 original images, wherein N is an integer greater than or equal to 2.

[0170] In some embodiments of the present disclosure, the image module is specifically used to:

[0171] In response to the spaces corresponding to the multiple original images not overlapping, arranging the multiple original images into N rows and N columns to form a first image;

[0172] In response to the spaces corresponding to the multiple original images overlapping, overlapping areas of the multiple original images are superimposed to form a first target image.

[0173] In some embodiments of the present disclosure, the adjustment module is specifically used to:

[0174] Determining a scaling ratio of the annotated area of ​​the original image to the annotated area of ​​the second image according to parameters of an image acquisition device corresponding to the original image and parameters of an image acquisition device corresponding to the second image;

[0175] scaling the first image according to the scaling ratio to form a third image;

[0176] The shape and size of the third image are adjusted to be consistent with the shape and size of the original image to form a second image.

[0177] In some embodiments of the present disclosure, when the adjustment module is used to adjust the shape and size of the third image to a shape and size consistent with the original image to form the second image, it is specifically used to:

[0178] The third image is filled and / or cropped to form a second image having the same shape and size as the original image.

[0179] In some embodiments of the present disclosure, the location module is specifically used to:

[0180] Obtaining coordinates of the marked area in the first image according to coordinate conversion of the marked area in the original image;

[0181] Obtaining coordinates of the marked area in the third image according to coordinate conversion of the marked area in the first image;

[0182] The coordinates of the marked area in the second image are obtained according to the coordinate conversion of the marked area in the third image.

[0183] In some embodiments of the present disclosure, when the position module is used to obtain the coordinates of the annotated area in the first image according to the coordinate conversion of the annotated area in the original image, it is specifically used to:

[0184] In response to the spaces corresponding to the multiple original images not overlapping, obtaining coordinates of the annotated area in a first coordinate system of the first image according to coordinate transformation of the annotated area in the original image, wherein the first coordinate system takes the center of the first image as an origin;

[0185] In response to the overlap of the spaces corresponding to the multiple original images, coordinates of the annotated area in a second coordinate system of the first image are obtained according to coordinate transformation of the annotated area in the original image, wherein the second coordinate system takes the center of the overlapped area as an origin.

[0186] In some embodiments of the present disclosure, when the position module is used to obtain the coordinates of the annotated area in the second image according to the coordinate conversion of the annotated area in the third image, it is specifically used to:

[0187] The coordinates of the marked area in a third coordinate system are obtained according to the coordinate transformation of the marked area in the scaled first coordinate system or the second coordinate system, wherein the third coordinate system takes the upper left corner of the second image as an origin.

[0188] According to a fifth aspect of an embodiment of the present invention, there is provided a training device for an image recognition network, comprising:

[0189] A training module is used to train the image recognition network using an image training set, wherein the images in the image training set are processed using the processing method described in the first aspect.

[0190] According to a sixth aspect of an embodiment of the present invention, there is provided an image recognition device, including:

[0191] A recognition module is used to recognize an image using an image recognition network, wherein the image recognition network is trained using the training method described in the second aspect.

[0192] Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment of the method in the third aspect, and will not be elaborated here.

[0193] Please refer to the attached Fig.11According to the seventh aspect of an embodiment of the present invention, a device is provided, comprising a memory and a processor, wherein the memory is used to store computer instructions executable on the processor, and the processor is used to process an image based on the method described in the first aspect, train an image recognition network based on the method described in the second aspect, or recognize an image based on the method described in the third aspect when executing the computer instructions.

[0194] According to an eighth aspect of an embodiment of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in the first aspect, the second aspect or the third aspect is implemented.

[0195] In the present invention, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. The term "plurality" refers to two or more than two, unless otherwise clearly defined.

[0196] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the disclosure disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0197] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Acquire multiple original images in the original image set, wherein at least one original image has a marked area; splicing at least one first image according to the multiple original images; Adjusting the shape and / or size of the first image according to the original image to form a second image; Obtaining a position of the annotated area in the second image according to the position of the annotated area in the original image; The step of adjusting the shape and / or size of the first image according to the original image to form a second image comprises: Determining a scaling ratio of the annotated area of ​​the original image to the annotated area of ​​the second image according to parameters of an image acquisition device corresponding to the original image and parameters of an image acquisition device corresponding to the second image; scaling the first image according to the scaling ratio to form a third image; The shape and size of the third image are adjusted to be consistent with the shape and size of the original image to form a second image.

2. The image processing method according to claim 1, characterized in that: The original image set includes a plurality of process subsets, the process subsets include a plurality of type nodes, the type nodes include a plurality of original images, and the annotated areas of the original images in the type nodes are annotated with labels corresponding to the types; The obtaining of multiple original images in the original image set includes: Multiple original images are obtained from the same type of nodes in the same process subset, multiple original images are obtained from different types of nodes in the same process subset, or multiple original images are obtained from the same type of nodes in different process subsets.

3. The image processing method according to claim 2, characterized in that: The original image in a type node of the process subset is an unlabeled image.

4. The image processing method according to any one of claims 1 to 3, characterized in that: The obtaining of multiple original images in the original image set includes: Get N 2 original images, wherein N is an integer greater than or equal to 2.

5. The image processing method according to claim 4, characterized in that: The step of stitching at least one first image according to the plurality of original images comprises: In response to the spaces corresponding to the multiple original images not overlapping, arranging the multiple original images into N rows and N columns to form a first image; In response to the overlap of spaces corresponding to the multiple original images, overlapping areas of the multiple original images are superimposed to form a first image.

6. The image processing method according to claim 5, characterized in that: The shape and size of the third image are adjusted to be consistent with the shape and size of the original image to form a second image, comprising: The third image is filled and / or cropped to form a second image having the same shape and size as the original image.

7. The image processing method according to claim 5, characterized in that: The converting the position of the marked area in the second image according to the position of the marked area in the original image includes: Obtaining coordinates of the marked area in the first image according to coordinate conversion of the marked area in the original image; Obtaining coordinates of the marked area in the third image according to coordinate conversion of the marked area in the first image; The coordinates of the marked area in the second image are obtained according to the coordinate conversion of the marked area in the third image.

8. The image processing method according to claim 7, characterized in that: The step of converting the coordinates of the marked area in the original image to obtain the coordinates of the marked area in the first image includes: In response to the fact that the spaces corresponding to the multiple original images do not overlap, obtaining coordinates of the annotated area in a first coordinate system of the first image according to coordinate transformation of the annotated area in the original image, wherein the first coordinate system takes the center of the first image as an origin; In response to the overlap of the spaces corresponding to the multiple original images, coordinates of the annotated area in a second coordinate system of the first image are obtained according to coordinate transformation of the annotated area in the original image, wherein the second coordinate system takes the center of the overlapped area as an origin.

9. The image processing method according to claim 8, characterized in that: The step of converting the coordinates of the marked area in the second image according to the coordinates of the marked area in the third image comprises: The coordinates of the marked area in a third coordinate system are obtained according to the coordinate transformation of the marked area in the scaled first coordinate system or the second coordinate system, wherein the third coordinate system takes the upper left corner of the second image as an origin.

10. A method for training an image recognition network, characterized in that: The image recognition network is trained using an image training set, wherein the images in the image training set are processed using the processing method according to any one of claims 1 to 9.

11. An image recognition method, characterized in that: An image recognition network is used to recognize an image, wherein the image recognition network is trained using the training method according to claim 10.

12. An image processing device, characterized in that: include: An acquisition module, used for acquiring a plurality of original images in an original image set, wherein at least one original image has a marked area; An image module, configured to stitch at least one first image according to the plurality of original images; an adjustment module, configured to adjust the shape and / or size of the first image according to the original image to form a second image; A position module, used for converting the position of the marked area in the second image according to the position of the marked area in the original image; The adjustment module is used for: Determining a scaling ratio of the annotated area of ​​the original image to the annotated area of ​​the second image according to parameters of an image acquisition device corresponding to the original image and parameters of an image acquisition device corresponding to the second image; scaling the first image according to the scaling ratio to form a third image; The shape and size of the third image are adjusted to be consistent with the shape and size of the original image to form a second image.

13. A training device for an image recognition network, characterized in that: include: A training module, used for training the image recognition network using an image training set, wherein the images in the image training set are processed using the processing method according to any one of claims 1 to 9.

14. An image recognition device, characterized in that: include: A recognition module, used to recognize images using an image recognition network, wherein the image recognition network is trained using the training method according to claim 10.

15. A device, characterized in that The device includes a memory and a processor, wherein the memory is used to store computer instructions that can be executed on the processor, and the processor is used to process an image based on the method described in any one of claims 1 to 9, train an image recognition network based on the method described in claim 10, or recognize an image based on the method described in claim 11 when executing the computer instructions.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images