Point inspection method and apparatus, and device and computer-readable storage medium

Abstract

Disclosed in the present application are a point inspection method and apparatus, and a device and a computer-readable storage medium. The point inspection method comprises: acquiring a target image of a point inspection sheet on a product to be inspected; performing image processing on the target image to obtain a heatmap of key points in the target image; determining the coordinates of the key points on the basis of an association relationship between the heatmap and preset key points; and matching the coordinates of the key points with the target image to obtain predicted key points in the target image.

Description

Inspection method, device, equipment and computer-readable storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 202410147722.3, filed on February 1, 2024, entitled “Inspection method, device, equipment and computer-readable storage medium,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to the field of spot inspection technology, and in particular to a spot inspection method, apparatus, device, and computer-readable storage medium. Background Art

[0004] The production and processing of batteries includes multiple steps. Many steps in the pre-production process of batteries require product size inspection. For example, processes such as coating, slitting, and die-cutting may require dimensional inspection of product-related dimensions such as pole piece width, pole piece length, pole tab width and height, pole tab spacing, inner and outer dimensional offset, defect area, defect width and height, etc.

[0005] An image acquisition system is required for dimensional inspections. To ensure accuracy, the image acquisition system's accuracy must be tested. This typically involves placing an inspection sheet on the product under test and capturing an image of it. The locations of the sheet's key points in the image are then predicted, and the sheet's dimensions are calculated based on their locations. The calculated dimensions are then compared with the sheet's actual dimensions to determine whether the image acquisition system meets the required accuracy.

[0006] However, the accuracy of key point position prediction in current related technologies is low.

[0007] Summary of the Invention

[0008] The embodiments of the present application provide a spot inspection method, apparatus, device, and computer-readable storage medium, which can improve the accuracy of the predicted positions of key points.

[0009] In a first aspect, an embodiment of the present application provides a spot inspection method, which includes: obtaining a target image of a spot inspection sheet on a product to be tested; performing image processing on the target image to obtain a heat map of key points in the target image; determining the coordinates of the key points based on the correlation between the heat map and pre-set key points; matching the coordinates of the key points with the target image to obtain predicted key points in the target image.

[0010] Based on the technical solution of the embodiment of this application, the correlation between key points is taken into account. When predicting the key points in the target image of the battery inspection film, the coordinates of the key points are determined based on the heat map of the key points in the target image and the correlation between the key points. This is equivalent to using the correlation between the key points to correct the coordinates of the key points, and then matching the coordinates of the key points with the target image to obtain the predicted key points in the target image. This improves the accuracy of the predicted key point positions.

[0011] According to the implementation method of the first aspect of the present application, the coordinates of the key points are determined based on the association relationship between the heat map and the pre-set key points, including: inputting the heat map into a pre-trained key point regression model to obtain the coordinates of the key points output by the key point regression model, and the key point regression model is obtained by learning the association relationship between the key points in the sample image of the inspection film.

[0012] According to any of the aforementioned embodiments of the first aspect of the present application, the association relationship between the key points in the sample image includes the relative position relationship between the horizontal coordinates of multiple key points in the sample image and / or the relative position relationship between the vertical coordinates of multiple key points in the sample image.

[0013] According to any of the aforementioned embodiments of the first aspect of the present application, the association relationship between the key points in the sample image includes that the difference between the horizontal coordinate of the i-th key point in the sample image and the horizontal coordinate of the j-th key point in the sample image is less than or equal to the first difference, and / or, the difference between the vertical coordinate of the i-th key point and the vertical coordinate of the j-th key point is less than or equal to the second difference, i≠j, i and j are both positive integers.

[0014] According to any of the aforementioned embodiments of the first aspect of the present application, before inputting the heat map into a pre-trained key point regression model to obtain the coordinates of the key points output by the key point regression model, the method also includes: setting a first error function of the key point regression model; obtaining the heat map corresponding to the key points in the sample image, the actual positions of the key points in the sample image and the correlation relationship between the key points in the sample image; inputting the heat map corresponding to the key points in the sample image into the key point regression model to obtain the predicted positions of the key points in the sample image; calculating the first error between the predicted positions of the key points in the sample image and the actual positions of the key points in the sample image based on the first error function; and obtaining a trained key point regression model when the first error is less than or equal to a first preset threshold and the predicted positions of the key points in the sample image satisfy the correlation relationship between the key points in the sample image.

[0015] According to any of the aforementioned embodiments of the first aspect of the present application, image processing is performed on the target image to obtain a heat map of key points in the target image, including: inputting the target image into a pre-trained heat map output model, and obtaining a heat map of key points in the target image output by the heat map output model.

[0016] According to any of the aforementioned embodiments of the first aspect of the present application, before the target image is input into a pre-trained heat map output model to obtain the heat map of the key points in the target image output by the heat map output model, the method also includes: setting a second error function of the heat map output model; obtaining a sample image of the inspection piece and a target heat map corresponding to the key points in the sample image; extracting the target features of the sample image; inputting the target features of the sample image into the heat map output model to obtain a predicted heat map of the key points in the sample image; calculating the second error between the predicted heat map of the key points in the sample image and the target heat map corresponding to the key points in the sample image based on the second error function; and obtaining a trained heat map output model when the second error is less than or equal to a second preset threshold.

[0017] According to any of the aforementioned implementations of the first aspect of the present application, extracting target features from a sample image includes: converting the sample image into a grayscale image; and extracting target features from the grayscale image.

[0018] According to any of the aforementioned embodiments of the first aspect of the present application, extracting target features from a grayscale image includes: extracting edge features of the grayscale image from the grayscale image based on an edge detection operator, where the target features include at least edge features.

[0019] According to any of the aforementioned embodiments of the first aspect of the present application, the heat map of the key points in the target image is obtained based on a pre-trained heat map output model, and the coordinates of the key points are obtained based on a pre-trained key point regression model.

[0020] According to any of the aforementioned embodiments of the first aspect of the present application, the key point regression model is obtained by connecting a preset number of fully connected layers after the output layer of the heat map output model.

[0021] According to any of the aforementioned embodiments of the first aspect of the present application, before obtaining the target image of the inspection piece on the product to be tested, the method also includes: freezing the heat map output model, training the key point regression model, and obtaining a trained key point regression model; freezing the key point regression model, training the heat map output model, and obtaining a trained heat map output model.

[0022] According to any of the aforementioned embodiments of the first aspect of the present application, the sample image includes a first sample image and a second sample image, and the second sample image is obtained by position adjustment and / or shearing of the first sample image, and the position adjustment includes rotation and / or movement.

[0023] In the second aspect, an embodiment of the present application provides a spot inspection device, which includes: an acquisition module for acquiring a target image of a spot inspection sheet on a product to be tested; a heat map output module for performing image processing on the target image to obtain a heat map of key points in the target image; a key point determination module for determining the coordinates of key points based on the correlation between the heat map and pre-set key points; and a key point matching module for matching the coordinates of key points with the target image to obtain predicted key points in the target image.

[0024] In a third aspect, an embodiment of the present application provides an electronic device, which includes: a processor, a memory, and a computer program stored in the memory and runnable on the processor. When the computer program is executed by the processor, the steps of the inspection method provided in the first aspect are implemented.

[0025] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the inspection method provided in the first aspect are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The features, advantages and technical effects of exemplary embodiments of the present application will be described below with reference to the accompanying drawings.

[0027] FIG1 is a flow chart of a spot inspection method provided in an embodiment of the present application;

[0028] FIG2 schematically shows a plurality of key points in a target image of a spot inspection film;

[0029] FIG3 schematically illustrates a plurality of key points in a sample image;

[0030] FIG4 is another schematic flow chart of the spot inspection method provided in an embodiment of the present application;

[0031] FIG5 is a schematic diagram of another flow chart of the inspection method provided in an embodiment of the present application;

[0032] FIG6 is a schematic diagram of a flow chart of S403 in the spot inspection method provided in an embodiment of the present application;

[0033] FIG7 is a schematic diagram of another flow chart of the inspection method provided in an embodiment of the present application;

[0034] FIG8 is a schematic structural diagram of a spot inspection device provided in an embodiment of the present application;

[0035] FIG9 shows a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0036] In the accompanying drawings, the drawings are not necessarily drawn to scale. DETAILED DESCRIPTION

[0037] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present application by illustrating the examples of the present application.

[0038] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, the elements defined by the phrase "comprising..." do not exclude the presence of other identical elements in the process, method, article, or device comprising the elements.

[0039] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0040] It will be apparent to those skilled in the art that various modifications and variations can be made in this application without departing from the spirit or scope of this application. Therefore, this application is intended to cover modifications and variations of this application that fall within the scope of the corresponding claims (technical solutions claimed for protection) and their equivalents. It should be noted that the embodiments provided in the examples of this application can be combined with each other without contradiction.

[0041] Before describing the technical solutions provided by the embodiments of the present application, in order to facilitate understanding of the embodiments of the present application, the present application first specifically describes the problems existing in the related art:

[0042] The production and processing of batteries includes multiple steps. Many steps in the pre-production process of batteries require product size inspection. For example, processes such as coating, slitting, and die-cutting may require dimensional inspection of product-related dimensions such as pole piece width, pole piece length, pole tab width and height, pole tab spacing, inner and outer dimensional offset, defect area, defect width and height, etc.

[0043] An image acquisition system is required for dimensional inspections. To ensure accuracy, the image acquisition system's accuracy must be tested. This typically involves placing an inspection sheet on the product under test and capturing an image of it. The locations of the sheet's key points in the image are then predicted, and the sheet's dimensions are calculated based on their locations. The calculated dimensions are then compared with the sheet's actual dimensions to determine whether the image acquisition system meets the required accuracy.

[0044] However, when predicting the position of key points, related technologies only predict the position of each key point separately, without fully considering the correlation between key points. For example, some key points may be located in the same row or column, or there may be other relative position relationships between some key points, etc., resulting in low accuracy and poor robustness of the predicted key point positions.

[0045] In view of the above research findings of the inventors, the embodiments of the present application provide a spot inspection method, apparatus, device and computer-readable storage medium, which can solve the technical problem of low accuracy of the predicted position of key points existing in the related art.

[0046] The technical concept of the embodiments of the present application is to consider the correlation between key points. When predicting key points in the target image of the inspection film on the product to be tested, the coordinates of the key points are determined based on the heat map of the key points in the target image and the correlation between the key points. This is equivalent to using the correlation between the key points to calibrate the coordinates of the key points. The coordinates of the key points are then matched with the target image to obtain the predicted key points in the target image. This improves the accuracy of the predicted key point positions.

[0047] The following first introduces the inspection method provided in the embodiment of the present application.

[0048] Figure 1 is a flow chart of a spot inspection method provided by an embodiment of the present application. As shown in Figure 1 , the spot inspection method provided by an embodiment of the present application may include the following steps S101 to S104.

[0049] S101 , obtaining a target image of a spot inspection sheet on a product to be tested.

[0050] This application does not limit the type of product to be tested. For example, in some examples, the product to be tested may include a battery electrode or roller. The inspection sheet includes, but is not limited to, film. The inspection sheet can be attached to the product to be tested.

[0051] The target image of the inspection sheet on the product to be tested can be acquired, for example, by an image acquisition system, which is not limited in this application. For example, the image acquisition system can include a camera, such as a line scan camera.

[0052] S102: Perform image processing on the target image to obtain a heat map of key points in the target image.

[0053] In S102, the target image may be processed to generate heat maps for each key point in the target image. For example, if the target image has K key points, K heat maps may be generated, each corresponding to each of the K key points, where K is a positive integer. The jth heat map may include the distribution probability of the jth key point, with a greater distribution probability indicating a closer proximity to the jth key point. 1 ≤ j ≤ K.

[0054] By obtaining the heat map of the key points in the target image, the influence of some interference information (such as noise) in the target image can be effectively reduced, and the useful information related to the key points can be retained, which is conducive to speeding up the prediction rate of key points and improving the accuracy of the prediction of key points.

[0055] Figure 2 schematically shows a plurality of key points in the target image of the inspection sheet. The black dot in Figure 2 represents the key point p1. For the sake of simplicity, Figure 2 only marks some of the key points p1. As shown in Figure 2, in some embodiments, the inspection sheet 20 may include a plurality of polygonal patterns 210. Figure 2 shows four polygonal patterns 210 as an example. In other embodiments, the inspection sheet 20 may also include other numbers of polygonal patterns 210. The shapes of the multiple polygonal patterns 210 may be the same or different. This application does not limit the number and shape of the polygonal patterns 210. For example, in some examples, the polygonal pattern 210 may be a rectangle, a rounded rectangle, a rectangle with a single corner cut off, or other shapes.

[0056] The key point p1 can be the intersection of two adjacent sides of the polygonal pattern 210 (such as the key point p11 shown in FIG2 ) and / or the center point of the polygonal pattern 210 (such as the key point p12 shown in FIG2 ). For example, FIG2 shows 14 key points p1 as an example. That is, in some examples, the intersection of two adjacent sides of the polygonal pattern 210 in the inspection film 20 and / or the center point of the polygonal pattern 210 can be used as the key point p1.

[0057] In some embodiments, multiple sample images of the inspection film can be obtained, and the position of the key point can be marked in each sample image. The heat map output model is trained based on the multiple sample images of the inspection film with the key point marked to obtain a trained heat map output model. After the target image is input into the trained heat map output model, the trained heat map output model can identify K key points in the target image, such as identifying the key points located at the intersection of two adjacent sides of the polygonal pattern 210 and / or the center of the polygonal pattern 210, and generate K heat maps corresponding to each of the K key points.

[0058] S103: Determine the coordinates of the key points based on the association relationship between the heat map and the pre-set key points.

[0059] For example, the association relationship between key points may include the relative position relationship between two or more key points. If the coordinates of the key points are obtained directly based on the heat map, there may be some deviations. For example, in practice, the horizontal coordinate of the x1th key point should be the same as the horizontal coordinate of the x2th key point, but there may be some deviations between the horizontal coordinates of the x1th key point and the x2th key point obtained based on the heat map. Among them, x1 and x2 are positive integers. Therefore, combined with the association relationship between key points, the coordinates of the key points can be corrected so that the predicted key points meet the pre-set association relationship between key points, thereby improving the accuracy of the predicted position of the key points.

[0060] S104: Match the coordinates of the key points with the target image to obtain the predicted key points in the target image.

[0061] It is understood that an image coordinate system is established for the target image, and the coordinates of each point (e.g., pixel) in the target image are known. In S104, the coordinates of the key points determined in S103 can be matched with the coordinates of multiple points in the target image. Points in the target image with the same coordinates as the key points determined in S103 are key points, thereby obtaining predicted key points in the target image.

[0062] The inspection method provided in the embodiments of the present application takes into account the correlation between key points. When predicting key points in the target image of the inspection film on the product to be tested, the coordinates of the key points are determined based on the heat map of the key points in the target image and the correlation between the key points. This is equivalent to using the correlation between the key points to calibrate the coordinates of the key points, and then matching the coordinates of the key points with the target image to obtain the predicted key points in the target image. This improves the accuracy of the predicted key point positions and the robustness of the network, which in turn helps improve the accuracy of precision detection of the image acquisition system.

[0063] According to some embodiments of the present application, optionally, S103, determining the coordinates of the key points based on the association relationship between the heat map and the pre-set key points, may include the following steps:

[0064] The heat map is input into a pre-trained key point regression model to obtain the coordinates of the key points output by the key point regression model. The key point regression model is obtained by learning the association relationship between the key points in the sample images of the inspection piece.

[0065] Before S103, a key point regression model can be established and trained. The key point regression model can be obtained by learning the correlation between key points in the sample image of the inspection film. The key point regression model can be used to determine the coordinates of the key points based on the distribution probability of the key points in the heat map. In some examples, the key point regression model can, for example, use the coordinates corresponding to the maximum distribution probability in the heat map as the coordinates of the key points. In other examples, the key point regression model can also, for example, select the coordinates corresponding to the top N distribution probabilities in the heat map in the order of the distribution probabilities in the heat map from large to small to calculate and obtain the coordinates of the key points. This application does not limit this.

[0066] When training the key point regression model, it can be determined whether the predicted key points of the sample image are consistent with the correlation relationship between the key points in the sample image. If the predicted key points of the sample image do not conform to the correlation relationship between the key points in the sample image, the key point regression model can continue to be trained until the predicted key points of the sample image conform to the correlation relationship between the key points in the sample image. In some examples, the correlation relationship between the key points in the target image can be the same as the correlation relationship between the key points in at least one sample image. Therefore, when the key point heat map in the target image is input into the key point regression model, the coordinates of the key points finally output by the key point regression model conform to the correlation relationship between the key points in the target image. In this way, it is equivalent to correcting the coordinates of the key points by utilizing the correlation relationship between the key points. In this way, the accuracy of the predicted position of the key points is improved.

[0067] This application does not limit the type of key point regression model. For example, in some examples, a key point regression model can be established based on a linear regression algorithm.

[0068] In some embodiments, when the target image has K key points, K heat maps can be sequentially input into a pre-trained key point regression model to obtain the coordinates of the K key points, where K is a positive integer.

[0069] According to some embodiments of the present application, optionally, the association relationship between key points in the sample image may include the relative position relationship between the horizontal coordinates of multiple key points in the sample image and / or the relative position relationship between the vertical coordinates of multiple key points in the sample image.

[0070] FIG3 schematically illustrates a plurality of key points in a sample image. The black dots in FIG3 represent key points p2. As shown in FIG3 , the coordinates of the plurality of key points p2 in the sample image of the inspection sheet 20 are known. The coordinates of the key point p2 may include the horizontal coordinate of the key point p2 and the vertical coordinate of the key point p2. Therefore, the relative positional relationship between the horizontal coordinates of the plurality of key points p2 in the sample image and / or the relative positional relationship between the vertical coordinates of the plurality of key points p2 in the sample image are also known. For example, the horizontal coordinates or vertical coordinates of a portion of the key points p2 are the same, for example, the horizontal coordinates or vertical coordinates of any two key points p2 differ by a preset difference.

[0071] As shown in FIG2 and FIG3 , since the spot inspection piece usually does not change, the association relationship between the key points p1 in the target image of the spot inspection piece and the association relationship between the key points p2 in at least one sample image of the spot inspection piece can be the same.

[0072] In some specific examples, optionally, the association relationship between the key points in the sample image may include that the difference between the horizontal coordinate of the i-th key point in the sample image and the horizontal coordinate of the j-th key point in the sample image is less than or equal to a first difference, and / or, the difference between the vertical coordinate of the i-th key point and the vertical coordinate of the j-th key point is less than or equal to a second difference, i≠j, i and j are both positive integers. The i-th key point and the j-th key point are any two different key points in the sample image. The first difference and the second difference can be flexibly adjusted according to actual conditions, and this application does not limit this. The first difference and the second difference can be greater than or equal to 0.

[0073] FIG4 is another flow diagram of the spot inspection method provided in an embodiment of the present application. As shown in FIG4 , according to some embodiments of the present application, optionally, before step S103, inputting the heat map into a pre-trained keypoint regression model to obtain the coordinates of the keypoints output by the keypoint regression model, the spot inspection method provided in an embodiment of the present application may further include the following steps S301 to S305.

[0074] S301: Set the first error function of the key point regression model.

[0075] In some embodiments, for example, the expression of the first error function is as follows:

[0076] in, represents the first error function, y represents the actual position (i.e., actual coordinate) of the key point in the sample image, and y i represents the actual position (i.e. actual coordinates) of the i-th key point in the sample image, represents the predicted position (i.e., predicted coordinates) of the key points in the sample image, represents the predicted position (i.e., predicted coordinates) of the i-th key point in the sample image, n represents the number of key points in the sample image, 1≤i≤n.

[0077] S302: Obtain a heat map corresponding to key points in the sample image, actual positions of the key points in the sample image, and correlation relationships between the key points in the sample image.

[0078] S303: Input the heat map corresponding to the key points in the sample image into the key point regression model to obtain the predicted positions of the key points in the sample image.

[0079] S304 : Calculate a first error between the predicted position of the key point in the sample image and the actual position of the key point in the sample image based on the first error function.

[0080] S305: Obtain a trained keypoint regression model when the first error is less than or equal to a first preset threshold and the predicted positions of the keypoints in the sample image satisfy the correlation relationship between the keypoints in the sample image. The first preset threshold can be flexibly adjusted according to actual circumstances and is not limited in this application.

[0081] In this way, for example, through the above steps S301 to S305, a key point regression model for learning the association relationship between key points can be obtained.

[0082] According to some embodiments of the present application, optionally, S102, performing image processing on the target image to obtain a heat map of key points in the target image, may include the following steps:

[0083] The target image is input into a pre-trained heat map output model to obtain the heat map of the key points in the target image output by the heat map output model.

[0084] Prior to S102, a heatmap output model may be established and trained. The heatmap output model may output a heatmap of key points in the target image based on the target image. When the target image has K key points, the heatmap output model may output K heatmaps, where the K heatmaps correspond one-to-one to the K key points, where K is a positive integer.

[0085] The heatmap output model can first preliminarily predict the coordinates of each keypoint in the target image. Then, based on the predicted coordinates of each keypoint in the target image, it generates a heatmap for each keypoint. The jth heatmap can include the distribution probability of the jth keypoint. A larger distribution probability indicates a closer proximity to the jth keypoint. 1≤j≤K.

[0086] In this way, by pre-training the heat map output model, the heat map of the key points in the target image can be directly output based on the heat map output model, thereby improving the rate of predicting key points.

[0087] The following describes the training process of the heat map output model.

[0088] FIG5 is another flow chart of the spot inspection method provided in an embodiment of the present application. As shown in FIG5, according to some embodiments of the present application, optionally, before step S102, inputting the target image into a pre-trained heat map output model to obtain a heat map of key points in the target image output by the heat map output model, the spot inspection method provided in an embodiment of the present application may further include the following steps S401 to S406.

[0089] S401. Set a second error function of the heat map output model.

[0090] In some embodiments, for example, the expression of the second error function is as follows:

[0091] in, represents the second error function, y′ represents the target heat map corresponding to the key points in the sample image, and y′ i represents the target heat map corresponding to the i-th key point in the sample image, Represents the predicted heatmap of key points in the sample image, Represents the predicted heat map of the i-th key point in the sample image, N represents the number of key points in the sample image, 1≤i≤N. In some examples, in order to reduce the amount of calculation, such as y′ i The coordinates corresponding to the maximum distribution probability in the target heat map corresponding to the i-th key point in the sample image can be used instead. The coordinates corresponding to the maximum distribution probability in the predicted heat map of the i-th key point in the sample image can be used instead.

[0092] S402: Obtain a sample image of the inspection film and a target heat map corresponding to key points in the sample image.

[0093] The target heat map corresponding to the key points in the sample image may be a heat map generated based on the actual coordinates of the key points in the sample image, that is, an actual heat map.

[0094] S403: Extract target features of the sample image.

[0095] To reduce computational complexity, target features can be extracted from the sample image. For example, in some examples, the target features can include at least edge features of the sample image. Of course, the target features can also include other features in the sample image, which is not limited in this application.

[0096] S404: Input the target features of the sample image into the heat map output model to obtain a predicted heat map of key points in the sample image.

[0097] In this way, inputting the target features of the sample image into the heat map output model can reduce the computational complexity of the heat map output model and improve the computational speed of the heat map output model.

[0098] S405 . Calculate, based on a second error function, a second error between a predicted heat map of key points in the sample image and a target heat map corresponding to the key points in the sample image.

[0099] S406: When the second error is less than or equal to a second preset threshold, a trained heat map output model is obtained. The second preset threshold can be flexibly adjusted according to actual conditions, and this application does not limit this.

[0100] In this way, for example, through the above steps S401 to S406, a trained heat map output model can be obtained, and subsequently based on the heat map output model, the heat map of the key points in the target image can be directly output, thereby improving the rate of predicting key points.

[0101] Figure 6 is a flow chart of S403 in the spot inspection method provided in an embodiment of the present application. As shown in Figure 6, according to some embodiments of the present application, optionally, S403, extracting target features of the sample image, may include the following steps S501 and S502.

[0102] S501: Convert the sample image into a grayscale image.

[0103] S502: Extract target features from the grayscale image.

[0104] In this way, by first converting the sample image into a grayscale image and then extracting the target features from the grayscale image, the amount of data of the sample image can be reduced, thereby reducing the amount of computation required to extract the target features.

[0105] It is understandable that for the target image, the target image can also be first converted into a grayscale image, and then the target features can be extracted from the grayscale image of the target image. The target features of the grayscale image of the target image are input into a pre-trained heat map output model to reduce the amount of computation. The target features of the grayscale image of the target image can at least include edge features of the grayscale image of the target image. Of course, the target features of the grayscale image of the target image can also include other features of the grayscale image of the target image, and this application does not limit this.

[0106] According to some embodiments of the present application, optionally, S502, extracting target features from the grayscale image, may include the following steps:

[0107] Based on an edge detection operator, edge features of the grayscale image of the sample image are extracted from the grayscale image of the sample image, wherein the target features at least include the edge features of the grayscale image of the sample image.

[0108] In this way, by extracting the edge features of the grayscale image of the sample image and inputting them into the heat map output model, the recognition ability of the heat map output model can be improved.

[0109] As mentioned above, according to some embodiments of the present application, optionally, the heat map of the key points in the target image can be obtained based on a pre-trained heat map output model, and the coordinates of the key points can be obtained based on a pre-trained key point regression model.

[0110] In this way, the heat map output model can directly output the heat map of the key points in the target image, and the key point regression model can directly output the coordinates of the key points, thereby improving the speed of predicting key points.

[0111] According to some embodiments of the present application, the key point regression model can optionally be obtained by connecting a preset number of fully connected layers after the output layer of the heat map output model. That is, based on the heat map output model, a preset number of fully connected layers can be connected as a key point regression model to perform key point regression and predict the coordinates of the key points, thereby simplifying the network structure.

[0112] FIG7 is another flow chart of the spot inspection method provided in an embodiment of the present application. As shown in FIG7 , according to some embodiments of the present application, optionally, before S101, obtaining the target image of the spot inspection sheet on the product to be tested, the spot inspection method provided in an embodiment of the present application may further include the following steps S601 and S602.

[0113] S601: Freeze the heat map output model and train the key point regression model to obtain a trained key point regression model.

[0114] When training the key point regression model, the heat map output model can be frozen first, that is, the heat map output model is not trained until a trained key point regression model is obtained.

[0115] S602: Freeze the key point regression model and train the heat map output model to obtain a trained heat map output model.

[0116] When training the heat map output model, the key point regression model can be frozen first, that is, the key point regression model is not trained until a trained heat map output model is obtained.

[0117] The training process of the heat map output model and key point regression model has been described in detail above and will not be repeated here.

[0118] It should be noted that the execution order of S601 and S602 can be swapped, and this application does not limit this.

[0119] In this way, when training the key point regression model, the heat map output model is frozen; when training the heat map output model, the key point regression model is frozen, which can reduce the amount of calculation during model training and increase the speed of model training.

[0120] According to some embodiments of the present application, optionally, in order to improve the accuracy of the heat map output model and the key point regression model prediction, the number of sample images can be multiple. Moreover, the coordinates of the key points in at least some of the sample images can be different.

[0121] In some specific examples, the sample images may optionally include a first sample image and a second sample image. The second sample image may be obtained by position adjustment and / or cropping the first sample image, where the position adjustment includes rotation and / or translation. The number of first sample images may be flexibly adjusted based on actual circumstances, and the number of first sample images may be multiple.

[0122] In this way, by adjusting the position and / or cropping the first sample image to obtain the second sample image, the number of sample images can be increased, thereby facilitating improving the accuracy of the heat map output model and the key point regression model prediction.

[0123] Based on the spot inspection method provided in the above embodiment, the present application also provides a specific implementation of the spot inspection device. Please refer to the following embodiment.

[0124] FIG8 is a schematic diagram of a structure of a spot inspection device provided in an embodiment of the present application. As shown in FIG8 , the spot inspection device 70 provided in an embodiment of the present application may include the following modules:

[0125] An acquisition module 701 is used to acquire a target image of a spot inspection sheet on a product to be tested;

[0126] The heat map output module 702 is used to perform image processing on the target image to obtain a heat map of key points in the target image;

[0127] A key point determination module 703 is used to determine the coordinates of the key points based on the association relationship between the heat map and the pre-set key points;

[0128] The key point matching module 704 is used to match the coordinates of the key points obtained from the heat map with the target image to obtain the predicted key points in the target image.

[0129] The inspection device provided in the embodiment of the present application takes into account the correlation between key points. When predicting key points in the target image of the inspection film on the product to be tested, the coordinates of the key points are jointly determined based on the heat map of the key points in the target image and the correlation between the key points. This is equivalent to using the correlation between the key points to correct the coordinates of the key points, and then matching the coordinates of the key points with the target image to obtain the predicted key points in the target image. This improves the accuracy of the predicted key point positions and the robustness of the network, which in turn helps to improve the accuracy of the precision detection of the image acquisition system.

[0130] According to some embodiments of the present application, optionally, the key point determination module 703 is specifically used to input the heat map into a pre-trained key point regression model to obtain the coordinates of the key points output by the key point regression model, and the key point regression model is obtained by learning the association relationship between the key points in the sample image of the inspection piece.

[0131] According to some embodiments of the present application, optionally, the association relationship between key points in the sample image includes the relative position relationship between the horizontal coordinates of multiple key points in the sample image and / or the relative position relationship between the vertical coordinates of multiple key points in the sample image.

[0132] According to some embodiments of the present application, optionally, the association relationship between the key points in the sample image includes that the difference between the horizontal coordinate of the i-th key point in the sample image and the horizontal coordinate of the j-th key point in the sample image is less than or equal to a first difference, and / or, the difference between the vertical coordinate of the i-th key point and the vertical coordinate of the j-th key point is less than or equal to a second difference, i≠j, i and j are both positive integers.

[0133] The inspection device 70 provided in the embodiment of the present application may include a first training module for setting a first error function of a key point regression model; obtaining a heat map corresponding to the key points in the sample image, the actual positions of the key points in the sample image, and the correlation between the key points in the sample image; inputting the heat map corresponding to the key points in the sample image into the key point regression model to obtain the predicted positions of the key points in the sample image; calculating the first error between the predicted positions of the key points in the sample image and the actual positions of the key points in the sample image based on the first error function; and obtaining a trained key point regression model when the first error is less than or equal to a first preset threshold and the predicted positions of the key points in the sample image satisfy the correlation between the key points in the sample image.

[0134] According to some embodiments of the present application, optionally, the heat map output module 702 is specifically used to input the target image into a pre-trained heat map output model to obtain a heat map of key points in the target image output by the heat map output model.

[0135] According to some embodiments of the present application, optionally, the inspection device 70 provided in the embodiments of the present application may include a second training module for setting a second error function of the thermal map output model; obtaining a sample image of the inspection piece and a target thermal map corresponding to the key points in the sample image; extracting the target features of the sample image; inputting the target features of the sample image into the thermal map output model to obtain a predicted thermal map of the key points in the sample image; calculating the second error between the predicted thermal map of the key points in the sample image and the target thermal map corresponding to the key points in the sample image based on the second error function; and obtaining a trained thermal map output model when the second error is less than or equal to a second preset threshold.

[0136] According to some embodiments of the present application, optionally, the inspection device 70 provided in the embodiments of the present application may include a second training module specifically used to convert the sample image into a grayscale image; and extract target features from the grayscale image.

[0137] According to some embodiments of the present application, optionally, the inspection device 70 provided in the embodiments of the present application may include a second training module specifically used to extract edge features of the grayscale image from the grayscale image based on an edge detection operator, and the target features at least include edge features.

[0138] According to some embodiments of the present application, optionally, the heat map of the key points in the target image is obtained based on a pre-trained heat map output model, and the coordinates of the key points are obtained based on a pre-trained key point regression model.

[0139] According to some embodiments of the present application, optionally, the key point regression model is obtained by connecting a preset number of fully connected layers after the output layer of the heat map output model.

[0140] According to some embodiments of the present application, optionally, the inspection device 70 provided in the embodiments of the present application may include a freezing module for freezing the heat map output model, training the key point regression model, and obtaining a trained key point regression model; freezing the key point regression model, training the heat map output model, and obtaining a trained heat map output model.

[0141] According to some embodiments of the present application, optionally, the sample image includes a first sample image and a second sample image, and the second sample image is obtained by position adjustment and / or shearing the first sample image, and the position adjustment includes rotation and / or movement.

[0142] Each module / unit in the device shown in FIG8 has the function of implementing each step in the inspection method provided in the above method embodiment and can achieve its corresponding technical effect. For the sake of brevity, they will not be described here in detail.

[0143] Based on the inspection method provided in the above embodiment, the present application also provides a specific implementation of the electronic device. Please refer to the following embodiment.

[0144] FIG9 shows a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0145] The electronic device 800 may include a processor 801 and a memory 802 storing computer program instructions.

[0146] Specifically, the processor 801 may include a central processing unit (CPU) or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0147] The memory 802 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 802 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In one example, the memory 802 may include removable or non-removable (or fixed) media, or the memory 802 may be a non-volatile solid-state memory. The memory 802 may be internal or external to the electronic device.

[0148] In one example, the memory 802 may be a read-only memory (ROM). In one example, the ROM may be a mask-programmable ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0149] The memory 802 may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present application.

[0150] The processor 801 implements the method / steps in the above-mentioned method embodiment by reading and executing the computer program instructions stored in the memory 802, and achieves the corresponding technical effects achieved by the method embodiment executing its method / steps. For the sake of brevity, it will not be repeated here.

[0151] In one example, the electronic device 800 may further include a communication interface 803 and a bus 810. As shown in FIG9, the processor 801, the memory 802, and the communication interface 803 are connected via the bus 810 and communicate with each other.

[0152] The communication interface 803 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0153] Bus 810 includes hardware, software or both, couples the parts of electronic equipment to each other.For example, but not limitation, bus may include Accelerated Graphics Port (AGP) or other graphics buses, Enhanced Industry Standard Architecture (EISA) bus, Front Side Bus (FSB), Hyper Transport (HT) interconnection, Industry Standard Architecture (ISA) bus, InfiniBand interconnection, Low Pin Count (LPC) bus, memory bus, Micro Channel Architecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, PCI-Express (PCI-X) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 810 may include one or more buses. Although the present application embodiment describes and shows specific bus, the application considers any suitable bus or interconnection.

[0154] In addition, in conjunction with the spot inspection method in the above embodiment, the embodiment of the present application may provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any of the spot inspection methods in the above embodiment is implemented. Examples of computer-readable storage media include non-transitory computer-readable storage media, such as electronic circuits, semiconductor memory devices, ROM, random access memory, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, and hard disks.

[0155] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0156] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier wave. "Machine-readable medium" can include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0157] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps. In other words, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0158] Aspects of the present application have been described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed via the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. This processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit. It is also understood that each box in the block diagram and / or the flowchart and the combination of the boxes in the block diagram and / or the flowchart can also be implemented by the dedicated hardware that performs the specified function or action, or can be implemented by the combination of dedicated hardware and computer instructions.

[0159] The above description is only a specific embodiment of the present application. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the scope of protection of the present application.

Claims

1. A spot inspection method, comprising: Acquire the target image of the inspection sheet on the product to be tested; Performing image processing on the target image to obtain a heat map of key points in the target image; Determining the coordinates of the key points based on the correlation between the heat map and the pre-set key points; The coordinates of the key points are matched with the target image to obtain predicted key points in the target image.

2. The method according to claim 1, wherein The determining the coordinates of the key points based on the association relationship between the heat map and the preset key points includes: The heat map is input into a pre-trained key point regression model to obtain the coordinates of the key points output by the key point regression model, where the key point regression model is obtained by learning the association relationship between the key points in the sample image of the inspection piece.

3. The method according to claim 2, wherein: The association relationship between the key points in the sample image includes the relative position relationship between the horizontal coordinates of the multiple key points in the sample image.

4. The method according to claim 2, wherein: The association relationship between the key points in the sample image includes the relative position relationship between the vertical coordinates of the multiple key points in the sample image.

5. The method according to claim 2, wherein: The association relationship between the key points in the sample image includes the relative position relationship between the horizontal coordinates of the multiple key points in the sample image and the relative position relationship between the vertical coordinates of the multiple key points in the sample image.

6. The method according to claim 2, wherein: The association relationship between the key points in the sample image includes that the difference between the horizontal coordinate of the i-th key point in the sample image and the horizontal coordinate of the j-th key point in the sample image is less than or equal to the first difference, i≠j, i and j are both positive integers.

7. The method according to claim 2, wherein: The association relationship between the key points in the sample image includes that the difference between the longitudinal coordinate of the i-th key point and the longitudinal coordinate of the j-th key point is less than or equal to the second difference, i≠j, i and j are both positive integers.

8. The method according to claim 2, wherein: The association relationship between the key points in the sample image includes that the difference between the horizontal coordinate of the i-th key point in the sample image and the horizontal coordinate of the j-th key point in the sample image is less than or equal to a first difference, and the difference between the vertical coordinate of the i-th key point and the vertical coordinate of the j-th key point is less than or equal to a second difference, i≠j, i and j are both positive integers.

9. The method according to any one of claims 1 to 8, wherein Before inputting the heat map into a pre-trained key point regression model to obtain the coordinates of the key points output by the key point regression model, the method further includes: Setting a first error function of the keypoint regression model; Obtaining a heat map corresponding to the key points in the sample image, the actual positions of the key points in the sample image, and the correlation relationship between the key points in the sample image; Inputting the heat map corresponding to the key points in the sample image into the key point regression model to obtain the predicted positions of the key points in the sample image; calculating a first error between a predicted position of a key point in the sample image and an actual position of the key point in the sample image based on the first error function; When the first error is less than or equal to a first preset threshold and the predicted positions of the key points in the sample image satisfy the association relationship between the key points in the sample image, the trained key point regression model is obtained.

10. The method according to any one of claims 1 to 8, wherein The performing image processing on the target image to obtain a heat map of key points in the target image includes: The target image is input into a pre-trained heat map output model to obtain a heat map of key points in the target image output by the heat map output model.

11. The method according to claim 10, wherein: Before inputting the target image into a pre-trained heat map output model to obtain a heat map of key points in the target image output by the heat map output model, the method further includes: Setting a second error function of the heat map output model; Obtaining a sample image of the inspection film and a target heat map corresponding to key points in the sample image; extracting target features of the sample image; Inputting the target features of the sample image into the heat map output model to obtain a predicted heat map of key points in the sample image; Calculating a second error between a predicted heat map of the key points in the sample image and a target heat map corresponding to the key points in the sample image based on the second error function; When the second error is less than or equal to a second preset threshold, the trained heat map output model is obtained.

12. The method according to claim 11, wherein The extracting target features of the sample image comprises: Converting the sample image into a grayscale image; The target feature is extracted from the grayscale image.

13. The method according to claim 12, wherein: The step of extracting the target feature from the grayscale image includes: Based on the edge detection operator, the edge features of the grayscale image are extracted from the grayscale image. The target features at least include the edge features.

14. The method according to claim 1, wherein The heat map of the key points in the target image is obtained based on a pre-trained heat map output model, and the coordinates of the key points are obtained based on a pre-trained key point regression model.

15. The method according to claim 14, wherein The key point regression model is obtained by connecting a preset number of fully connected layers after the output layer of the heat map output model.

16. The method according to claim 14 or 15, wherein: Before obtaining the target image of the inspection sheet on the product to be tested, the method further includes: Freezing the heat map output model, and training the key point regression model to obtain the trained key point regression model; Freeze the key point regression model, train the heat map output model, and obtain the trained heat map output model.

17. The method according to any one of claims 2 to 13, wherein The sample images include a first sample image and a second sample image. The second sample image is obtained by adjusting the position and / or cutting the first sample image. The position adjustment includes rotation and / or movement.

18. A spot inspection device comprising: An acquisition module is used to acquire a target image of the inspection sheet on the product to be tested; A heat map output module is used to perform image processing on the target image to obtain a heat map of key points in the target image; A key point determination module, configured to determine the coordinates of key points based on the association between the heat map and pre-set key points; The key point matching module is used to match the coordinates of the key points obtained from the heat map with the target image to obtain the predicted key points in the target image.

19. An electronic device, comprising: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the inspection method according to any one of claims 1 to 17 are implemented.

20. A computer-readable storage medium storing a computer program, wherein the computer-readable storage medium implements the steps of the inspection method according to any one of claims 1 to 17 when the computer program is executed by a processor.

Images