Method and device for counting number of SMT (Surface Mount Technology) materials

Abstract

The embodiment of the invention provides an SMT material quantity statistical method and device, and the method comprises the steps: obtaining an original image through photographing an SMT material, precisely cutting out an image of a material placement region from the original image, effectively removing background information irrelevant to the material quantity statistics, determining the material size of the SMT material through employing a material region image which does not contain the background information, and improving the statistical accuracy of the SMT material. According to the SMT material quantity statistics method and device, the material size is obtained, the matched material detection model is selected according to the material size, the matched material detection model can accurately output the detection result reflecting the quantity of the materials of the size, and then the quantity of the SMT materials is obtained through statistics according to the detection result, so that the SMT material quantity statistics pertinence is improved, and the SMT material quantity statistics method and device are suitable for SMT material statistics scenes of various sizes; and therefore, the statistical accuracy of the SMT material quantity is improved.

Description

Technical Field

[0001] The present application relates to the technical field of industrial material counting, and in particular to a method and device for counting the number of SMT materials. Background Art

[0002] In modern manufacturing, the accuracy of the quantity statistics of SMT (Surface Mount Technology) materials, such as resistors and integrated circuits, plays a vital role in production cost control, product quality assurance and production efficiency improvement.

[0003] Traditional SMT material quantity counting methods mainly rely on manual operations, such as counting one by one manually. Due to the small size of SMT materials themselves and the large quantity, they are easily affected by factors such as human negligence or fatigue, which leads to deviations in statistical results, that is, low accuracy. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method and device for counting the number of SMT materials to improve the accuracy of counting the number of SMT materials. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for counting the number of SMT materials, the method comprising:

[0006] Get the original image obtained by photographing the SMT material;

[0007] Cropping the original image to obtain images of all SMT material placement areas as material area images;

[0008] Determining the size of the SMT material according to the material area image;

[0009] Based on the detection result of the material area image by the target material detection model, the quantity of the SMT material is obtained by statistics; wherein the target material detection model is a material detection model corresponding to the size of the SMT material.

[0010] In one possible implementation,

[0011] Determining the size of the SMT material according to the material area image includes:

[0012] According to the visual features of the material area image, the material areas where all the SMT materials in the material area image are located are detected; wherein the visual features are used to represent the contours in the material area image and the positions of the contours;

[0013] Obtaining statistical values ​​of the areas of the material regions by counting; wherein the statistical values ​​are used to represent the area of ​​a single SMT material;

[0014] If the area represented by the statistical value is greater than the upper area threshold, determining that the size of the SMT material is a first size;

[0015] If the area represented by the statistical value is not greater than the upper area threshold, determining that the size of the SMT material is a second size;

[0016] The detection result of the material area image based on the target material detection model is used to obtain the quantity of the SMT material by counting, including:

[0017] If the size of the SMT material is the first size, the number of SMT materials is obtained by counting the detection result of the material area image based on a first material detection model, wherein the first material detection model is a material detection model for performing SMT material detection based on visual features;

[0018] If the size of the SMT material is the second size, the number of SMT materials is statistically obtained based on the detection result of the material area image by a second material detection model, wherein the second material detection model is a material detection model for SMT material detection based on semantic features.

[0019] In one possible implementation,

[0020] The first material detection model is a target rotation regression head model, and the second material detection model is a target semantic category regression head model;

[0021] The detecting and obtaining the material regions where all SMT materials in the material region image are located according to the visual features of the material region image includes:

[0022] Inputting the material area image into the target rotation regression head model, so that the target rotation regression head model extracts visual features of the material area image, and outputs the material areas where all SMT materials in the material area image are located based on the visual features;

[0023] The method of obtaining the quantity of the SMT material by counting the detection result of the material area image based on the first material detection model includes:

[0024] Determine the number of the material areas output by the target rotary regression head model as the number of the SMT materials;

[0025] The method further comprises:

[0026] If the size of the SMT material is the second size, input the material area image into the target semantic category regression head model, so that the target semantic category regression head model extracts semantic features of the material area image, and outputs the material center point of each of the SMT materials in the material area image based on the semantic features;

[0027] The method of obtaining the quantity of the SMT material by counting the detection result of the material area image based on the second material detection model includes:

[0028] Determine the number of material center points output by the target semantic category regression head model as the number of the SMT materials;

[0029] Among them, the target rotated rectangular regression head model is obtained by training the first sample image, the material size of the SMT material in the first sample image is greater than the upper limit area threshold, and the first sample image is annotated with the material area where each of the SMT materials in the first sample image is located; the target semantic category regression head model is obtained by training the second sample image, and the second sample image is annotated with the material center point of each of the SMT materials in the second sample image.

[0030] In one possible implementation,

[0031] The target rotated rectangular regression head model is pre-trained in the following manner, including:

[0032] Inputting the first sample image into the original rotated rectangular regression head model, so that the original rotated regression head model extracts visual features of the first sample image, and outputs all predicted regions containing objects in the first sample image and the predicted category and confidence of each predicted region based on the visual features; wherein the predicted category is used to represent the category of the object existing in the predicted region, and the confidence is used to represent the accuracy of the category;

[0033] Calculating a first loss according to a first difference between each predicted area and a material area where each SMT material marked in the first sample image is located; wherein the first loss is positively correlated with the first difference;

[0034] Calculating a second loss according to a second difference between the predicted category and the target category of the predicted area; wherein the target category is a category of an object existing in the area of ​​the first sample image at the same position as the predicted area, if the predicted category is the same as the target category, then the second difference is negatively correlated with the confidence of the predicted area, and if the predicted category is different from the target category, then the second difference is positively correlated with the confidence of the predicted area, and the second loss is positively correlated with the second difference;

[0035] Adjusting the parameters of the original rotated rectangular regression head model in the direction of gradient descent of the first loss and the second loss to obtain a target rotated rectangular regression head model;

[0036] The target semantic category regression head model is pre-trained in the following ways, including:

[0037] Inputting the second sample image into the original semantic category regression head model, so that the target semantic category regression head model extracts the semantic features of the second sample image, and outputs the predicted material center point of each of the SMT materials in the material area image based on the semantic features;

[0038] Calculating a third loss according to a third difference between each predicted material center point and a material center point of each SMT material marked in the second sample image; wherein the third loss is positively correlated with the third difference;

[0039] In the direction of the third loss gradient descent, the parameters of the original semantic category regression head model are adjusted to obtain the target semantic category regression head model.

[0040] In a possible implementation, the method further includes:

[0041] A model that meets a first preset condition is determined from among multiple first preset models as a target rotation regression head model, and a model that meets a second preset condition is determined from among multiple second preset models as a target semantic category regression head model; wherein the multiple first preset models are trained based on the same first sample image, and different first preset models have different structures, and the multiple second preset models are trained based on the same second sample image, and different second preset models have different structures.

[0042] In one possible implementation,

[0043] The step of outputting the material center point of each of the SMT materials in the material region image based on the semantic feature includes:

[0044] Outputting a material center point mask image according to the semantic features of the material area image; wherein the material center point mask is used to represent the material center point of each of the SMT materials in the material area image, the size of the material center point mask image is the same as the material area image, and each element is used to represent the confidence that the pixel at the same position in the material area image is the material center point;

[0045] The step of determining the number of material center points output by the target semantic category regression head model as the number of the SMT materials includes:

[0046] Pooling the material center point mask image to obtain a pooled material center point mask image;

[0047] In the pooled material center point mask image, a maximum value greater than a preset screening threshold is determined, and the number of the obtained maximum values ​​is determined as the number of the SMT material.

[0048] In one possible implementation,

[0049] The first sample image and the second sample image are pre-annotated in the following manner, including:

[0050] Acquire a sample material area image obtained by photographing an SMT sample material; wherein each of the sample material area images is obtained by photographing SMT sample materials of different sizes, and each of the sample material area images includes a plurality of SMT sample materials, and the SMT sample materials of different sizes include a first size material and a second size material, the first size material is an SMT sample material having a size greater than the upper limit area threshold, and the second size material is an SMT sample material having a size not greater than the upper limit area threshold;

[0051] Marking the material center point of each of the SMT sample materials in the sample material area image to obtain an original sample image;

[0052] In the original sample image, determine the original sample image obtained by photographing the material of the first size as the first preprocessed sample image, and mark the material area where the SMT sample material is located in the first preprocessed sample image to obtain the first sample image;

[0053] The original sample image is determined as the second sample image, or other images in the original sample image except the first preprocessed sample image are determined as the second sample image.

[0054] In one possible implementation,

[0055] The step of determining the original sample image as the second sample image, or determining other images in the original sample image except the first preprocessed sample image as the second sample image, includes:

[0056] Determine the original sample image as a second preprocessed sample image, or determine other images in the original sample image except the first preprocessed sample image as a second preprocessed sample image;

[0057] According to the category of the photographed SMT sample material, each of the second pre-processed sample images is uniformly sampled to obtain a sampling result as a second sample image;

[0058] The step of marking the material area where the SMT sample material is located in the first preprocessed sample image to obtain the first sample image includes:

[0059] Marking the material area where the SMT sample material is located in the first preprocessed sample image to obtain a third preprocessed sample image;

[0060] According to the category of the photographed SMT sample material, each of the third pre-processed sample images is uniformly sampled to obtain a sampling result as the first sample image.

[0061] In a possible implementation manner, the cropping of the original image to obtain images of all SMT material placement areas as material area images includes:

[0062] Based on the principle of maximizing the grayscale value variance between the foreground area and the background area after binarization, the original image is subjected to grayscale binarization processing to obtain a preprocessed image; wherein the original image is an X-ray image;

[0063] Determining a grayscale value boundary in the preprocessed image;

[0064] A sub-image within the boundary line is cut out from the pre-processed image as a material area image.

[0065] In one possible implementation,

[0066] The method of obtaining the number of SMT materials by counting the detection result of the material area image based on the second material detection model includes:

[0067] If the area represented by the statistical value is greater than the lower area threshold, the number of SMT materials is obtained by counting the detection result of the second material detection model on the material area image that has not been magnified;

[0068] If the area represented by the statistical value is not greater than the lower area threshold, the number of SMT materials is statistically obtained based on the detection result of the second material detection model on the enlarged material area image.

[0069] In one possible implementation,

[0070] The detection obtains the material regions where all SMT materials in the material region image are located, including:

[0071] Detecting and obtaining the candidate regions where all potential SMT materials in the material region image are located, and the confidence of each candidate region; wherein the confidence is used to indicate the probability that an SMT material exists in the candidate region;

[0072] In the candidate regions, regions whose confidence levels are not lower than a preset confidence threshold are determined as material regions where all SMT materials in the material region image are located.

[0073] In a second aspect, an embodiment of the present application provides a device for counting the number of SMT materials, the device comprising:

[0074] An image acquisition module is used to acquire an original image obtained by photographing the SMT material;

[0075] An image cropping module, used for cropping images of all SMT material placement areas from the original image as material area images;

[0076] A size determination module, used to determine the size of the SMT material according to the material area image;

[0077] A quantity statistics module is used to obtain the quantity of the SMT material based on the detection result of the material area image by a target material detection model; wherein the target material detection model is a material detection model corresponding to the size of the SMT material.

[0078] In a possible implementation, the size determination module includes:

[0079] A first size determination submodule is used to detect the material regions where all SMT materials in the material region image are located according to the visual features of the material region image; wherein the visual features are used to represent the contours in the material region image and the positions of the contours;

[0080] The second size determination submodule is used to obtain a statistical value of the area of ​​each material region; wherein the statistical value is used to represent the area of ​​a single SMT material;

[0081] A third size determination submodule, configured to determine that the size of the SMT material is a first size if the area represented by the statistical value is greater than an upper area threshold;

[0082] a fourth size determination submodule, configured to determine that the size of the SMT material is a second size if the area represented by the statistical value is not greater than the upper area threshold;

[0083] The quantity statistics module includes:

[0084] A first quantity statistics submodule, configured to obtain the quantity of the SMT material based on the detection result of the material area image by a first material detection model if the size of the SMT material is the first size, wherein the first material detection model is a material detection model for performing SMT material detection based on visual features;

[0085] The second quantity statistics submodule is used to obtain the quantity of the SMT material based on the detection result of the material area image by a second material detection model if the size of the SMT material is the second size, wherein the second material detection model is a material detection model for SMT material detection based on semantic features.

[0086] In a possible implementation, the first material detection model is a target rotation regression head model, and the second material detection model is a target semantic category regression head model;

[0087] The first size determination submodule includes:

[0088] a first size determination unit, configured to input the material area image into the target rotation regression head model, so that the target rotation regression head model extracts visual features of the material area image, and outputs material areas where all SMT materials in the material area image are located based on the visual features;

[0089] The first quantity statistics submodule includes:

[0090] A first quantity counting unit, used for determining the quantity of the material areas output by the target rotation regression head model as the quantity of the SMT material;

[0091] The device also includes:

[0092] a center point output module, for inputting the material area image into the target semantic category regression head model if the size of the SMT material is the second size, so that the target semantic category regression head model extracts the semantic features of the material area image, and outputs the material center point of each of the SMT materials in the material area image based on the semantic features;

[0093] The second quantity statistics submodule includes:

[0094] A second quantity counting unit is used to determine the number of material center points output by the target semantic category regression head model as the number of the SMT material;

[0095] Among them, the target rotated rectangular regression head model is obtained by training the first sample image, the material size of the SMT material in the first sample image is greater than the upper limit area threshold, and the first sample image is annotated with the material area where each of the SMT materials in the first sample image is located; the target semantic category regression head model is obtained by training the second sample image, and the second sample image is annotated with the material center point of each of the SMT materials in the second sample image.

[0096] In a possible implementation manner, the target rotated rectangular regression head model is pre-trained in the following manner, including:

[0097] Inputting the first sample image into the original rotated rectangular regression head model, so that the original rotated regression head model extracts visual features of the first sample image, and outputs all predicted regions containing objects in the first sample image and the predicted category and confidence of each predicted region based on the visual features; wherein the predicted category is used to represent the category of the object existing in the predicted region, and the confidence is used to represent the accuracy of the category;

[0098] Calculating a first loss according to a first difference between each predicted area and a material area where each SMT material marked in the first sample image is located; wherein the first loss is positively correlated with the first difference;

[0099] Calculating a second loss according to a second difference between the predicted category and the target category of the predicted area; wherein the target category is a category of an object existing in the area of ​​the first sample image at the same position as the predicted area, if the predicted category is the same as the target category, then the second difference is negatively correlated with the confidence of the predicted area, and if the predicted category is different from the target category, then the second difference is positively correlated with the confidence of the predicted area, and the second loss is positively correlated with the second difference;

[0100] Adjusting the parameters of the original rotated rectangular regression head model in the direction of gradient descent of the first loss and the second loss to obtain a target rotated rectangular regression head model;

[0101] The target semantic category regression head model is pre-trained in the following ways, including:

[0102] Inputting the second sample image into the original semantic category regression head model, so that the target semantic category regression head model extracts the semantic features of the second sample image, and outputs the predicted material center point of each of the SMT materials in the material area image based on the semantic features;

[0103] Calculating a third loss according to a third difference between each predicted material center point and a material center point of each SMT material marked in the second sample image; wherein the third loss is positively correlated with the third difference;

[0104] In the direction of the third loss gradient descent, the parameters of the original semantic category regression head model are adjusted to obtain the target semantic category regression head model.

[0105] In a possible implementation, the device further includes:

[0106] A model selection module is used to determine a model that meets a first preset condition from multiple first preset models as a target rotation regression head model, and to determine a model that meets a second preset condition from multiple second preset models as a target semantic category regression head model; wherein the multiple first preset models are trained based on the same first sample image, and different first preset models have different structures, and the multiple second preset models are trained based on the same second sample image, and different second preset models have different structures.

[0107] In a possible implementation, the central point output module includes:

[0108] A first center point output submodule is used to output a material center point mask image according to the semantic features of the material area image; wherein the material center point mask is used to represent the material center points of each of the SMT materials in the material area image, the size of the material center point mask image is the same as that of the material area image, and each element is used to represent the confidence that the pixel at the same position in the material area image is the material center point;

[0109] The second quantity statistics submodule includes:

[0110] a third quantity counting unit, configured to pool the material center point mask image to obtain a pooled material center point mask image;

[0111] The fourth quantity counting unit is used to determine the maximum value greater than the preset screening threshold in the pooled material center point mask image, and determine the number of the obtained maximum values ​​as the number of the SMT material.

[0112] In a possible implementation manner, the first sample image and the second sample image are pre-annotated in the following manner, including:

[0113] Acquire a sample material area image obtained by photographing an SMT sample material; wherein each of the sample material area images is obtained by photographing SMT sample materials of different sizes, and each of the sample material area images includes a plurality of SMT sample materials, and the SMT sample materials of different sizes include a first size material and a second size material, the first size material is an SMT sample material having a size greater than the upper limit area threshold, and the second size material is an SMT sample material having a size not greater than the upper limit area threshold;

[0114] Marking the material center point of each of the SMT sample materials in the sample material area image to obtain an original sample image;

[0115] In the original sample image, determine the original sample image obtained by photographing the material of the first size as the first preprocessed sample image, and mark the material area where the SMT sample material is located in the first preprocessed sample image to obtain the first sample image;

[0116] The original sample image is determined as the second sample image, or other images in the original sample image except the first preprocessed sample image are determined as the second sample image.

[0117] In one possible implementation,

[0118] The step of determining the original sample image as the second sample image, or determining other images in the original sample image except the first preprocessed sample image as the second sample image, includes:

[0119] Determine the original sample image as a second preprocessed sample image, or determine other images in the original sample image except the first preprocessed sample image as a second preprocessed sample image;

[0120] According to the category of the photographed SMT sample material, each of the second pre-processed sample images is uniformly sampled to obtain a sampling result as a second sample image;

[0121] The step of marking the material area where the SMT sample material is located in the first preprocessed sample image to obtain the first sample image includes:

[0122] Marking the material area where the SMT sample material is located in the first preprocessed sample image to obtain a third preprocessed sample image;

[0123] According to the category of the photographed SMT sample material, each of the third pre-processed sample images is uniformly sampled to obtain a sampling result as the first sample image.

[0124] In a possible implementation, the image cropping module includes:

[0125] A first image cropping submodule is used to perform grayscale binarization processing on the original image based on the principle of maximizing the grayscale value variance between the foreground area and the background area after binarization to obtain a preprocessed image; wherein the original image is an X-ray image;

[0126] A second image cropping submodule, used for determining a grayscale value dividing line in the pre-processed image;

[0127] The third image cropping submodule is used to crop the sub-image within the boundary line from the pre-processed image as the material area image.

[0128] In a possible implementation manner, the second quantity statistics submodule includes:

[0129] a fifth quantity counting unit, configured to obtain the quantity of the SMT material by counting the detection result of the material area image that has not been magnified by the second material detection model if the area represented by the statistical value is greater than the lower area threshold;

[0130] The sixth quantity counting unit is used to obtain the quantity of the SMT material based on the detection result of the amplified material area image by the second material detection model if the area represented by the statistical value is not greater than the lower area threshold.

[0131] In a possible implementation manner, the first size determination submodule includes:

[0132] A second size determination unit is used to detect and obtain the candidate regions where all potential SMT materials are located in the material region image, and the confidence of each candidate region; wherein the confidence is used to indicate the probability that the SMT material exists in the candidate region;

[0133] The third size determination unit is used to determine the area whose confidence is not less than a preset confidence threshold in the candidate area as the material area where all the SMT materials in the material area image are located.

[0134] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0135] Memory, used to store computer programs;

[0136] The processor is used to implement any of the above-mentioned material quantity counting methods when executing the program stored in the memory.

[0137] In a fourth aspect, an embodiment of the present application provides a material counting machine, the material counting machine comprising an image acquisition device, a processor and a display;

[0138] The image acquisition device is used to obtain an original image by photographing the SMT material;

[0139] The processor is used to obtain the original image obtained by the image acquisition device by photographing the SMT material; crop the image of all material placement areas from the original image as the material area image; determine the size of the SMT material according to the material area image; based on the detection result of the material area image by the target material detection model, obtain the number of the SMT materials by statistics, and send the number of the SMT materials to the display; wherein the target material detection model is a material detection model corresponding to the size of the SMT material;

[0140] The display is used to display the quantity of the SMT material.

[0141] In a fifth aspect, an embodiment of the present application provides an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the material quantity counting methods described above.

[0142] Beneficial effects of the embodiments of the present application:

[0143] A material quantity counting method and device provided in an embodiment of the present application obtains an original image by photographing an SMT material, and accurately crops an image of an SMT material placement area from the original image, effectively removing background information irrelevant to the SMT material quantity counting, and using a material area image that does not contain background information to determine the material size of the SMT material, and according to the material size, selects a matching material detection model, which can accurately output a detection result reflecting the number of SMT materials of this size, and then according to the detection result, the number of SMT materials is statistically obtained, thereby improving the pertinence of the SMT material quantity counting, being suitable for SMT material quantity counting scenarios of various sizes, and thereby improving the accuracy of the SMT material quantity counting.

[0144] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0145] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0146] Figure 1 A schematic diagram of a flow chart of a method for counting the number of SMT materials provided in an embodiment of the present application;

[0147] Figure 2 A first schematic diagram of an SMT material provided in an embodiment of the present application;

[0148] Figure 3 A second schematic diagram of the SMT material provided in the embodiment of the present application;

[0149] Figure 4 A schematic diagram of image preprocessing provided in an embodiment of the present application;

[0150] Figure 5 A schematic diagram of sample annotations provided in an embodiment of the present application;

[0151] Figure 6 A schematic diagram of a depth model provided in an embodiment of the present application;

[0152] Figure 7 A schematic diagram of the structure of the hardware system provided in the embodiment of the present application;

[0153] Figure 8 A schematic diagram of the structure of a data acquisition module provided in an embodiment of the present application;

[0154] Fig. 9 A schematic diagram of a process for obtaining data by a data acquisition module provided in an embodiment of the present application;

[0155] Fig.10 A schematic diagram of the structure of the counting model module provided in the embodiment of the present application;

[0156] Fig.11 A schematic diagram of the structure of the SMT material quantity statistics module provided in the embodiment of the present application;

[0157] Fig.12 A schematic diagram of the counting process of the SMT material quantity statistics module provided in the embodiment of the present application;

[0158] Fig.13 A schematic diagram of the structure of the SMT material quantity counting device provided in an embodiment of the present application;

[0159] Fig.14 A schematic diagram of the structure of a material counting machine provided in an embodiment of the present application;

[0160] Fig.15 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0161] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0162] With the rapid development of modern information technology, a wide variety of electronic products have entered the public eye. Among them, circuit boards are key components that control the core functions of electronic products, and their design and manufacturing process is particularly important. The construction of circuit boards involves a large number of diversified materials, such as resistors, inductors, etc. Therefore, in order to improve the manufacturing efficiency of circuit boards, SMT is often used in the production process. In the SMT production line, SMT materials (surface mount technology materials), such as resistors, inductors and other small particle materials, are used very frequently. Therefore, small particle materials such as resistors and inductors need to be frequently shipped out and put into the warehouse, and then the material inventory management is reasonably carried out to accurately count the changes in the quantity of small particle materials such as resistors and inductors during the frequent shipping and warehousing process for easy management. Regular and comprehensive material inventory management can not only grasp the material retention status in real time, prevent the common shortage, loss, leakage or shortage of materials in the supply chain, but also provide decision support for the optimization of the production line, thereby improving production efficiency.

[0163] When managing the inventory of small particles of materials such as resistors and inductors, since resistors, inductors and other materials are often placed in material trays, and since the size of small particles of materials such as resistors and inductors is small, in order to save space, each material tray will hold a large number of materials. Therefore, how to count the number of SMT materials (hereinafter referred to as material quantity) on the material tray (hereinafter referred to as material quantity statistics) is a difficult problem.

[0164] In the related art, material quantity statistics can be achieved by manually counting each component one by one, or by using an image recognition algorithm. A large number of different types of SMT materials with features such as shape and size can be manually extracted in advance, and images of material trays containing components can be taken. The extracted features can be used to identify components from the images of the material trays using image recognition algorithms, and the material quantity can be obtained by counting.

[0165] However, the above methods have the following problems. First, the statistical method of manually counting components one by one has high labor costs, low efficiency, and human errors. Secondly, the above-mentioned method using image recognition algorithm requires manual design of feature extraction. Due to the uncertainty of the shape and size of the material, the discrete distribution, and the problem that the materials are very close to each other and multiple SMT materials are connected together, this type of recognition method has a high missed detection rate, that is, the accuracy is low.

[0166] Based on this, the present application embodiment proposes a SMT material quantity counting method for improving the accuracy of SMT material quantity counting, see Figure 1 , Figure 1 A flow chart of a method for counting the number of SMT materials provided in an embodiment of the present application, the method comprising:

[0167] S101, acquiring an original image obtained by photographing the SMT material.

[0168] S102, cropping the original image to obtain images of all SMT material placement areas as material area images.

[0169] S103, determining the size of the SMT material according to the material area image.

[0170] S104, based on the detection result of the target material detection model on the material area image, the number of SMT materials is obtained by counting.

[0171] The target material detection model is a material detection model corresponding to the size of the SMT material.

[0172] By applying the above embodiment, the original image is obtained by photographing the SMT material, and the image of the SMT material placement area is accurately cropped from the original image, effectively removing background information irrelevant to the SMT material quantity statistics. The material area image that does not contain background information is used to determine the material size of the SMT material, and according to the material size, a matching material detection model is selected. The matching material detection model can accurately output a detection result reflecting the number of SMT materials of this size, and then according to the detection result, the number of SMT materials is statistically obtained, thereby improving the pertinence of the SMT material quantity statistics, being suitable for SMT material statistics scenarios of various sizes, and thereby improving the accuracy of the SMT material quantity statistics.

[0173] The above steps S101 to S104 are described in detail below:

[0174] In step S101 , there are multiple SMT materials, and SMT materials of the same size are placed in the same position.

[0175] The material area image in this article can be obtained by collecting the real material that the user wants to count by X-Ray image acquisition equipment, or it can be collected by visible light image acquisition equipment. Figure 2 , Figure 2 This is a first schematic diagram of an SMT material provided in an embodiment of the present application, wherein 201 is a material area image obtained by photographing the left side of an SMT material roll, and 202 is the SMT material in the material area image. Figure 3 , Figure 3This is a second schematic diagram of the SMT material provided in an embodiment of the present application, wherein 301 is a material area image obtained by photographing the left side of another SMT material roll, and 302 is the SMT material in the material area image. For ease of understanding and display, the figure only shows a relatively sparse arrangement of the SMT materials, while in actual applications the arrangement of the SMT materials will be denser.

[0176] In step S102, in the process of capturing the material area image, the background area other than the SMT material may be inevitably captured. If there are materials identical to the SMT material or objects similar to the SMT material in the background area, then when counting the number, the SMT materials or objects in the background area may be mistakenly counted into the number of materials, resulting in inaccurate material quantity statistics. On the other hand, when the material area image contains too many background images, this not only increases the complexity of image processing, but also reduces the efficiency of material quantity statistics.

[0177] Based on this, in order to improve the accuracy of material quantity counting, reduce the complexity of image processing, and thus improve the efficiency of material quantity counting, images of all material placement areas can be cropped from the original image as material area images, wherein the placement area can be the area where the SMT material reel is located, the area where the SMT material tray is located, or the area at any position where the SMT material is located.

[0178] In a possible embodiment, a preset cropping method can be set according to the working experience of professional technicians to crop the area where the materials are placed. However, the applicability of this method is low, and it may cause the cropped material area image to include the background area image, thereby resulting in a low accuracy rate of material quantity statistics. Therefore, in another possible embodiment, in order to improve the accuracy rate of material quantity statistics, the aforementioned step S102 includes step S1021, step S1022 and step S1023, specifically:

[0179] S1021, based on the principle of maximizing the grayscale value variance between the foreground area and the background area after binarization, grayscale binarization is performed on the original image to obtain a preprocessed image. The original image is an X-ray image.

[0180] Among them, the grayscale threshold can be determined based on the principle of maximizing the grayscale value variance between the foreground area and the background area after binarization, and the grayscale threshold is used to perform grayscale binarization processing on the original image to obtain a preprocessed image. Exemplarily, a function between the grayscale value variance and the grayscale threshold can be constructed to solve the grayscale value when the grayscale value variance is the largest, and the value is used as the grayscale threshold. It can also be based on multiple grayscale values, respectively, the variance of the grayscale value under each grayscale value is calculated until the grayscale value when the grayscale value variance is the largest is obtained, and the value is used as the grayscale threshold.

[0181] S1022, determining a grayscale value boundary in the preprocessed image.

[0182] The boundary line may be a circumscribed rectangular boundary of the area where the SMT material is located, or a boundary of other shapes, which is not specifically limited here.

[0183] S1023, cropping a sub-image within the boundary line from the pre-processed image as a material area image.

[0184] Taking the boundary line as the circumscribed rectangular boundary of the area where the SMT material is located as an example, a sub-image within the circumscribed rectangular boundary is cropped from the pre-processed image as the material area image. The cropped image is the non-material area image.

[0185] See also Figure 4 , Figure 4 A schematic diagram of image preprocessing provided in an embodiment of the present application, wherein taking the material area image as an example of photographing the material contained in the SMT material roll, 401 is the original image obtained by photographing the SMT material roll, and 402 is the material area image obtained after preprocessing.

[0186] By applying the above embodiment, grayscale binarization processing technology is used, and based on the principle of maximizing the grayscale value variance between the foreground area and the background area, the boundary between the material area and the background is made clearer. On this basis, the dividing line of the grayscale value is further determined, and the sub-image within the dividing line is cropped as the material area image, thereby removing redundant background information, reducing noise interference, and reducing the complexity of image processing, which not only improves the efficiency of material quantity counting, but also improves the accuracy of material quantity counting.

[0187] In step S103, the size of the SMT material is used to indicate the specification of the material, that is, whether the SMT material is a large-size material or a small-size material. In a possible embodiment, a professional staff member may set a threshold value based on his / her own experience, and determine the size of the material through the threshold value and the image recognition algorithm, but the accuracy of this method is low.

[0188] In another possible embodiment, the visual features of the material region image may be extracted, and the visual features are used to represent the contour in the material region image and the position of the contour. The size of the SMT material is determined according to the contour in the material region image and the position of the contour. The aforementioned step S103 includes step S1031, step S1032, step S1033 and step S1034. Specifically:

[0189] S1031. Detect and obtain the material regions where all SMT materials in the material region image are located according to the visual features of the material region image.

[0190] S1032. Obtain statistical values ​​of the areas of each material region.

[0191] Among them, the statistical value is used to represent the area of ​​a single SMT material.

[0192] S1033: If the area represented by the statistical value is greater than the upper area threshold, determine that the size of the SMT material is the first size.

[0193] S1034: If the area represented by the statistical value is not greater than the upper area threshold, determine that the size of the SMT material is the second size.

[0194] The visual features of the material region image are macroscopic features of the material, such as features such as contour, color, etc. After the visual features are obtained, the material regions where all the materials in the material region image are located are detected based on the visual features.

[0195] In a possible embodiment, statistical values ​​of each material area may be obtained by statistics, and the statistical values ​​are used to represent the area of ​​a single material. Taking the material area represented by a material frame as an example, the statistical value may be the average value of the areas of all material frames detected, or the median of the areas of all material frames detected, etc.

[0196] In another possible embodiment, the statistical values ​​of the diagonal length, perimeter, side lengths, etc. of each material area can be obtained by statistics. Taking the material area represented by a material frame as an example, the statistical value can be the maximum value of the diagonal length of all material frames detected, or the maximum value of the side lengths of all material frames, etc.

[0197] The upper limit area threshold can be set according to the actual needs or experience of the user. This application does not impose any restrictions on this. It can be understood that if the user needs to control the efficiency within a higher range, a lower upper limit area threshold can be set. If the user needs to control the accuracy within a higher range, a higher upper limit area threshold can be set.

[0198] The first size is used to indicate that the SMT material is a large-size material, and the second size is used to indicate that the SMT material is a small-size material. When the area represented by the statistical value is greater than the upper area threshold, the size of the SMT material is determined to be the first size, that is, the SMT material is a large-size material. When the area represented by the statistical value is not greater than the upper area threshold, the size of the SMT material is determined to be the second size, that is, the SMT material is a small-size material.

[0199] In a possible embodiment, after obtaining the material area image, the material area image can be input into an algorithm model pre-set by the user based on experience, and the algorithm model extracts visual features, and detects the material area of ​​the SMT material based on the visual features. However, the user's experience is often limited, so the algorithm model set based on experience is difficult to apply to various scenarios.

[0200] Based on this, in another possible embodiment, the material area image can be input into the depth model, and the depth model extracts visual features based on the material area image, and outputs the material area where the SMT material is located in the material area image according to the visual features. In this embodiment, the depth model is a target rotation regression head model, and the aforementioned step S1031 may include step S10311, specifically:

[0201] S10311, input the material area image to the target rotation regression head model, so that the target rotation regression head model extracts the visual features of the material area image, and outputs the material areas where all SMT materials in the material area image are located based on the visual features.

[0202] Among them, the material area is represented in the form of a material frame, and the target rotation regression head model is pre-trained. The specific training method will be described below and will not be repeated here. It can be understood that although the material area is represented in the form of a material frame in this embodiment, this does not limit the material area in this application to be represented in the form of a material frame. In other possible embodiments, it can also be represented in other forms other than the material frame, such as a mask or vertex coordinates. For the convenience of description below, only the case where the material area is represented in the form of a material frame is used as an example. For the case where the material area is represented in other forms, the principle is the same and will not be repeated here.

[0203] By applying the above embodiment, the material area image is input into the target rotation regression head model, and the target rotation regression head model extracts visual features from the image, and accurately outputs all material areas where materials exist in the material area image based on these features, effectively reducing the detection error caused by detecting the material area based on the algorithm model set based on experience, improving the accuracy of material area detection, and thus improving the accuracy of material quantity statistics. In addition, the material area is output based on the deep model, without relying on manual labor, realizing the automation of material quantity statistics and reducing labor costs.

[0204] Considering the accuracy of material area detection, some non-material areas may be mistakenly detected as SMT material areas, which may cause the number of material frames to be greater than the actual number of materials. Therefore, if the number of material frames is directly determined as the number of SMT materials, the number of materials obtained by statistics may be too large. Based on this, in a possible embodiment, each area in the material area image is screened, and the screened area is used as the material area. In this embodiment, the aforementioned step S1031 may also specifically include step S10312 and step S10313, specifically:

[0205] S10312, detecting and obtaining the candidate regions where all potential SMT materials in the material region image are located, and the confidence level of each candidate region.

[0206] S10313, determining an area whose confidence level is not lower than a preset confidence threshold in the candidate area as the material area where all the SMT materials in the material area image are located.

[0207] Among them, in order to ensure the independence between material areas and reduce the material quantity statistical errors caused by detection accuracy, before detecting the candidate area, the potential areas where all potential SMT materials in the material area image are located can be detected first, and then the potential areas can be screened. Specifically, the areas that meet the preset non-overlapping conditions can be determined in the potential areas as candidate areas.

[0208] A specific method for determining the region satisfying the preset non-overlapping condition in the candidate regions may be achieved by performing a non-maximum suppression (NMS) operation on the candidate regions. Therefore, the preset non-overlapping condition may be understood as follows: during the NMS operation, the intersection-and-union ratio between any two retained candidate regions is lower than a preset threshold. Similarly, the threshold may be set based on actual needs or experience of the user.

[0209] The specific method for determining the area that meets the preset non-overlapping condition in the candidate area can also be achieved by methods other than the NMS operation, for example, calculating the area of ​​the overlapping area. If the area is greater than a preset area threshold, it is determined that the overlapping condition is met; if the area is not greater than the preset area threshold, it is determined that the overlapping condition is not met.

[0210] The confidence level is used to indicate the probability of the existence of SMT materials in the initial area. The confidence level threshold can be set according to the actual needs or experience of the user, and this application does not limit this. It is understandable that if the user needs to control the missed detection rate within a lower range, a lower confidence level threshold can be set; if the user needs to control the false detection rate within a lower range, a higher confidence level threshold can be set.

[0211] By applying the above embodiment, the candidate areas where all potential materials in the material area image are located are detected, and each candidate area is assigned a confidence level. By setting a preset confidence threshold, areas with confidence levels not lower than the threshold are screened out as the material areas where all materials in the material area image are located. This process effectively eliminates low-probability false detection areas (i.e., areas where no materials exist), improves detection accuracy, reduces material quantity statistics errors caused by detection accuracy, and further improves the accuracy of material quantity statistics.

[0212] In step S104, a material detection model that can accurately detect and obtain the quantity of materials of the size is selected according to the size of the SMT material. Step S104 includes step S1041 and step S1042. Specifically:

[0213] S1041: If the size of the SMT material is the first size, the number of the SMT materials is obtained by counting the detection result of the material area image by the first material detection model.

[0214] Among them, the first material detection model is a material detection model that performs material detection based on visual features.

[0215] S1042: If the size of the SMT material is the second size, the number of the SMT materials is obtained by counting the detection result of the material area image by the second material detection model.

[0216] Among them, the second material detection model is a material detection model that performs material detection based on semantic features.

[0217] Both semantic features and visual features are features directly obtained based on the material area image. When the area of ​​the SMT material is greater than the upper area threshold, it can be considered that the size of the SMT material is the first size, that is, the SMT material is a large-size material. Therefore, the number of SMT materials can be determined based on the macroscopic visual features of the material area image. When the area of ​​the SMT material is not greater than the upper area threshold, it can be considered that the size of the SMT material is the second size, that is, the SMT material is a small-size material. Therefore, the number of SMT materials can be determined based on the semantic features at the pixel level of the material area image.

[0218] The material detection model may be any model that can detect and obtain detection results for characterizing the quantity of materials. In a possible embodiment, the first material detection model is a target rotation regression head model, and the second material detection model is a target semantic category regression head model.

[0219] When the first material detection model is a target rotating regression head model, the aforementioned step S1041 includes step S10411, specifically:

[0220] S10411, determine the number of material areas output by the target rotating regression head model as the number of materials.

[0221] When the second material detection model is the target semantic category regression head model, if the size of the SMT material is the second size, the material area image is input into the target semantic category regression head model, so that the target semantic category regression head model extracts the semantic features of the material area image, and outputs the material center point of each SMT material in the material area image based on the semantic features.

[0222] The aforementioned step S1042 includes step S10421, specifically:

[0223] S10421, determine the number of material center points output by the target semantic category regression head model as the number of SMT materials.

[0224] Applying the above embodiment, when the SMT material is a large-sized material, the large-sized material can be accurately characterized by macroscopic visual features. Therefore, it can be considered that the number of large-sized materials can be accurately detected based on visual features, so the number of materials can be directly determined based on the number of material areas determined by visual features. When the SMT material is a small-sized material, the material area image is input into the target semantic category regression head model, and the semantic features in the material area image are extracted by the target semantic category regression head model, and the material center points of each material are accurately output based on the semantic features. By counting the number of material center points, the total number of materials can be directly obtained. By adopting this method, there is no need to rely on manual labor, and there are different quantity counting methods for SMT materials of different sizes, which not only improves the degree of automation of material quantity statistics, but also improves the accuracy of material quantity statistics.

[0225] In a possible embodiment, the material center point mask is determined according to the semantic features of the material area image, and the number of materials is obtained by the area ratio of the mask area. For example, there is a reference mask area of ​​A, and a mask area area of ​​B is given. Then the number of materials in the mask area can be estimated according to the value of B / A, and the mask area is divided equally, and the centers of each mask area after equal division are taken as the material center. This method relies on the area ratio of the mask area to estimate the number of materials. This area-based estimation method may result in a decrease in accuracy when the SMT material shape and size vary greatly or the SMT materials are densely arranged. In addition, the practice of dividing the mask area equally to obtain the material center may result in inaccurate positioning of the material center when the SMT material distribution is uneven or the SMT material shape is irregular, thereby affecting the accuracy of material quantity statistics.

[0226] Based on this, in another possible embodiment, the aforementioned step outputs the material center point of each SMT material in the material area image based on the semantic feature, specifically including:

[0227] According to the semantic features of the material area image, the material center point mask image is output.

[0228] Among them, the material center point mask is used to represent the material center point of each SMT material in the material area image. The size of the material center point mask image is the same as the material area image, and each element is used to represent the confidence that the pixel at the same position in the material area image is the material center point.

[0229] On this basis, the aforementioned step S10421 specifically includes:

[0230] The material center point mask image is pooled to obtain a pooled material center point mask image.

[0231] In the pooled material center point mask image, the maximum value greater than the preset screening threshold is determined, and the number of the determined maximum values ​​is used as the number of materials.

[0232] Among them, the material center point mask is used to represent the aforementioned material center point, and the material center point mask image is used to represent the material center point image. The maximum value is determined in the pooled material center point mask image, and the maximum value higher than the preset screening threshold is determined. The maximum value higher than the preset screening threshold can be recorded as a valid maximum value point, and the number of valid maximum value points is counted as the number of materials, and the position of each valid maximum value point is the material center point of each material in the material area image. And, it can be understood that in this embodiment, in response to a maximum value being greater than the preset screening threshold, the maximum value is recorded as a valid maximum value point, so the preset screening threshold can be regarded as a response threshold. The preset screening threshold can be set according to the actual needs or experience of the user, and this application does not limit this.

[0233] By applying the above embodiment, the material center point mask image is determined according to the semantic features of the material area image. By performing pooling processing on the material center point mask image, data redundancy is effectively reduced, and noise interference is also effectively reduced. In the pooled material center point mask image, the maximum value is determined by setting a preset screening threshold, and the inaccuracy caused by adhesion is reduced, so that the real material center point is accurately obtained, and then the number of materials is accurately counted. The result is accurate, and the accuracy of material quantity statistics is improved.

[0234] In some scenarios, there may be SMT materials whose area is much smaller than the upper area threshold among the small-sized materials. For these SMT materials with too small sizes, it is difficult to accurately detect the center point of the material directly based on the original material area image. Therefore, in a possible embodiment, a lower area threshold may be pre-set. The aforementioned step S1042 may also include steps S10422 and S10423. Specifically:

[0235] S10422: If the area represented by the statistical value is greater than the lower area threshold, the number of SMT materials is obtained based on the detection result of the second material detection model on the material area image that has not been magnified.

[0236] S10423: If the area represented by the statistical value is not greater than the lower area threshold, the number of SMT materials is obtained based on the detection result of the second material detection model on the enlarged material area image.

[0237] Among them, the lower area threshold can be set according to the actual needs or experience of the user, and this application does not impose any restrictions on this, but the lower area threshold should be much smaller than the upper area threshold.

[0238] Enlarging the material area image may be to upsample and enlarge the material area image, that is, to increase the number of pixels of the image through an interpolation algorithm (such as bilinear interpolation, bicubic interpolation or more advanced algorithms), thereby improving the resolution of the image. In addition, other methods can be used to enlarge the material area image, such as using image enhancement technology to improve the visual effect of the image, so that the SMT material features in the image are clearer and easier to detect. The image enhancement technology can be to adjust the image brightness, contrast, sharpening and other parameters, or to apply filters and other means to reduce noise and interference. In some cases, multiple methods can be combined to achieve the best enlargement effect, which is not specifically limited here.

[0239] By applying the above embodiment, it is determined whether the area represented by the statistical value is greater than the lower area threshold. If the area is greater than the lower area threshold, the number of materials is statistically obtained based on the detection result of the second material detection model on the material area image that has not been magnified, which can save computing resources and improve the speed of material quantity statistics. When the area represented by the statistical value is not greater than the lower area threshold, that is, when the image details are blurred or the materials are densely distributed, the method amplifies the material area image to obtain a clearer and easier to detect material area image, and more accurately counts the material quantity based on the detection result of the second material detection model on the magnified material area image. This method not only improves the accuracy of material quantity statistics, but also enhances the scope of application of the method, so that it can cope with material area images of different sizes and complexities, and effectively improves the accuracy of material quantity statistics.

[0240] The following will explain in detail how to train the aforementioned target rotated rectangle regression head model and the target semantic category regression head model. The target rotated rectangle regression head model is obtained by training the first sample image, and the target semantic category regression head model is obtained by training the second sample image. The following will explain in detail the two stages of sample preprocessing and model training:

[0241] 1. Sample preprocessing

[0242] The material size of the SMT material in the first sample image is greater than the upper area threshold, and the first sample image is marked with the material area where each SMT material in the first sample image is located. The material size of the SMT material in the second sample image is not greater than the upper area threshold, and the second sample image is marked with the material center point of each SMT material in the second sample image.

[0243] The first sample image and the second sample image are pre-annotated in the following manner, the method comprising:

[0244] S201, acquiring a sample material area image obtained by photographing an SMT sample material.

[0245] Each sample material area image is obtained by photographing SMT sample materials of different sizes, and each sample material area image includes multiple SMT sample materials, and the materials of different sizes include first-size materials and second-size materials, the first-size materials are sample materials whose sizes are greater than the upper area threshold, and the second-size materials are sample materials whose sizes are not greater than the upper area threshold. The first-size materials are sample materials whose material size is the first size, and the second-size materials are sample materials whose material size is the second size.

[0246] S202 , marking the material center point of each SMT sample material in the sample material region image to obtain an original sample image.

[0247] The material center point of the SMT sample material may be obtained by manual marking or by other marking methods, which is not specifically limited here.

[0248] S203, in the original sample image, determine the original sample image obtained by photographing the material of the first size as the first preprocessed sample image, mark the material area where the sample material is located in the first preprocessed sample image, and obtain the first sample image.

[0249] S204: Determine the original sample image as the second sample image, or determine other images in the original sample image except the first preprocessed sample image as the second sample image.

[0250] That is, when training the target rotation regression head model, images of the material areas where the large-sized materials are located are used for training; when training the target semantic category regression head model, images of the material center points of the small-sized materials are used for training, or images of the material center points of the large-sized materials are used for training, or images of the material center points of the large-sized materials are used for training.

[0251] For example, an image of the material area where the large-sized materials are located is used as the first sample image, and an image of the material center point of each small-sized material is used as the second sample image. Figure 5 , Figure 5 A schematic diagram of sample annotations provided in an embodiment of the present application, Figure 5501 and 502 are sample images obtained by X-Ray equipment collecting real SMT material rolls, and the material center points of each sample material in the sample image are marked. The material center points of sample image 501 are marked to obtain a marked sample image 5011, and the material center points of sample image 502 are marked to obtain a marked sample image 5021. The first size material is marked twice, wherein the materials in sample image 502 and the marked sample image 5021 are first size materials, and the material areas where each sample material in the sample image is located are marked. Taking the material area as a material frame, where the material frame is a rectangular frame as an example, the marked sample image 5021 is marked twice to obtain a sample image 5022 after the second marking. This application is applicable to scenes with small-sized and large numbers of SMT materials. For ease of understanding and display, the figure only presents the relatively sparse arrangement of SMT materials, while the arrangement of SMT materials in actual applications will be denser.

[0252] Applying the above embodiment, firstly, a plurality of sample material area images obtained by photographing SMT sample materials are obtained, and these images contain sample materials of various sizes. The material center points are marked to obtain the original sample images, which not only ensures the diversity of the samples, but also enables the trained model to better adapt to materials of different sizes. The original sample images are divided into two groups according to the material size, one group is the first pre-processed sample images obtained by photographing the first size material (size greater than the upper limit area threshold), and the other group is the images obtained by photographing the first size material (size greater than the upper limit area threshold) and the second size material (size not greater than the upper limit area threshold), as the second sample image, the material area where the sample material is located in the first pre-processed sample image is marked to obtain the first sample image, and the model is trained using the first sample image and the second sample image, which helps the model learn the characteristics of materials of different sizes. In summary, this method improves the quality of the samples, enhances the accuracy and generalization ability of the trained model, is suitable for material quantity statistics scenarios of various sizes, and improves the accuracy of material quantity statistics.

[0253] Since there are many categories of SMT materials, the size in this article refers to the specifications of SMT materials, and the category in this article refers to the type of SMT materials. For example, resistors and inductors are different categories of SMT materials, and large-size resistors and small-size resistors are SMT materials of different sizes. In order to further improve the generalization ability of the trained model, in a possible embodiment, the labeled sample images can be uniformly sampled according to the category of the sample materials, and the sampled sample images are used as the first sample image and the second sample image. The above step S203 includes step S2031 and step S2032, specifically:

[0254] S2031: Determine the original sample image as the second preprocessed sample image, or determine other images in the original sample image except the first preprocessed sample image as the second preprocessed sample image.

[0255] S2032: uniformly sample each second preprocessed sample image according to the category of the photographed SMT sample material to obtain a sampling result as a second sample image.

[0256] The above step S204 includes steps S2041 and S2042, specifically:

[0257] S2041 , marking the material area where the SMT sample material is located in the first preprocessed sample image to obtain a third preprocessed sample image.

[0258] S2042 , uniformly sampling each third pre-processed sample image according to the category of the photographed SMT sample material to obtain a sampling result as a first sample image.

[0259] By applying the above embodiment, each second preprocessed sample image and each third preprocessed sample image are uniformly sampled according to the category of the sample material photographed, so that the obtained second sample image and the first sample image can comprehensively and evenly reflect the sample material of each category, thereby enhancing the generalization ability of the model trained by the first sample image and the second sample image, so that the entire model can show better performance when facing sample materials of different categories and sizes, thereby making the material quantity counting method applicable to material counting scenarios of various sizes, thereby improving the accuracy of material quantity counting.

[0260] Through the above method, the first sample image and the second sample image can be obtained, and the above first sample image and the second sample image together constitute the training set of the training model in this application. In order to further improve the performance of the model, in a possible embodiment, a verification set can also be set. The method of obtaining the verification set is the same as the method of obtaining the training set. The specific use of the verification set will be explained below and will not be repeated here. However, the obtained training set and verification set contain each real material category and the training set and verification set samples do not overlap. If the number of samples of a certain category is insufficient, the sampling requirements of the training set will be given priority.

[0261] 2. Model Training

[0262] The target rotation rectangle regression head model is pre-trained by the following methods, including:

[0263] S301, inputting a first sample image into an original rotated rectangular regression head model, so that the original rotated regression head model extracts visual features of the first sample image, and outputs all predicted regions containing objects in the first sample image and predicted categories and confidences of the predicted regions based on the visual features.

[0264] The prediction category is used to indicate the category of the object existing in the prediction area, the confidence is used to indicate the accuracy of the category, and the category of the object includes SMT materials and non-SMT materials.

[0265] S302: Calculate a first loss according to a first difference between each predicted area and a material area where each material marked in the first sample image is located.

[0266] Among them, the first loss is positively correlated with the first difference.

[0267] S303: Calculate a second loss according to a second difference between the predicted category and the target category of the predicted area.

[0268] Among them, the target category is the category of the object existing in the area with the same position as the prediction area in the first sample image. If the prediction category is the same as the target category, the second difference is negatively correlated with the confidence of the prediction area. If the prediction category is different from the target category, the second difference is positively correlated with the confidence of the prediction area, and the second loss is positively correlated with the second difference.

[0269] S304, adjusting the parameters of the original rotated rectangular regression head model in the direction of gradient descent of the first loss and the second loss to obtain a target rotated rectangular regression head model.

[0270] The target semantic category regression head model is pre-trained by the following methods, including:

[0271] S401, inputting the second sample image into the original semantic category regression head model, so that the target semantic category regression head model extracts the semantic features of the second sample image, and outputs the predicted material center point of each material in the material area image based on the semantic features.

[0272] S402: Calculate a third loss according to a third difference between each predicted material center point and a material center point of each material marked in the second sample image.

[0273] Among them, the third loss is positively correlated with the third difference.

[0274] S403: Adjust the parameters of the original semantic category regression head model in the direction of the third loss gradient descent to obtain the target semantic category regression head model.

[0275] In the above model training process, the first difference can be calculated using IOU (Intersection over Union) loss, and the second difference and the third difference can be calculated using cross entropy loss.

[0276] During the training process of deep learning models, due to the iterative nature of the training algorithm (such as gradient descent) and possible randomness, different combinations of model parameters may be obtained each time the training is performed. These different combinations of model parameters may correspond to different performances of the model on the training data. Therefore, multiple different training configurations are usually tried to generate multiple sets of training parameters.

[0277] In order to evaluate the model performance under these different parameter combinations, the above validation set can be used to determine whether the model is overfitting, underfitting, or whether it has achieved a good generalization ability. Then, the best set is selected based on the performance of the model on the validation set and used as the final parameters of the deep learning model. These parameters will be used for the final deployment of the model to perform prediction or classification tasks in practical applications.

[0278] By applying the above embodiment, the target rotated rectangle regression head model comprehensively considers the matching degree between the predicted area and the actual material area and the accuracy of the predicted category, and continuously optimizes the model parameters using the gradient descent method, thereby realizing the accurate prediction of the material position and category in the image. At the same time, the target semantic category regression head model focuses on the prediction of the material center point, and further improves the accuracy of material center recognition by minimizing the difference between the predicted material center point and the actual labeled center point. The target rotated rectangle regression head model and the target semantic category regression head model obtained by the above training method significantly improve the accuracy of material quantity statistics.

[0279] In the model training stage, in order to make the model applicable to various sizes and different categories of material statistics scenarios, multiple deep models with different structures are trained. A single model includes an image encoder, an image decoder, a rotated rectangle regression head or a semantic category classification head. The structure of the model can be found in Figure 6 , Figure 6 A schematic diagram of a depth model provided in an embodiment of the present application, such as Figure 6As shown in the figure, the deep model includes an image encoder, an image decoder, a rotated rectangle regression head or a semantic category classification head. The image encoder is a feature extractor composed of a convolution layer, a pooling layer, a normalization layer, an activation layer, etc., and its function is to downsample the input image and map it to an abstract feature space. The image decoder is a feature interpreter composed of a convolution layer, a normalization layer, an activation layer, an upsampling layer, etc., and its function is to gradually decode the abstract features extracted by the encoder into the required task output. The function of the output head is to convert the features output by the image decoder into the required material bounding box coordinates or material center point mask.

[0280] The deep model can be divided into two categories, including a first preset model and a second preset model. The first preset model includes a rotated rectangular regression head, and the second preset model includes a semantic category classification head. Multiple first preset models are trained based on the same first sample image, and the structures of different first preset models are different. Multiple second preset models are trained based on the same second sample image, and the structures of different second preset models are different.

[0281] In the structure of the first preset model, the image encoder can be a common target detection network feature extraction module, including but not limited to SSD (Single Shot MultiBox Detector, a multi-box prediction algorithm for a single detection), R-CNN (Regions with Convolutional Neural Networks, a target detection algorithm based on convolutional neural network), YOLO (You Only Look Once, a real-time target detection algorithm) series, etc. The image decoder can be a convolutional layer or a feature pyramid, etc. The rotated rectangular regression head is a plurality of sub-networks composed of convolutional layers.

[0282] In the structure of the second preset model, the image encoder can be a common downsampling network, including but not limited to common backbone networks such as ResNet (a residual network), VGGNet (a deep convolutional neural network), MobileNet (a lightweight convolutional neural network) and their improved structures, self-constructed deep convolutional networks, etc. The image decoder can be an upsampling module of common networks such as FCN (Fully Convolutional Networks, full convolutional network for semantic segmentation) series, Unet (deep learning network structure for image segmentation) series, DeepLab (deep learning model for image segmentation) series, self-constructed upsampling network, etc. The semantic category classification head is a softmax (a function) or sigmoid (a function) activation layer.

[0283] The first preset model including the rotated rectangle regression head is based on the target detection framework, and has better performance for counting some long, difficult, irregular, multi-peak, large-sized SMT material categories, but has poor performance for counting small-sized, sticky, large-quantity SMT material categories. The second preset model including the semantic category classification head is based on the semantic segmentation framework, and has better performance for counting small-sized, sticky, etc. SMT material categories. When using the model to count the number of materials, the appropriate model can be selected according to the size of the SMT material.

[0284] As mentioned above, the deep model includes multiple first preset models and multiple second preset models. For materials of similar size, different first preset models or different second preset models may be applicable due to factors such as material type and shape. Therefore, in a possible embodiment, when performing material quantity statistics, the applicable first preset model or second preset model can be selected according to preset conditions as the target rotation regression head model or the target semantic category regression head model, specifically including:

[0285] A model satisfying a first preset condition is determined from among a plurality of first preset models as a target rotation regression head model, and a model satisfying a second preset condition is determined from among a plurality of second preset models as a target semantic category regression head model.

[0286] Among them, the first preset condition and the second preset condition can be set according to the actual needs or experience of the user, and this application does not limit this.

[0287] By applying the above embodiment, by comparing multiple models with different structures and determining a model that meets specific conditions as a target model, the accuracy and generalization ability of the model can be significantly improved, thereby improving the accuracy of subsequent image processing and material quantity statistics.

[0288] In order to more clearly explain the aforementioned material quantity counting method, the following will be described in detail in combination with the hardware system, hardware module and hardware unit that execute the material quantity counting method:

[0289] See also Figure 7 , Figure 7 The structural diagram of the hardware system provided for the embodiment of the present application includes a data acquisition module 701, a counting model construction module 702 and a material quantity statistics module 703.

[0290] The data acquisition module 701 is used to execute each method step in the aforementioned sample preprocessing.

[0291] The counting model construction module 702 is used to execute each method step in the aforementioned model training.

[0292] The material quantity statistics module 703 is used to execute the aforementioned steps S101 to S104. The detailed description of the method steps executed by each module is as mentioned above and will not be repeated here.

[0293] See also Figure 8 , Figure 8 A schematic diagram of the structure of the data acquisition module provided in the embodiment of the present application is shown in FIG. Figure 8 As shown, the data acquisition module 701 includes a data collection unit 7011 , a data annotation unit 7012 , a data classification unit 7013 and a data division unit 7014 .

[0294] The data collection unit 7011 is used to execute the aforementioned step S201.

[0295] The data annotation unit 7012 is used to execute the aforementioned steps S201 , S202 , and S203 .

[0296] The data classification unit 7013 is used to execute the aforementioned step S203.

[0297] The data partitioning unit 7014 is used to execute the aforementioned steps S2031 to S2042. The detailed description of the method steps executed by each unit can be found above and will not be repeated here.

[0298] See also Fig. 9 , Fig. 9 A schematic diagram of a process flow of obtaining data by a data acquisition module provided in an embodiment of the present application, with the aforementioned Figure 5 Taking the sample image 501 and the sample image 502 in FIG. 7 as an example, the data collection unit 7011 collects and obtains the sample image 501 and the sample image 502 .

[0299] The data annotation unit 7012 annotates each sample image to obtain an annotated sample image 5011 and annotated sample image 5021 .

[0300] The data classification unit 7013 classifies the sample materials according to the material size, and divides them into large-size sample materials (i.e., first-size materials) and small-size sample materials (i.e., second-size materials). The materials in the sample image 502 and the labeled sample image 5021 are first-size materials. The material area is still represented by a material frame, and the material frame is a rectangular frame. For example, 5021 is labeled twice to obtain the sample image 5022 after the second labeling.

[0301] The data division unit 7014 uniformly samples the labeled sample images according to the categories of the sample materials to obtain a training set 901 and a validation set 902 .

[0302] See also Fig.10 , Fig.10A schematic diagram of the structure of the counting model module provided in the embodiment of the present application is shown in FIG. Fig.10 As shown, the counting model construction module 702 includes a counting model building unit 7021 and a counting model training unit 7022.

[0303] The counting model construction unit 7021 is used to construct the original rotated rectangular regression head model and the original semantic category regression head model in the aforementioned step S301 and step S401, and the counting model training unit 7022 is used to execute the aforementioned steps S301-step S304 and step S401-step S403. The detailed description of the method steps executed by each unit is found above and will not be repeated here.

[0304] See also Fig.11 , Fig.11 A schematic diagram of the structure of the SMT material quantity statistics module provided in the embodiment of the present application, such as Fig.11 As shown, the material quantity statistics module 703 includes a data preprocessing unit 7031, a material size estimation unit 7032, a model reasoning unit 7033 and a result screening unit 7034.

[0305] The data preprocessing unit 7031 is used to execute the aforementioned steps S1021-S1023, the material size estimation unit 7032 is used to execute the aforementioned steps S1031-S1034, the model inference unit 7033 is used to execute the aforementioned steps S104, S1041 and S1042, and the result screening unit 7034 is used to execute step S10421. The detailed description of the method steps executed by each unit is given above and will not be repeated here.

[0306] See also Fig.12 , Fig.12 A schematic diagram of the flow chart of the SMT material quantity statistics module counting provided in the embodiment of the present application, based on the aforementioned Figure 4 Taking the original image 401 in FIG. 4 as an example, the data preprocessing unit 7031 performs material area cropping on the original image 401 to obtain a material area image 402 .

[0307] The material size estimation unit 7032 inputs the material region image 402 to the target rotated rectangular regression head model 1201 to obtain all material regions where materials exist in the material region image. For materials whose material region area statistics are greater than the upper area threshold, the number of material regions can be used as the number of materials 1203. For materials whose material region area statistics are not greater than the upper area threshold, the material region image is input to the target semantic category regression head model 1202 to obtain a material center point mask image, and the number of material center point masks in the material center point mask image is used as the number of materials.

[0308] The result screening unit 7034 is used to screen the material center point mask image by taking the maximum value through pooling, and use the number of maximum value points obtained by screening as the number of materials 1204.

[0309] Corresponding to the aforementioned SMT material quantity counting method, the present application embodiment also provides an SMT material quantity counting device, see Fig.13 , Fig.13 The schematic diagram of the structure of the SMT material quantity counting device provided in the embodiment of the present application includes:

[0310] The image acquisition module 1301 is used to acquire an original image obtained by photographing the SMT material.

[0311] The image cropping module 1302 is used to crop the images of all SMT material placement areas from the original image as material area images.

[0312] The size determination module 1303 is used to determine the size of the SMT material according to the material area image.

[0313] The quantity statistics module 1304 is used to obtain the quantity of the SMT material based on the detection result of the material area image by the target material detection model.

[0314] The target material detection model is a material detection model corresponding to the size of the SMT material.

[0315] By applying the above embodiment, the original image is obtained by photographing the SMT material, and the image of the material placement area is accurately cropped from the original image, thereby effectively removing background information irrelevant to the SMT material quantity statistics. The material area image that does not contain background information is used to determine the material size of the SMT material, and according to the material size, a matching material detection model is selected. The matching material detection model can accurately output a detection result reflecting the number of SMT materials of this size, and then according to the detection result, the number of SMT materials is statistically obtained, thereby improving the pertinence of the SMT material quantity statistics, being suitable for SMT material statistics scenarios of various sizes, and thereby improving the accuracy of the SMT material quantity statistics.

[0316] In a possible implementation, the size determination module includes:

[0317] A first size determination submodule is used to detect the material regions where all SMT materials in the material region image are located according to the visual features of the material region image; wherein the visual features are used to represent the contours in the material region image and the positions of the contours;

[0318] The second size determination submodule is used to obtain a statistical value of the area of ​​each material region; wherein the statistical value is used to represent the area of ​​a single SMT material;

[0319] A third size determination submodule, configured to determine that the size of the SMT material is a first size if the area represented by the statistical value is greater than an upper area threshold;

[0320] a fourth size determination submodule, configured to determine that the size of the SMT material is a second size if the area represented by the statistical value is not greater than the upper area threshold;

[0321] The quantity statistics module includes:

[0322] A first quantity statistics submodule, configured to obtain the quantity of the SMT material based on the detection result of the material area image by a first material detection model if the size of the SMT material is the first size, wherein the first material detection model is a material detection model for performing SMT material detection based on visual features;

[0323] The second quantity statistics submodule is used to obtain the quantity of the SMT material based on the detection result of the material area image by a second material detection model if the size of the SMT material is the second size, wherein the second material detection model is a material detection model for SMT material detection based on semantic features.

[0324] In a possible implementation, the first material detection model is a target rotation regression head model, and the second material detection model is a target semantic category regression head model;

[0325] The first size determination submodule includes:

[0326] a first size determination unit, configured to input the material area image into the target rotation regression head model, so that the target rotation regression head model extracts visual features of the material area image, and outputs material areas where all SMT materials in the material area image are located based on the visual features;

[0327] The first quantity statistics submodule includes:

[0328] A first quantity counting unit, used for determining the quantity of the material areas output by the target rotation regression head model as the quantity of the SMT material;

[0329] The device also includes:

[0330] a center point output module, for inputting the material area image into the target semantic category regression head model if the size of the SMT material is the second size, so that the target semantic category regression head model extracts the semantic features of the material area image, and outputs the material center point of each of the SMT materials in the material area image based on the semantic features;

[0331] The second quantity statistics submodule includes:

[0332] A second quantity counting unit is used to determine the number of material center points output by the target semantic category regression head model as the number of the SMT material;

[0333] Among them, the target rotated rectangular regression head model is obtained by training the first sample image, the material size of the SMT material in the first sample image is greater than the upper limit area threshold, and the first sample image is annotated with the material area where each of the SMT materials in the first sample image is located; the target semantic category regression head model is obtained by training the second sample image, and the second sample image is annotated with the material center point of each of the SMT materials in the second sample image.

[0334] In a possible implementation manner, the target rotated rectangular regression head model is pre-trained in the following manner, including:

[0335] Inputting the first sample image into the original rotated rectangular regression head model, so that the original rotated regression head model extracts visual features of the first sample image, and outputs all predicted regions containing objects in the first sample image and the predicted category and confidence of each predicted region based on the visual features; wherein the predicted category is used to represent the category of the object existing in the predicted region, and the confidence is used to represent the accuracy of the category;

[0336] Calculating a first loss according to a first difference between each predicted area and a material area where each SMT material marked in the first sample image is located; wherein the first loss is positively correlated with the first difference;

[0337] Calculating a second loss according to a second difference between the predicted category and the target category of the predicted area; wherein the target category is a category of an object existing in the area of ​​the first sample image at the same position as the predicted area, if the predicted category is the same as the target category, then the second difference is negatively correlated with the confidence of the predicted area, and if the predicted category is different from the target category, then the second difference is positively correlated with the confidence of the predicted area, and the second loss is positively correlated with the second difference;

[0338] Adjusting the parameters of the original rotated rectangular regression head model in the direction of gradient descent of the first loss and the second loss to obtain a target rotated rectangular regression head model;

[0339] The target semantic category regression head model is pre-trained in the following ways, including:

[0340] Inputting the second sample image into the original semantic category regression head model, so that the target semantic category regression head model extracts the semantic features of the second sample image, and outputs the predicted material center point of each of the SMT materials in the material area image based on the semantic features;

[0341] Calculating a third loss according to a third difference between each predicted material center point and a material center point of each SMT material marked in the second sample image; wherein the third loss is positively correlated with the third difference;

[0342] In the direction of the third loss gradient descent, the parameters of the original semantic category regression head model are adjusted to obtain the target semantic category regression head model.

[0343] In a possible implementation, the device further includes:

[0344] A model selection module is used to determine a model that meets a first preset condition from multiple first preset models as a target rotation regression head model, and to determine a model that meets a second preset condition from multiple second preset models as a target semantic category regression head model; wherein the multiple first preset models are trained based on the same first sample image, and different first preset models have different structures, and the multiple second preset models are trained based on the same second sample image, and different second preset models have different structures.

[0345] In a possible implementation, the central point output module includes:

[0346] A first center point output submodule is used to output a material center point mask image according to the semantic features of the material area image; wherein the material center point mask is used to represent the material center points of each of the SMT materials in the material area image, the size of the material center point mask image is the same as that of the material area image, and each element is used to represent the confidence that the pixel at the same position in the material area image is the material center point;

[0347] The second quantity statistics submodule includes:

[0348] a third quantity counting unit, configured to pool the material center point mask image to obtain a pooled material center point mask image;

[0349] The fourth quantity counting unit is used to determine the maximum value greater than the preset screening threshold in the pooled material center point mask image, and determine the number of the obtained maximum values ​​as the number of the SMT material.

[0350] In a possible implementation manner, the first sample image and the second sample image are pre-annotated in the following manner, including:

[0351] Acquire a sample material area image obtained by photographing an SMT sample material; wherein each of the sample material area images is obtained by photographing SMT sample materials of different sizes, and each of the sample material area images includes a plurality of SMT sample materials, and the SMT sample materials of different sizes include a first size material and a second size material, the first size material is an SMT sample material having a size greater than the upper limit area threshold, and the second size material is an SMT sample material having a size not greater than the upper limit area threshold;

[0352] Marking the material center point of each of the SMT sample materials in the sample material area image to obtain an original sample image;

[0353] In the original sample image, determine the original sample image obtained by photographing the material of the first size as the first preprocessed sample image, and mark the material area where the SMT sample material is located in the first preprocessed sample image to obtain the first sample image;

[0354] The original sample image is determined as the second sample image, or other images in the original sample image except the first preprocessed sample image are determined as the second sample image.

[0355] In one possible implementation,

[0356] The step of determining the original sample image as the second sample image, or determining other images in the original sample image except the first preprocessed sample image as the second sample image, includes:

[0357] Determine the original sample image as a second preprocessed sample image, or determine other images in the original sample image except the first preprocessed sample image as a second preprocessed sample image;

[0358] According to the category of the photographed SMT sample material, each of the second pre-processed sample images is uniformly sampled to obtain a sampling result as a second sample image;

[0359] The step of marking the material area where the SMT sample material is located in the first preprocessed sample image to obtain the first sample image includes:

[0360] Marking the material area where the SMT sample material is located in the first preprocessed sample image to obtain a third preprocessed sample image;

[0361] According to the category of the photographed SMT sample material, each of the third pre-processed sample images is uniformly sampled to obtain a sampling result as the first sample image.

[0362] In a possible implementation, the image cropping module includes:

[0363] A first image cropping submodule is used to perform grayscale binarization processing on the original image based on the principle of maximizing the grayscale value variance between the foreground area and the background area after binarization to obtain a preprocessed image; wherein the original image is an X-ray image;

[0364] A second image cropping submodule, used for determining a grayscale value dividing line in the pre-processed image;

[0365] The third image cropping submodule is used to crop the sub-image within the boundary line from the pre-processed image as the material area image.

[0366] In a possible implementation manner, the second quantity statistics submodule includes:

[0367] a fifth quantity counting unit, configured to obtain the quantity of the SMT material by counting the detection result of the material area image that has not been magnified by the second material detection model if the area represented by the statistical value is greater than the lower area threshold;

[0368] The sixth quantity counting unit is used to obtain the quantity of the SMT material based on the detection result of the amplified material area image by the second material detection model if the area represented by the statistical value is not greater than the lower area threshold.

[0369] In a possible implementation manner, the first size determination submodule includes:

[0370] A second size determination unit is used to detect and obtain the candidate regions where all potential SMT materials are located in the material region image, and the confidence of each candidate region; wherein the confidence is used to indicate the probability that the SMT material exists in the candidate region;

[0371] The third size determination unit is used to determine the area whose confidence is not less than a preset confidence threshold in the candidate area as the material area where all the SMT materials in the material area image are located.

[0372] Corresponding to the aforementioned SMT material quantity counting method, the present application embodiment also provides a material counting machine, see Fig.14 , Fig.14 This is a schematic diagram of the structure of a material counting machine provided in an embodiment of the present application. The material counting machine 1400 includes an image acquisition device 1401, a processor 1402, and a display 1403;

[0373] The image acquisition device 1401 is used to obtain an original image by photographing the SMT material.

[0374] Processor 1402 is used to obtain the original image obtained by the image acquisition device by photographing the SMT material; crop the image of all material placement areas from the original image as the material area image; determine the size of the SMT material according to the material area image; based on the detection result of the material area image by the target material detection model, obtain the number of the SMT materials by statistics, and send the number of the SMT materials to the display; wherein the target material detection model is a material detection model corresponding to the size of the SMT material.

[0375] The display is used to display the quantity of the SMT material.

[0376] A communication connection is established between the image acquisition device 1401 and the processor 1402 , and a communication connection is established between the processor 1402 and the display 1403 .

[0377] The steps executed by the image acquisition device 1401 and the processor 1402 are the aforementioned steps S101 to S105, the specific contents of which have been described above and will not be repeated here.

[0378] The display can be any device with a display function, for example, a digital display, a computer, a television, etc., which are not specifically limited in this article. When displaying the quantity of SMT materials, it can be displayed in the form of a chart, in the form of text, or in the form of a picture, which will not be exhaustively listed here.

[0379] By applying the above embodiment, the original image is obtained by photographing the SMT material, and the image of the SMT material placement area is accurately cropped from the original image, effectively removing the background information irrelevant to the SMT material quantity statistics, and using the material area image that does not contain background information, the material size of the SMT material is determined, and according to the material size, a matching material detection model is selected, and the matching material detection model can accurately output the detection result reflecting the number of SMT materials of this size, and then according to the detection result, the number of SMT materials is statistically obtained, thereby improving the pertinence of the SMT material quantity statistics, and being applicable to the statistical scenes of SMT materials of various sizes, thereby improving the accuracy of the SMT material quantity statistics. After the SMT material quantity is statistically obtained, the SMT material quantity is sent to the display for display, thereby increasing the transparency of SMT material management and providing timely and accurate data support for SMT material management.

[0380] The present application also provides an electronic device, such as Fig.15 As shown, including:

[0381] Memory 1501, used for storing computer programs;

[0382] The processor 1502 is used to implement the following steps when executing the program stored in the memory 1501:

[0383] Get the original image obtained by photographing the SMT material;

[0384] Cropping the original image to obtain images of all SMT material placement areas as material area images;

[0385] Determining the size of the SMT material according to the material area image;

[0386] Based on the detection result of the material area image by the target material detection model, the quantity of the SMT material is obtained by statistics; wherein the target material detection model is a material detection model corresponding to the size of the SMT material.

[0387] In addition, the electronic device may further include a communication bus and / or a communication interface, and the processor 1502, the communication interface, and the memory 1501 communicate with each other via the communication bus.

[0388] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one size of bus.

[0389] The communication interface is used for communication between the above electronic device and other devices.

[0390] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0391] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0392] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned material quantity counting methods are implemented.

[0393] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the material quantity counting methods in the above embodiments.

[0394] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0395] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0396] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0397] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A method for counting the number of SMT materials, characterized in that: The method comprises: Get the original image obtained by photographing the SMT material; Cropping the original image to obtain images of all SMT material placement areas as material area images; Determining the size of the SMT material according to the material area image; Based on the detection result of the material area image by the target material detection model, the quantity of the SMT material is obtained by statistics; wherein the target material detection model is a material detection model corresponding to the size of the SMT material.

2. The method according to claim 1, characterized in that Determining the size of the SMT material according to the material area image includes: According to the visual features of the material area image, the material areas where all the SMT materials in the material area image are located are detected; wherein the visual features are used to represent the contours in the material area image and the positions of the contours; Obtaining statistical values ​​of the areas of the material regions by counting; wherein the statistical values ​​are used to represent the area of ​​a single SMT material; If the area represented by the statistical value is greater than the upper area threshold, determining that the size of the SMT material is a first size; If the area represented by the statistical value is not greater than the upper area threshold, determining that the size of the SMT material is a second size; The detection result of the material area image based on the target material detection model is used to obtain the quantity of the SMT material by counting, including: If the size of the SMT material is the first size, the number of SMT materials is obtained by counting the detection result of the material area image based on a first material detection model, wherein the first material detection model is a material detection model for performing SMT material detection based on visual features; If the size of the SMT material is the second size, the number of SMT materials is statistically obtained based on the detection result of the material area image by a second material detection model, wherein the second material detection model is a material detection model for SMT material detection based on semantic features.

3. The method according to claim 2, characterized in that The first material detection model is a target rotation regression head model, and the second material detection model is a target semantic category regression head model; The detecting and obtaining the material regions where all SMT materials in the material region image are located according to the visual features of the material region image includes: Inputting the material area image into the target rotation regression head model, so that the target rotation regression head model extracts visual features of the material area image, and outputs the material areas where all SMT materials in the material area image are located based on the visual features; The method of obtaining the quantity of the SMT material by counting the detection result of the material area image based on the first material detection model includes: Determine the number of the material areas output by the target rotary regression head model as the number of the SMT materials; The method further comprises: If the size of the SMT material is the second size, input the material area image into the target semantic category regression head model, so that the target semantic category regression head model extracts semantic features of the material area image, and outputs the material center point of each of the SMT materials in the material area image based on the semantic features; The method of obtaining the quantity of the SMT material by counting the detection result of the material area image based on the second material detection model includes: Determine the number of material center points output by the target semantic category regression head model as the number of the SMT materials; Among them, the target rotated rectangular regression head model is obtained by training the first sample image, the material size of the SMT material in the first sample image is greater than the upper limit area threshold, and the first sample image is annotated with the material area where each of the SMT materials in the first sample image is located; the target semantic category regression head model is obtained by training the second sample image, and the second sample image is annotated with the material center point of each of the SMT materials in the second sample image.

4. The method according to claim 3, characterized in that: The target rotated rectangular regression head model is pre-trained in the following manner, including: Inputting the first sample image into the original rotated rectangular regression head model, so that the original rotated regression head model extracts visual features of the first sample image, and outputs all predicted regions containing objects in the first sample image and the predicted category and confidence of each predicted region based on the visual features; wherein the predicted category is used to represent the category of the object existing in the predicted region, and the confidence is used to represent the accuracy of the category; Calculating a first loss according to a first difference between each predicted area and a material area where each SMT material marked in the first sample image is located; wherein the first loss is positively correlated with the first difference; Calculating a second loss according to a second difference between the predicted category and the target category of the predicted area; wherein the target category is a category of an object existing in the area of ​​the first sample image at the same position as the predicted area, if the predicted category is the same as the target category, then the second difference is negatively correlated with the confidence of the predicted area, and if the predicted category is different from the target category, then the second difference is positively correlated with the confidence of the predicted area, and the second loss is positively correlated with the second difference; Adjusting the parameters of the original rotated rectangular regression head model in the direction of gradient descent of the first loss and the second loss to obtain a target rotated rectangular regression head model; The target semantic category regression head model is pre-trained in the following ways, including: Inputting the second sample image into the original semantic category regression head model, so that the target semantic category regression head model extracts the semantic features of the second sample image, and outputs the predicted material center point of each of the SMT materials in the material area image based on the semantic features; Calculating a third loss according to a third difference between each predicted material center point and a material center point of each SMT material marked in the second sample image; wherein the third loss is positively correlated with the third difference; In the direction of the third loss gradient descent, the parameters of the original semantic category regression head model are adjusted to obtain the target semantic category regression head model.

5. The method according to claim 3, characterized in that: The method further comprises: A model that meets a first preset condition is determined from among multiple first preset models as a target rotation regression head model, and a model that meets a second preset condition is determined from among multiple second preset models as a target semantic category regression head model; wherein the multiple first preset models are trained based on the same first sample image, and different first preset models have different structures, and the multiple second preset models are trained based on the same second sample image, and different second preset models have different structures.

6. The method according to claim 3, characterized in that The step of outputting the material center point of each of the SMT materials in the material region image based on the semantic feature includes: Outputting a material center point mask image according to the semantic features of the material area image; wherein the material center point mask is used to represent the material center point of each of the SMT materials in the material area image, the size of the material center point mask image is the same as the material area image, and each element is used to represent the confidence that the pixel at the same position in the material area image is the material center point; The step of determining the number of material center points output by the target semantic category regression head model as the number of the SMT materials includes: Pooling the material center point mask image to obtain a pooled material center point mask image; In the pooled material center point mask image, a maximum value greater than a preset screening threshold is determined, and the number of the obtained maximum values ​​is determined as the number of the SMT material.

7. The method according to claim 3, characterized in that The first sample image and the second sample image are pre-annotated in the following manner, including: Acquire a sample material area image obtained by photographing an SMT sample material; wherein each of the sample material area images is obtained by photographing SMT sample materials of different sizes, and each of the sample material area images includes a plurality of SMT sample materials, and the SMT sample materials of different sizes include a first size material and a second size material, the first size material is an SMT sample material having a size greater than the upper limit area threshold, and the second size material is an SMT sample material having a size not greater than the upper limit area threshold; Marking the material center point of each of the SMT sample materials in the sample material area image to obtain an original sample image; In the original sample image, determine the original sample image obtained by photographing the material of the first size as the first preprocessed sample image, and mark the material area where the SMT sample material is located in the first preprocessed sample image to obtain the first sample image; The original sample image is determined as the second sample image, or other images in the original sample image except the first preprocessed sample image are determined as the second sample image.

8. The method according to claim 7, characterized in that The step of determining the original sample image as the second sample image, or determining other images in the original sample image except the first preprocessed sample image as the second sample image, includes: Determine the original sample image as a second preprocessed sample image, or determine other images in the original sample image except the first preprocessed sample image as a second preprocessed sample image; According to the category of the photographed SMT sample material, each of the second pre-processed sample images is uniformly sampled to obtain a sampling result as a second sample image; The step of marking the material area where the SMT sample material is located in the first preprocessed sample image to obtain the first sample image includes: Marking the material area where the SMT sample material is located in the first preprocessed sample image to obtain a third preprocessed sample image; According to the category of the photographed SMT sample material, each of the third pre-processed sample images is uniformly sampled to obtain a sampling result as the first sample image.

9. The method according to claim 1, characterized in that: The image of all SMT material placement areas is obtained by cutting out from the original image as the material area image, including: Based on the principle of maximizing the grayscale value variance between the foreground area and the background area after binarization, the original image is subjected to grayscale binarization processing to obtain a preprocessed image; wherein the original image is an X-ray image; Determining a grayscale value boundary in the preprocessed image; A sub-image within the boundary line is cut out from the pre-processed image as a material area image.

10. The method according to claim 2, characterized in that The method of obtaining the number of SMT materials by counting the detection result of the material area image based on the second material detection model includes: If the area represented by the statistical value is greater than the lower area threshold, the number of SMT materials is obtained by counting the detection result of the second material detection model on the material area image that has not been magnified; If the area represented by the statistical value is not greater than the lower area threshold, the number of SMT materials is statistically obtained based on the detection result of the second material detection model on the enlarged material area image.

11. The method according to claim 2, characterized in that The detection obtains the material regions where all SMT materials in the material region image are located, including: Detecting and obtaining the candidate regions where all potential SMT materials in the material region image are located, and the confidence of each candidate region; wherein the confidence is used to indicate the probability that an SMT material exists in the candidate region; In the candidate regions, regions whose confidence levels are not lower than a preset confidence threshold are determined as material regions where all SMT materials in the material region image are located.

12. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-11 when executing a program stored in a memory.

13. A material counting machine, characterized in that: The material counting machine includes an image acquisition device, a processor and a display; The image acquisition device is used to obtain an original image by photographing the SMT material; The processor is used to obtain the original image obtained by the image acquisition device by photographing the SMT material; crop the image of all material placement areas from the original image as the material area image; and determine the size of the SMT material according to the material area image; Based on the detection result of the target material detection model on the material area image, the number of the SMT materials is obtained by counting, and the number of the SMT materials is sent to the display; wherein the target material detection model is a material detection model corresponding to the size of the SMT material; The display is used to display the quantity of the SMT material.

Images