Pin size measurement method and device suitable for power device module

Abstract

The present invention relates to a method and device for measuring the pin size of a power device module. The method comprises: obtaining a pin image of the power device module for which the pin size is to be measured, and loading the pin image into a pre-constructed pin target recognition model; the pin target recognition model performs target recognition on the pins in the pin image, so as to obtain pin priori frame prediction information of the pins on the power device module after target recognition; post-processing the pin priori frame prediction information, so as to generate pin identification information of the pins after post-processing; and calculating the physical size of the pins based on the pin identification information of the pins. The present invention can meet the industrial demand for measuring the pin size on the power device module, and improve the efficiency and accuracy of pin size measurement.

Description

Technical Field

[0001] The invention relates to a measuring method and a device, in particular to a pin size measuring method and a device suitable for a power device module. Background Art

[0002] IGBT (Insulate-Gate Bipolar Transistor) is a power semiconductor device. After packaging IGBT, you can get IGBT module. In order to facilitate connection, IGBT module generally has pins. The pin size on the IGBT module is very important, such as whether the entire IGBT module can be better embedded in power equipment or new energy vehicles. In other words, the compliance of pin size is a necessary condition for the compliance of the entire IGBT module.

[0003] Common target detection methods can be used to detect IGBT modules and pins on IGBT modules. Target detection methods can be classified according to algorithm types, including one-stage algorithms and two-stage algorithms. In the two-stage algorithm model, the representative algorithm is Faster-RCNN, which achieves fast and accurate target positioning by introducing Region Proposal Network (RPN). It combines two stages of training based on convolutional neural networks, first generating candidate regions, then classification and fine regression, but the disadvantage is that the algorithm has a large amount of calculation and redundant calculation.

[0004] In recent years, there have been continuous studies on pin size detection algorithms on IGBT modules, but they are difficult to adapt to the industrial demand for measuring pin size on IGBT modules. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a method and device for measuring the pin size of a power device module, which can meet the industrial demand for measuring the pin size on a power device module and improve the efficiency and accuracy of the pin size measurement.

[0006] According to the technical solution provided by the present invention, a method for measuring the pin size of a power device module is provided. The method for measuring the pin size of the power device module comprises:

[0007] For a power device module to be measured for pin size, obtaining a pin image of the power device module, and loading the pin image into a pre-built pin target recognition model, wherein the pin image of the power device module includes a pin of the power device module;

[0008] The pin target recognition model performs target recognition on the pins in the pin image to obtain pin priori box prediction information of the pins on the power device module after target recognition;

[0009] Post-processing the stitch priori frame prediction information to generate stitch identification information of the stitch after post-processing, wherein the stitch identification information includes a pixel width of the stitch and a pixel height of the stitch;

[0010] Based on the stitch identification information of the stitch, the physical size of the stitch is calculated, wherein the physical size of the stitch includes the physical width of the stitch in the vertical projection direction and the physical height of the stitch in the vertical projection direction.

[0011] When calculating the physical size of the pin based on the pin identification information of the pin, we have:

[0012]

[0013] in, is the physical width of the pin, is the physical height of the pin, is the pixel width of the pin, is the pixel height of the pin, is the physical size of a pixel after calibration.

[0014] The stitch target recognition model is constructed based on the YOLOV5 model. The stitch target recognition model includes a backbone network, a neck network and an output network connected in sequence, wherein:

[0015] The neck network includes a neck output connection convolution residual module, a neck downsampling unit adaptively connected to the backbone P3 layer in the backbone network, and a neck convolution unit adaptively connected to the backbone P5 layer in the backbone network;

[0016] The neck downsampling unit and the neck convolution unit are both connected to the neck output connection convolution residual module, and are adaptively connected to the output network through the neck output connection residual module;

[0017] The output network includes at least one output convolutional layer;

[0018] For the pin image of the power device module, the pin target recognition model outputs the pin recognition information through the output convolution layer.

[0019] The neck downsampling unit comprises a neck first downsampling module and a neck second downsampling module connected in sequence, wherein the output of the backbone P3 layer in the backbone network is loaded into the neck output connection residual module via the neck first sampling module and the neck second downsampling module;

[0020] The neck convolution unit includes at least one neck convolution layer, and the output of the backbone P5 layer in the backbone network is loaded into the neck output connection residual module via the neck convolution layer.

[0021] When constructing a pin target recognition model, the construction method includes:

[0022] A basic model for pin target recognition based on the YOLOV5 model and a training data set for model training of the basic model for pin target recognition are constructed, wherein:

[0023] The training data set includes a plurality of training samples, the training samples have the same pixel size, each training sample includes only one pin of the power device module, and each training sample is marked with the position information of the pin;

[0024] The stitch target recognition basic model is trained using the training data set. After the training of the stitch target recognition basic model reaches a target state, a stitch target recognition model is generated based on the stitch target recognition basic model that has reached the target training state.

[0025] When training the basic model for pin target recognition, after each generation of training, the training loss after the current generation of training is calculated;

[0026] When the training loss does not decrease after the preset generation training, the model training of the pin target recognition basic model reaches the target training state;

[0027] When calculating the training loss, the training loss includes positioning loss and confidence loss.

[0028] For the training loss, we have:

[0029]

[0030] in, is the training loss, is the positioning loss, is the weight coefficient of positioning loss, is the confidence loss, is the weight coefficient of confidence loss;

[0031] Positioning loss , then:

[0032]

[0033] in, is the prediction box of the training sample, is the true box of the training sample, is the intersection-over-union ratio of the predicted box and the true box of the training sample, is the square of the distance between the center of the predicted box and the true box, It is the square of the longest diagonal length of the minimum enclosing rectangle of the predicted box and the true box; is the width of the prediction box, is the height of the prediction box, is the width of the real frame, is the height of the real frame;

[0034] For confidence loss , then:

[0035]

[0036] in, is the total number of prior frames, To determine whether the current prior frame is responsible for the prediction of the real frame after matching the prior frame, if it is, it is 1, otherwise it is 0; To determine whether the current prior frame is responsible for the prediction of the real frame after matching the prior frame, if it is responsible, it is 0, otherwise it is 1; BCE is the binary cross entropy loss function, The confidence c predicted by the current prior box.

[0037] For any training sample, when preparing the training sample, it includes:

[0038] Capturing an image of the power device module to obtain a basic image of the pins of the power device module, wherein the basic image of the pins of the power device module includes at least one pin;

[0039] The basic image of the power device pins is annotated, and the pins are extracted after the annotation to generate a power device module pin training image after the extraction, wherein:

[0040] The power device module pin training image includes only one pin, and the pin is located at the center of the power device module pin training image, and the size of the power device module pin training image is the same as the preset pixel size;

[0041] The pin annotation information of the generated pins is annotated on the pin training image of the power device module;

[0042] Generate training samples based on the power device module pin training images.

[0043] When generating training samples based on the power device module pin training images, it also includes:

[0044] Performing image enhancement processing on the power device module pin training image, wherein the image enhancement processing on the power device module pin training image includes sequentially performing HSV brightness contrast transformation and random area Gaussian blur;

[0045] When performing HSV brightness contrast transformation processing, the power device module pin training image is converted into an HSV pin training image, and after the HSV brightness contrast transformation processing, an HSV brightness transformation processed pin training image is generated;

[0046] For the converted HSV stitch training image, add an enhanced first random factor to each pixel of the V channel of the HSV stitch training image, and randomly multiply each pixel of the S channel of the HSV stitch training image by an enhanced second random factor;

[0047] When performing random area Gaussian blur processing, a Gaussian window is randomly generated for the stitch training image after the HSV brightness transformation processing, and the generated Gaussian window is used to perform random Gaussian blur processing on the stitch training image after the HSV brightness transformation processing, so as to generate training samples after the random Gaussian blur processing.

[0048] A pin size measuring device suitable for a power device module comprises a pin size measuring terminal, wherein the pin size measuring terminal adopts the above-mentioned measuring method to determine the physical size of the pins of the power device module.

[0049] The advantages of the present invention are as follows: a pin target recognition model is constructed based on YOLOV5, and the pin target recognition model is used to perform target recognition on the pin image of the power device module to obtain pin recognition information, and the physical size of the pin can be calculated according to the pin recognition information; when the pin target recognition model is constructed based on YOLOV5, the weight occupancy is smaller and the processing speed is faster, thereby meeting the industrial measurement requirements for the pin size on the power device module and improving the efficiency and measurement accuracy of the pin size measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 The figure is a schematic diagram of a flow chart of an embodiment of the stitch size measurement of the present invention.

[0051] Figure 2 A block diagram of an embodiment of the stitch target recognition model of the present invention.

[0052] Figure 3 A schematic diagram of an embodiment of a training sample of the present invention. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with specific drawings and embodiments.

[0054] In order to meet the industrial demand for measuring the pin size on a power device module and improve the efficiency and accuracy of the pin size measurement, a method for measuring the pin size of a power device module is provided. In one embodiment of the present invention, the method for measuring the pin size of a power device module includes:

[0055] For a power device module to be measured for pin size, obtaining a pin image of the power device module, and loading the pin image into a pre-built pin target recognition model, wherein the pin image of the power device module includes a pin of the power device module;

[0056] The pin target recognition model performs target recognition on the pins in the pin image to obtain pin priori box prediction information of the pins on the power device module after target recognition;

[0057] Post-processing the stitch priori frame prediction information to generate stitch identification information of the stitch after post-processing, wherein the stitch identification information includes a pixel width of the stitch and a pixel height of the stitch;

[0058] Based on the stitch identification information of the stitch, the physical size of the stitch is calculated, wherein the physical size of the stitch includes the physical width of the stitch in the vertical projection direction and the physical height of the stitch in the vertical projection direction.

[0059] Specifically, the power device module can be a power device module formed by an IGBT device package, or a power device module formed by a MOSFET device package, and the type of the power device module can be selected as needed. As can be seen from the above description, for ease of use, the power device module can include multiple pins, so the pin size measurement specifically refers to measuring the physical size of one or more pins on the power device module, and the physical size is the actual size of the pin, and the physical size can be mentioned as follows: the physical width and physical height in the vertical projection direction of the pin.

[0060] When measuring the pins on the power device module, the size of one pin is generally measured at a time. Therefore, for multiple pins on the power device module, the same measurement method needs to be used multiple times to determine the corresponding pin size of each pin. Therefore, the pin image of the power device module includes a pin of the power device module. The pin image of the power device module can be generated through existing image acquisition and necessary processing. When obtaining the pin image of the power device module, in one embodiment of the present invention, the pins of the power device module are placed in a vertical state, the camera is placed directly above the pins, and the pins are photographed from directly above the pins using the camera. At this time, it can be known that the pin image of the power device module is an image of the vertical projection direction of the current pin, and the vertical projection direction is the projection direction from the top of the pin to the bottom of the pin. The bottom of the pin is the end in contact with the power device module, and the top of the pin is the end of the pin away from the power device module.

[0061] It should be noted that, limited by the depth of field of the camera, in order to improve the pin image of the power device module to be clearer and improve the accuracy of the target recognition of the pin image of the power device module, the camera shooting position needs to be calibrated. It can be seen that the camera shooting position calibration is to determine the optimal position for the camera to capture the pin image of the power device module. The following is an explanation of the method for calibrating the camera shooting position.

[0062] Specifically, when calibrating the shooting position of the camera, a feasible method includes:

[0063] Step 1, placing the power device module at a position to be photographed, and moving the camera to the top of a pin, and then using the camera to photograph an image of the pin directly below the pin, so as to obtain a pin calibration image after photographing, and calculating the pin clarity of the pin calibration image;

[0064] Step 2, move the camera downward by a preset distance, and after the movement, use the camera to capture a pin calibration image at the current position, and calculate the pin clarity of the pin calibration image at the current position;

[0065] Step 3, repeat the above step 2, and determine the change of the corresponding pin definition according to the order of camera movement, wherein, when the monotonicity of the pin definition changes, the position of the camera last time when the monotonicity change occurs is selected as the camera calibration position.

[0066] It should be noted that the camera for shooting position calibration is the same camera as the camera for obtaining the pin image of the power device module mentioned above, such as a telecentric lens camera. The power device module is placed in the position to be photographed, specifically refers to placing the power device module flat, at which time the pins of the power device module are in a vertical distribution state. Since the power device module has multiple pins, the camera can be located directly above any pin when calibrating the shooting position, and the height of the camera above the pin can be selected as needed. When the height position of the camera is determined, the pins of the power device module can be photographed and a pin calibration image can be obtained. The pin calibration image also only contains the pins directly below the camera.

[0067] For the stitch calibration image, the stitch clarity needs to be calculated. In order to adapt to the stitch characteristics in the stitch calibration image, in one embodiment of the present invention, the stitch clarity of the stitch calibration image is calculated as:

[0068]

[0069] in, For pin clarity, is the clarity based on the discrete Fourier transform of the two-dimensional image, is the clarity based on the Sobel operator, , is the weighting coefficient.

[0070] The clarity of the two-dimensional image based on discrete Fourier transform , then:

[0071]

[0072] Where M is the pixel width of the two-dimensional discrete Fourier transform image, N is the pixel height of the two-dimensional discrete Fourier transform image, for is the average value, = , The pixel coordinates in the two-dimensional discrete Fourier transform image are The pixel value of .

[0073] In order to obtain clarity during implementation It is necessary to perform a two-dimensional discrete Fourier transform on the pin calibration image to obtain a two-dimensional discrete Fourier transform image after the two-dimensional discrete Fourier transform. The method and process of performing a two-dimensional discrete Fourier transform on the pin calibration image can be consistent with the existing ones, that is, the method of obtaining the two-dimensional discrete Fourier transform image is consistent with the existing ones.

[0074] After obtaining the two-dimensional Fourier transform image, the pixel width M and pixel height N of the two-dimensional Fourier transform image can be obtained, and the pixel value corresponding to each pixel coordinate can be obtained , and then we can get ,as well as is the average , thus, the clarity of the discrete Fourier transform of the two-dimensional image can be obtained .

[0075] For clarity based on the sobel operator , in one embodiment of the present invention,

[0076]

[0077] in, is the sum of the convolution of a single pixel in the stitch calibration image using two 3*3 sobel convolution kernels; A is the pixel width of the stitch calibration image, and B is the pixel height of the stitch calibration image. The pixel coordinate of one pixel in the pin calibration image.

[0078] Specifically, for clarity based on the Sobel operator , the two 3*3 convolution kernels can be respectively , . Weighting coefficient , weighting coefficient The value can be 0.5. Of course, other values ​​are also available. The weighting coefficient can be used to adjust the pin clarity. .

[0079] In specific implementation, the pin clarity It can be seen from the description that the pin clarity It can better characterize the characteristics of the calibration pin image and improve the accuracy of camera shooting position calibration.

[0080] It should be noted that the camera is moved downward by a preset distance, which is the step length of each downward movement of the camera. The camera moves downward, specifically, the camera is moved closer to the pin direction. The preset distance can be 5 mm. Of course, it can also be other distances, which can be selected as needed. After the camera is moved downward by the preset distance, the position of the camera is maintained, and the pins are photographed again to obtain the pin calibration image at the current position, and the pin clarity of the pin calibration image is determined by the above method.

[0081] After the camera is moved multiple times at a preset distance, the pin clarity of multiple pin calibration images can be obtained, and the change trend of the pin clarity can be judged according to the position order of the camera moving downward. It can be understood that when the camera approaches the pin direction, the pin clarity will gradually increase, that is, the monotonicity of the pin clarity change trend should be gradually increasing. When the monotonicity of the pin clarity changes, that is, the pin clarity no longer increases, at this time, the position of the camera at the last time when the monotonic change occurs is selected as the camera calibration position.

[0082] It can be understood that in order to ensure that the trend of change in stitch clarity is monotonic, the camera can be placed at a higher position in step 1, wherein the higher position is related to the parameters of the camera, etc., and can be specifically selected based on experience to ensure that the trend of change in stitch clarity is monotonic.

[0083] After obtaining the camera calibration position, the camera is fixed at the camera calibration position, and the pins of the power device module are photographed to obtain the corresponding pin image of the power device module. From the above description, it can be seen that the pin image of the power device module has better clarity, which can improve the accuracy and reliability of pin target recognition.

[0084] In order to be able to perform target recognition on the pins on the power device module, it is necessary to pre-build a pin target recognition model. The pin target recognition model can be used to perform target recognition on the pin image on the power device module, and generate pin priori frame prediction information of the pin after target recognition. Thereafter, the pin priori frame prediction information of the pin is post-processed to generate pin recognition information. The generated pin recognition information may at least include the pixel width of the pin and the pixel height of the pin. For the post-processing of the pin priori frame prediction information, please refer to the following corresponding instructions.

[0085] Furthermore, according to the pin identification information, the physical size of the pin can be calculated, wherein the physical size of the pin includes the physical width of the pin in the vertical projection direction and the physical height of the pin in the vertical projection direction. It can be understood that the physical height of the pin in the vertical projection direction is the length of the pin in the vertical projection direction, and the physical width of the pin in the vertical projection direction is the physical height of the pin in the vertical projection direction. Figure 3 An embodiment of the training sample in Figure 3 Here, w is the physical width corresponding to the pin that needs to be measured in the present invention, and h is the physical height corresponding to the pin that needs to be measured in the present invention.

[0086] In one embodiment of the present invention, when calculating the physical size of the pin based on the pin identification information of the pin, the following is obtained:

[0087]

[0088] in, is the physical width of the pin, is the physical height of the pin, is the pixel width of the pin, is the pixel height of the pin, is the physical size of a pixel after calibration.

[0089] It can be seen from the above description that when calculating the physical size of the pin, it is necessary to provide the physical size of a pixel after calibration; in one embodiment of the present invention, a feasible calibration method for the physical size of a pixel obtained after calibration is:

[0090] Provide a reference ruler, and use an image acquisition device for capturing stitch images to acquire an image of the reference ruler, so as to obtain an image of the reference ruler, wherein the reference ruler is placed horizontally when acquiring the image of the reference ruler;

[0091] After obtaining the reference ruler image, draw a line segment with a scale of 0-1 cm on the reference ruler on the reference ruler image, and calculate the pixel length L of the line segment;

[0092] 1cm=10000μm, and the value of 10000 / L is configured as a pixel physical size. .

[0093] Specifically, the reference ruler can adopt an existing commonly used steel ruler, and the type of the reference ruler can be selected as needed to meet the requirements of calibration. The image acquisition device for capturing the pin image, that is, a device for capturing the pin of the power device module using the image acquisition device, can be an existing commonly used camera, such as the camera for calibrating the shooting position mentioned above. When the image acquisition device for capturing the pin image is used to capture the image of the reference ruler, it specifically refers to that the height and focal length of the camera used to capture the reference ruler image are consistent with the height and focal length when the pin image is captured, so as to minimize the calibration error, that is, when the camera captures the reference ruler image, it also needs to be located at the camera calibration position. At this time, the distance between the reference ruler and the camera is consistent with the position between the pin top and the camera.

[0094] When capturing an image of the reference ruler, the reference ruler is generally placed horizontally. After capturing the reference ruler image, a line segment with a length interval of 1 cm can be extracted or selected on the reference ruler image. After selecting the line segment, the pixel length L of the selected line segment can be calculated according to the commonly used technical means in the technical field.

[0095] Convert the actual length of a 1cm line segment to 10000μm. After calculating the pixel length L, the physical size of a pixel can be obtained. , so the physical length of a pixel is The unit is μm / pixel length. The physical width of the above pins , the physical height of the pin , the corresponding unit of measurement is μm.

[0096] Of course, other methods can also be used to calibrate a pixel physical length The specific calibration method can be selected according to actual needs, and no examples will be given here.

[0097] In one embodiment of the present invention, the stitch target recognition model is constructed based on the YOLOV5 model, and the stitch target recognition model includes a backbone network, a neck network and an output network connected in sequence, wherein:

[0098] The neck network includes a neck output connection convolution residual module, a neck downsampling unit adaptively connected to the backbone P3 layer in the backbone network, and a neck convolution unit adaptively connected to the backbone P5 layer in the backbone network;

[0099] The neck downsampling unit and the neck convolution unit are both connected to the neck output connection convolution residual module, and are adaptively connected to the output network through the neck output connection residual module;

[0100] The output network includes at least one output convolutional layer;

[0101] For the pin image of the power device module, the pin target recognition model outputs the pin prior box prediction information of the pin through the output convolution layer.

[0102] Figure 2 An embodiment of constructing a stitch target recognition model based on the YOLOV5 model is shown in the figure. In the figure, the stitch target recognition model includes a backbone network, a neck network and an output network connected in sequence, wherein the backbone network can adopt the backbone network in the YOLOV5 model, and the neck network and the output network in the YOLOV5 model are adjusted. In the figure, the corresponding outputs of the backbone P3 layer and the backbone P5 layer in the YOLOV5 model are loaded into the neck network, and a neck downsampling unit, a neck convolution unit and a neck output connection convolution residual module are set in the neck network, wherein the neck downsampling unit corresponds to the output of the backbone P3 layer, the neck convolution unit corresponds to the output of the backbone P5 layer, the neck downsampling unit and the neck convolution unit are connected to the neck output connection convolution residual module, and are connected to the output network through the neck output connection convolution residual module.

[0103] Depend on Figure 2 As can be seen from the above description, when the present invention constructs the pin target recognition model, it is necessary to improve the existing YOLOV5 network. Compared with the existing YOLOV5 network, the pin target recognition model of the present invention can omit the output of the backbone P4, reduce the network parameters of the pin target recognition model, increase the training amount of the pin target recognition model, and also improve the efficiency of target recognition of the pin image of the power device module.

[0104] In one embodiment of the present invention, the neck downsampling unit comprises a neck first downsampling module and a neck second downsampling module connected in sequence, wherein the output of the backbone P3 layer in the backbone network is loaded into the neck output connection residual module via the neck first downsampling module and the neck second downsampling module;

[0105] The neck convolution unit includes at least one neck convolution layer, and the output of the backbone P5 layer in the backbone network is loaded into the neck output connection residual module via the neck convolution layer.

[0106] Figure 2 In the output network, there is at least one output convolutional layer. Figure 2Also shown is an embodiment of the backbone network, the neck downsampling unit in the neck network, and the neck convolution unit; in the figure, the neck first downsampling module includes a downsampling first convolution residual module and a first downsampling layer adapted to be connected to the downsampling first convolution residual module, the neck second downsampling module includes a downsampling second convolution residual module and a second downsampling layer adapted to be connected to the downsampling second convolution residual module, the first downsampling layer is connected to the downsampling second convolution residual module, and the output of the backbone P3 layer is loaded into the downsampling first convolution residual module. It can be seen that the neck first sampling module and the neck second downsampling module adopt the same structural form.

[0107] It should be noted that the output of the backbone P3 layer contains more information about the pin borders in the backbone network. Therefore, the output of the backbone P4 layer in the original YOLOV5 network can be omitted to simplify the network structure, thereby increasing the amount of computation during model training and reasoning without affecting the target recognition of pins. In addition, when the information output by the backbone P3 layer is selected, in order to make the feature map output by the backbone P3 layer compatible with the feature map output by the backbone P5 layer, Figure 1 To enter the output network, it is necessary to adjust the first two dimensions of the feature map output by the backbone P3 layer, that is, it is necessary to downsample and adjust the dimension of the output feature map of the backbone P3 layer through the neck first downsampling module and the neck second downsampling module. After that, it is output to the output network through the neck output connection residual module.

[0108] also, Figure 2 Also shown is an embodiment in which the neck convolution unit includes a neck convolution layer, and the neck convolution layer and the second downsampling layer are both connected to the neck output convolution residual connection module. The output network includes an output convolution layer. Specifically, the output convolution layer can be a convolution layer with an output channel of 5 and a stride of 1. The number of output channels of the output convolution layer represents the pin prior prediction box information output by the output network. Each prior prediction box information contains the position information of a pin relative to the prior box. In specific implementation, the prior prediction box information includes the horizontal x-direction offset, the horizontal y-direction, the pixel width w coefficient, the pixel height h coefficient, and the confidence conf.

[0109] It should be noted that when the pin image of the power device module is subjected to target recognition, the pin pattern of the power device module is processed by the backbone network, the neck network and the output network in sequence, and the pin priori box prediction information of the pin is output by the output network. From the above description, it can be seen that the pin priori box prediction information of the pin includes the horizontal x direction offset, the horizontal y direction, the pixel width w coefficient, the pixel height h coefficient and the confidence conf.

[0110] After obtaining the stitch priori frame prediction information of the stitch, in order to obtain the stitch identification information of the stitch, it is also necessary to perform post-processing commonly used in the technical field on the stitch priori frame prediction information, wherein the post-processing may at least include decoding processing and candidate frame processing, wherein the decoding processing mainly decodes the stitch priori frame prediction information and the position information of the priori frame, so that the stitch candidate frame can be generated after decoding.

[0111] It can be understood that the position information of the prior frame corresponds to the output of the output network. In one embodiment of the present invention, the output grid can be a preset 20*20 frame, and the center position coordinates x and y of each frame are:

[0112] x=640*(i / 20+0.5), (i=0, 1, 2, 3...19),

[0113] y=640*(i / 20+0.5), (i=0, 1, 2, 3...19),

[0114] At the same time, the width and height of each box are w=125 and h=200.

[0115] It should be noted that after obtaining the stitch prior frame prediction information and the position information of the prior frame, the method and process of decoding and generating the stitch candidate frame can be consistent with the existing ones, which are well known to those in the technical field and will not be repeated here. Candidate frame processing specifically refers to processing the stitch candidate frame generated above. The candidate frame processing can adopt the existing non-maximum suppression (NMS). Non-maximum suppression (NMS) is a post-processing step commonly used in target detection. It is used to remove redundant detection stitch frames and retain only the most likely detection results. Therefore, after the candidate frame processing, the stitch recognition information of the stitch can be obtained.

[0116] Specifically, the a priori frame refers to the width and height values ​​of the preset frame based on the size of the pins of the actual power device module in the image. In actual scenarios, the width and height of the pins of the power device module are approximately 125*200 pixels. Therefore, in order to correspond to the pin image of the power device module, the width and height of the a priori frame can be set to 125*200. The position information of the a priori frame used for the above decoding can be obtained based on the width size of the a priori frame. The method of decoding the pin a priori frame prediction information and the position information of the a priori frame, as well as the method of processing the candidate frame for non-maximum suppression, can be consistent with the existing ones, and the specific processing process will not be repeated here.

[0117] Figure 2An embodiment of the backbone network is also shown in the figure, in which CSPDarknet is the backbone network, Neck is the neck network, and Head is the output network; the backbone network includes a backbone input layer (Input), a backbone focus layer (Focus), a first Conv2D_BIN_SiLU layer, a second Conv2D_BIN_SiLU layer, a first backbone convolution residual layer (Csplayer), a third Conv2D_BIN_SiLU layer, a second backbone convolution residual layer, a fourth Conv2D_BIN_SiLU layer, a third backbone convolution residual layer, a fifth Conv2D_BIN_SiLU layer, an SPBottleneck layer, and a fourth backbone convolution residual layer, wherein the second backbone convolution residual layer is connected to the down-sampled first convolution residual module, that is, the output of the second backbone convolution residual layer is used as the output of the backbone P3 layer, and the fourth backbone convolution residual layer is connected to the neck convolution layer, that is, the output of the fourth backbone convolution residual layer is used as the output of the backbone P5 layer.

[0118] Figure 2 The corresponding parameter settings in the backbone network are also shown. Figure 2 In the figure, the parameters of the backbone input layer (640, 640, 3) are the size of the pin image of the input power device module and the size of the following training sample is 640*640; the parameters of the backbone focus layer are (320, 320, 12), the parameters of the first Conv2D_BIN_SiLU layer are (320, 320, 64), the parameters of the second Conv2D_BIN_SiLU layer are (160, 160, 128), the parameters of the first backbone convolution residual layer are (160, 160, 128), and the parameters of the third Conv2D_BIN_Si The parameters of the LU layer are (80, 80, 256), the parameters of the second backbone convolution residual layer are (80, 80, 256), the parameters of the fourth Conv2D_BIN_SiLU layer are (40, 40, 512), the parameters of the third backbone convolution residual layer are (40, 40, 512), the parameters of the fifth Conv2D_BIN_SiLU layer are (20, 20, 1024), the parameters of the SPBottleneck layer are (20, 20, 1024), and the parameters of the fourth backbone convolution residual layer are (20, 20, 1024).

[0119] It should be noted that Figure 2And the above parameters are the parameters of the feature map obtained after the data input to the current layer passes through the current layer. For example, for the backbone input layer, the size of the pin image of the power device module is 640*640, and there are three channels, so the pin image input size is (640, 640, 3), that is, "3" here is the number of channels. After passing through the backbone focus layer, it becomes (320, 320, 12), (320, 320, 12), which is the size of the feature map after passing through the backbone focus layer. The size of the feature map is consistent with the existing one. For other parameters, please refer to the description here, which is well known to those in the field of technology and will not be listed here one by one.

[0120] The above stitch target recognition model can be constructed by the following method. In one embodiment of the present invention, when constructing the stitch target recognition model, the construction method includes:

[0121] A basic model for pin target recognition based on the YOLOV5 model and a training data set for model training of the basic model for pin target recognition are constructed, wherein:

[0122] The training data set includes a plurality of training samples, the training samples have the same pixel size, each training sample includes only one pin of the power device module, and each training sample is marked with the position information of the pin;

[0123] The stitch target recognition basic model is trained using the training data set. After the training of the stitch target recognition basic model reaches a target state, a stitch target recognition model is generated based on the stitch target recognition basic model that has reached the target training state.

[0124] Specifically, the pin target recognition basic model can be constructed based on the above-mentioned improved YOLOV5 network. The architecture of the constructed pin target recognition basic model can refer to the description of the above-mentioned pin target recognition model, that is, the pin target recognition basic model also includes a backbone network, a neck network and an output network. The backbone network, the neck network and the output network can be referred to Figure 2 As well as the above description, no further repetition is given here.

[0125] After the stitch target recognition basic model is constructed, the stitch target recognition basic model is trained using the training data set, and after the training reaches the target state, the stitch target recognition basic model that reaches the target training state can be configured as the stitch target recognition model of the present invention.

[0126] The training data set includes several training samples, wherein the number of training samples in the training data set can be selected according to actual needs. The training samples generally have the same pixel size, such as the pixel size of the training sample can be 640*640; each training sample also includes only one pin of a power device module, wherein the type of the power device module is the same as the type of the power device module to be measured for the pin size, such as a power device module obtained by the same packaging method for IGBT devices.

[0127] In one embodiment of the present invention, for any training sample, preparing the training sample includes:

[0128] Capturing an image of the power device module to obtain a basic image of the pins of the power device module, wherein the basic image of the pins of the power device module includes at least one pin;

[0129] The basic image of the power device pins is annotated, and the pins are extracted after the annotation to generate a power device module pin training image after the extraction, wherein:

[0130] The power device module pin training image includes only one pin, and the pin is located at the center of the power device module pin training image, and the size of the power device module pin training image is the same as the preset pixel size;

[0131] The pin annotation information of the generated pins is annotated on the pin training image of the power device module;

[0132] Generate training samples based on the power device module pin training images.

[0133] Specifically, when capturing images of a power device module, existing commonly used methods may be used, such as using the camera for calibrating the shooting position as mentioned above, and placing the camera at the camera calibration position; of course, the power device module corresponds to the placement state and placement position of the power device module during the above-mentioned shooting position calibration. Thereafter, the pins of the power device module may be photographed, and a basic pin image of the power device module may be obtained after each photographing. Generally, the basic pin image of the power device module preferably includes only one pin, and the number of pins in the basic pin image of the power device module may be adjusted and selected by controlling the shooting area, shooting angle, etc. of the telecentric lens camera.

[0134] The obtained basic pin image of the power device module is annotated using a commonly used annotation tool in the technical field. For example, the labelme tool can be used to annotate the basic pin image of the power device module. After the annotation, the position information of the pin is obtained. Figure 3 An embodiment of labeling the basic pin image of the power device module and forming a training sample is shown in FIG. Figure 3In the figure, the rectangular box is the pin box marked by the labelme tool. The pin is completely inside the pin box, and the center of the pin corresponds to the center of the pin box. At the same time, the width and height of the rectangular box are also marked by the labelme tool. Figure 3 In the figure, the silver area is the pin, the center of the frame (x, y) is the center of the pin frame, w is the labeled width of the pin frame, and h is the labeled length of the pin frame. That is, the pin labeling information includes the center position of the pin frame, the labeled width of the pin frame, and the labeled length of the pin frame. After labeling with the labelme tool, the center of the pin frame, the labeled width w, and the labeled length h can be directly obtained, where the labeled width w is the pixel width of the pin, and the labeled length h is the pixel height of the pin.

[0135] Since the captured image also includes other areas outside the pins, after marking, the pin area is extracted from the basic pin image of the power device module to generate a power device module pin training image after extraction.

[0136] When extracting the stitch area, the stitch training image of the power device module mainly includes only one stitch, and the stitch is located in the central area of ​​the stitch training image of the power device module. The size of the stitch training image of the power device module is consistent with the preset pixel size. The size of the stitch training image of the power device module can be 640*640. Figure 3 That is, an embodiment of a power device module pin training image, Figure 3 As can be seen from the above description, in the power device module pin training image, there is the above-mentioned pin frame, and the center of the pin frame, the marked width w, and the standard height h are marked.

[0137] In a specific implementation, a training sample can be generated based on a power device module pin training image, that is, a training sample can be obtained using a power device module pin training image. In addition, in order to ensure that the position information of the pins in the training sample is consistent with the scene, in an embodiment of the present invention, when generating a training sample based on a power device module pin training image, it also includes:

[0138] Performing image data enhancement processing on the power device module pin training image, wherein the image data enhancement processing on the power device module pin training image includes sequentially performing HSV brightness contrast transformation processing and random area Gaussian blur processing;

[0139] When performing HSV brightness contrast transformation processing, the power device module pin training image is converted into an HSV pin training image, and after the HSV brightness contrast transformation processing, an HSV brightness transformation processed pin training image is generated;

[0140] For the converted HSV stitch training image, add an enhanced first random factor to each pixel of the V channel of the HSV stitch training image, and randomly multiply each pixel of the S channel of the HSV stitch training image by an enhanced second random factor;

[0141] When performing random area Gaussian blur processing, a Gaussian window is randomly generated for the stitch training image after the HSV brightness transformation processing, and the generated Gaussian window is used to perform random Gaussian blur processing on the stitch training image after the HSV brightness transformation processing, so as to generate training samples after the random Gaussian blur processing.

[0142] It can be understood that due to limitations such as the shooting environment, the brightness of the power device module pin training image may fluctuate due to environmental influences. In order to increase the generalization ability of the pin target recognition model, image data enhancement processing is required.

[0143] The power device module pin training image can be converted into an HSV image using commonly used technical means in the technical field. At this time, the HSV pin training image can be obtained after the HSV image conversion. The specific method and process of converting the HSV pin training image can be selected according to needs, and no example is given here. That is, when performing HSV brightness contrast transformation processing, the power device module pin training image of the RGB channel needs to be converted into an HSV pin training image first. After data enhancement processing, it is converted back to the RGB channel.

[0144] It should be noted that, for the converted HSV stitch training image, including H channel, S channel and V channel, in one embodiment of the present invention, an enhanced first random factor is added to each pixel of the V channel of the HSV stitch training image, wherein the enhanced first random factor may be the product of the pixel value 50 and the random factor coefficient, and the random factor coefficient is 0-1; in specific implementation, the random factor coefficient of each pixel may be randomly taken, so that the enhanced first random factor added to each pixel of the V channel may be different.

[0145] Furthermore, each pixel of the S channel of the HSV stitch training image is multiplied by an enhanced second random factor, and the enhanced second random factor may be 0.8-1.2; the enhanced second random factor may be randomly selected, that is, the enhanced second random factor multiplied by each pixel may be the same or different.

[0146] After the HSV stitch training image is processed as above, the HSV brightness contrast transformation is performed, and at this time, the stitch training image after the HSV brightness transformation is obtained. Of course, other methods can also be used to perform the HSV brightness contrast transformation, so as to reduce the fluctuation of the image brightness affected by the environment.

[0147] After obtaining the stitch training image after HSV brightness transformation processing, the image is converted back to the training image of the RGB channel, that is, the stitch training image after HSV brightness transformation processing is an image of the RGB channel.

[0148] Further, the stitch training image after HSV brightness transformation is subjected to random area Gaussian blur processing, wherein Gaussian blur processing is to perform convolution operation processing on the stitch training image after HSV brightness transformation processing. When performing random area Gaussian blur processing, the size of the Gaussian frame can be set to 50*50~100*100, and the standard deviation σ can be set to a random number of 0.1~2. After setting the Gaussian frame and the standard deviation σ, the stitch training image after HSV brightness transformation processing can be subjected to Gaussian blur processing. The specific method and process of Gaussian blur processing can be consistent with the existing ones, and will not be repeated here.

[0149] In specific implementation, when the training samples are generated in the above manner, when the pin size on the power device module is measured, the pin image of the provided power device module also needs to be processed as above, such as using a telecentric lens camera to take a photo of the power device module, and extracting the pin area of ​​the image obtained by the photo in turn. Of course, after the pin area is extracted, the image enhancement processing step can be omitted. The specific method and process of extracting the pin area can refer to the above description. It can be seen from the above description that the size of the pin image of the power device module should generally be consistent with the size of the training sample, such as a pixel size of 640*640.

[0150] In one embodiment of the present invention, when training the basic model for pin target recognition, after each generation of training, the training loss after the current generation of training is calculated;

[0151] When the training loss does not decrease after the preset generation training, the model training of the pin target recognition basic model reaches the target training state;

[0152] When calculating the training loss, the training loss includes positioning loss and confidence loss.

[0153] It is understandable that after constructing the pin target recognition basic model and training data set, some training parameters need to be configured, such as the number of training epochs, optimizer and training loss of the model. The number of training epochs can be set to 3000. When the number of training epochs is set to 3000, the pin target recognition basic model is trained 3000 times using the training data set; the optimizer can select SGD (stochastic gradient descent); the training loss includes positioning loss and confidence loss.

[0154] For the target training state, in one embodiment of the present invention, the training loss may not decrease after a preset generation of training, and the preset generation of training may be 50 generations, wherein one generation of model training is to perform a complete model training on the stitch target recognition basic model using the training data set, which is specifically consistent with the existing.

[0155] For the training loss, we have:

[0156]

[0157] in, is the training loss, is the positioning loss, is the weight coefficient of positioning loss, is the confidence loss, is the weight coefficient of confidence loss.

[0158] In specific implementation, the weight coefficient of positioning loss , the weight coefficient of confidence loss The cumulative sum is 1. Generally, the weight coefficient of the positioning loss The weight coefficient of confidence loss Equal, such as taking the value of 0.5, of course, other values ​​can also be taken to meet the weight coefficient of positioning loss , the weight coefficient of confidence loss The cumulative sum of is 1, and the model training requirements for the pin target recognition basic model shall prevail.

[0159] In one embodiment of the present invention, the positioning loss , then:

[0160]

[0161] in, is the prediction box of the training sample, is the true box of the training sample, is the intersection-over-union ratio of the predicted box and the true box of the training sample, is the square of the distance between the center of the predicted box and the true box, It is the square of the longest diagonal length of the minimum enclosing rectangle of the predicted box and the true box; is the width of the prediction box, is the height of the prediction box, is the width of the real frame, is the height of the actual frame.

[0162] Specifically, the true box of the training sample That is, the pin box formed by the labelme tool in the training sample. When calculating the training loss, the real box of the training sample needs to be processed by anchor box matching to match the real box of the training sample to the output size of the output network of the pin target recognition basic model, and determine the specific prior box responsible for the prediction of the real box; among them, the prior box matching is implemented through an IoU-based strategy. This process is usually completed in the loss calculation part, mainly including IoU calculation and matching judgment steps. The method and process of prior box matching can be consistent with the existing ones.

[0163] It can be understood that by matching the real frame to the network output size, the square of the distance between the center of the predicted frame and the real frame can be calculated using the commonly used technical means in the technical field. And the square of the longest diagonal length of the minimum enclosing rectangle of the predicted box and the true box , which can also calculate the above positioning loss .

[0164] In one embodiment of the present invention, for the confidence loss , then:

[0165]

[0166] in, is the total number of prior frames, To determine whether the current prior frame is responsible for the prediction of the real frame after matching the prior frame, if it is, it is 1, otherwise it is 0; To determine whether the current prior frame is responsible for the prediction of the real frame after matching the prior frame, if it is responsible, it is 0, otherwise it is 1; BCE is the binary cross entropy loss function, The confidence c predicted by the current prior box.

[0167] From the above description, it can be seen that after the a priori box matching process is adopted, it can be determined whether the current a priori box is responsible for the prediction of the real box. The confidence c of the a priori box prediction can be directly output by the pin target recognition basic model. Therefore, the above confidence loss can be calculated .

[0168] Based on the binary cross entropy loss function, for , Then we have:

[0169]

[0170] It is understandable that the training loss is calculated on both the model training set and the model test set, but training stops based on the training loss calculated on the model training set.

[0171] From the above description, we can see that the positioning loss , which measures the difference between the pin candidate box and the true object bounding box. By optimizing the localization loss , which can improve the accuracy of the pin target recognition model in predicting the pin position and size.

[0172] Confidence loss :It can evaluate the confidence of the pin target recognition model on the existence of the target in the pin candidate box. By optimizing the confidence loss , which can improve the reliability and accuracy of the pin target recognition model in detecting pin marks, thereby reducing missed detections and false detections and improving the overall detection performance.

[0173] By positioning loss And the confidence loss The training loss , which can make the trained pin target recognition model perform well in precision and recall, and can ensure the accuracy of prediction results while achieving efficient detection.

[0174] In summary, a pin size measurement device suitable for a power device module can be obtained, including a pin size measurement terminal, wherein the pin size measurement terminal uses the above-mentioned measurement method to determine the physical size of the pins of the power device module.

[0175] Specifically, the pin size measurement terminal can be an existing commonly used computer device, and the type of the pin size measurement terminal can be selected according to actual needs, so as to meet the actual application scenario requirements. The situation of the power device module can refer to the above description. For the pins of the power device module, the above measurement method can be used to determine the physical size of the pins. The specific method and process of determining the physical size of the pins can refer to the above description, which will not be repeated here.

Claims

1. A method for measuring the pin size of a power device module, characterized in that: The method for measuring the pin size on the power device module includes: For a power device module to be measured for pin size, obtaining a pin image of the power device module, and loading the pin image into a pre-built pin target recognition model, wherein the pin image of the power device module includes a pin of the power device module; The pin target recognition model performs target recognition on the pins in the pin image to obtain pin priori box prediction information of the pins on the power device module after target recognition; Post-processing the stitch priori frame prediction information to generate stitch identification information of the stitch after post-processing, wherein the stitch identification information includes a pixel width of the stitch and a pixel height of the stitch; Based on the pin identification information of the pin, the physical size of the pin is calculated, wherein the physical size of the pin includes the physical width of the pin in the vertical projection direction and the physical height of the pin in the vertical projection direction; The vertical projection direction is the projection direction from the top of the pin to the bottom of the pin, the bottom of the pin is the end in contact with the power device module, and the top of the pin is the end of the pin away from the power device module; The stitch target recognition model is constructed based on the YOLOV5 model. The stitch target recognition model includes a backbone network, a neck network and an output network connected in sequence, wherein: The neck network includes a neck output connection convolution residual module, a neck downsampling unit adaptively connected to the backbone P3 layer in the backbone network, and a neck convolution unit adaptively connected to the backbone P5 layer in the backbone network; The neck downsampling unit and the neck convolution unit are both connected to the neck output connection convolution residual module, and are adaptively connected to the output network through the neck output connection residual module; The output network includes at least one output convolutional layer; For the pin image of the power device module, the pin target recognition model outputs the pin recognition information through the output convolution layer; The neck downsampling unit comprises a neck first downsampling module and a neck second downsampling module connected in sequence, wherein the output of the backbone P3 layer in the backbone network is loaded into the neck output connection residual module via the neck first sampling module and the neck second downsampling module; The neck convolution unit includes at least one neck convolution layer, and the output of the backbone P5 layer in the backbone network is loaded into the neck output connection residual module via the neck convolution layer.

2. The pin size measurement method for a power device module according to claim 1, characterized in that: When calculating the physical size of the pin based on the pin identification information of the pin, we have: Among them, w t is the physical width of the pin, h t is the physical height of the pin, w p is the pixel width of the pin, h p is the pixel height of the pin, L pixel is the physical size of a pixel after calibration.

3. The pin size measurement method for a power device module according to claim 1, characterized in that: When building a pin target recognition model, the construction method includes: A basic model for pin target recognition based on the YOLOV5 model and a training data set for model training of the basic model for pin target recognition are constructed, wherein: The training data set includes a plurality of training samples, the training samples have the same pixel size, each training sample includes only one pin of the power device module, and each training sample is marked with the position information of the pin; The stitch target recognition basic model is trained using the training data set. After the training of the stitch target recognition basic model reaches a target state, a stitch target recognition model is generated based on the stitch target recognition basic model that has reached the target training state.

4. The pin size measurement method for a power device module according to claim 3, characterized in that: When training the basic model for pin target recognition, after each generation of training, the training loss after the current generation of training is calculated; When the training loss does not decrease after the preset generation training, the model training of the pin target recognition basic model reaches the target training state; When calculating the training loss, the training loss includes positioning loss and confidence loss.

5. The pin size measurement method for a power device module according to claim 4, wherein The training loss is: Loss=λ bbox *LLocal+λ obj *Lobj Among them, Loss is the training loss, LLocal is the positioning loss, λ bbox is the weight coefficient of positioning loss, Lobj is the confidence loss, λ obj is the weight coefficient of confidence loss; For the positioning loss LLocal, we have: Where B is the predicted box of the training sample, Bgt is the real box of the training sample, IoU(B,Bgt) is the intersection and union ratio of the predicted box and the real box of the training sample, ρ 2 (B, Bgt) is the square of the distance between the center of the predicted box and the true box, c 2 is the square of the longest diagonal length of the minimum enclosing rectangle of the predicted box and the true box; w B is the width of the prediction box, h B is the height of the prediction box, w Bgt is the width of the real box, h Bgt is the height of the real frame; For the confidence loss Lobj, we have: Among them, S is the total number of prior frames, To determine whether the current prior frame is responsible for the prediction of the real frame after matching the prior frame, if it is responsible, it is 1, otherwise it is 0; To determine whether the current prior frame is responsible for the prediction of the real frame after matching the prior frame, if it is responsible, it is 0, otherwise it is 1; BCE is the binary cross entropy loss function, The confidence c predicted by the current prior box.

6. The pin size measurement method for a power device module according to any one of claims 3 to 5, characterized in that: For any training sample, when preparing the training sample, it includes: Capturing an image of the power device module to obtain a basic image of the pins of the power device module, wherein the basic image of the pins of the power device module includes at least one pin; The basic image of the power device pins is annotated, and the pins are extracted after the annotation to generate a power device module pin training image after the extraction, wherein: The power device module pin training image includes only one pin, and the pin is located at the center of the power device module pin training image, and the size of the power device module pin training image is the same as the preset pixel size; The pin annotation information of the generated pins is annotated on the pin training image of the power device module; Generate training samples based on the power device module pin training images.

7. The pin size measurement method for a power device module according to claim 6, characterized in that: When generating training samples based on the power device module pin training images, it also includes: Performing image enhancement processing on the power device module pin training image, wherein the image enhancement processing on the power device module pin training image includes sequentially performing HSV brightness contrast transformation and random area Gaussian blur; When performing HSV brightness contrast transformation processing, the power device module pin training image is converted into an HSV pin training image, and after the HSV brightness contrast transformation processing, an HSV brightness transformation processed pin training image is generated; For the converted HSV stitch training image, add an enhanced first random factor to each pixel of the V channel of the HSV stitch training image, and randomly multiply each pixel of the S channel of the HSV stitch training image by an enhanced second random factor; When performing random area Gaussian blur processing, a Gaussian window is randomly generated for the stitch training image after the HSV brightness transformation processing, and the generated Gaussian window is used to perform random Gaussian blur processing on the stitch training image after the HSV brightness transformation processing, so as to generate training samples after the random Gaussian blur processing.

8. A pin size measuring device suitable for a power device module, characterized in that: It comprises a pin size measurement terminal, wherein, for the pins of the power device module, the pin size measurement terminal adopts the measurement method described in any one of claims 1 to 7 to determine the physical size of the pins.

Images