Component detection method, system and flying probe test equipment based on machine vision

Abstract

The present application relates to a component detection method, system and flying probe test equipment based on machine vision, belonging to the technical field of circuit board testing. The component detection method based on machine vision includes obtaining a standard image sequence of a preset standard circuit board and a state image sequence of a circuit board to be detected to construct a component state detection model; inputting the standard image sequence and the state image sequence into the component state detection model to obtain template features and features to be detected; and generating a state template based on the template features, and generating a recognition result based on the features to be detected; calibrating the standard position frame according to the recognition position frame in the recognition result within a preset time window, obtaining an optimized position frame and generating an optimized state template based on the optimized position frame; comparing the recognition result with the optimized state template to generate a component detection result. The present application has the effect of facilitating the testing of the state changes of components on a circuit board.

Description

Technical Field

[0001] The present invention relates to the technical field of circuit board testing, and in particular to a component detection method and system based on machine vision and a flying probe testing device. Background Art

[0002] Machine vision component inspection is the process of using computer vision technology to achieve automated inspection.

[0003] At present, in flying probe testing, the layout of the circuit board and the location of components are usually known in advance, so that the probe can be controlled to move along the preset path and contact the preset test points (including component pins, solder joints, etc.) according to the preset circuit board structure, size and shape. When the probe contacts the test point, it is connected to the driver and sensor through a multiplex transmission system to measure various physical parameters, such as voltage, current value, temperature, etc. However, for some components on the circuit board whose status can change with the movement of the probe, it is impossible to obtain them by physical quantity measurement. It can only be judged whether the status changes of these components are normal by manual observation, and it is impossible to realize automatic detection of the component status. Therefore, how to test the status changes of components on the circuit board is a problem that needs to be solved urgently. Summary of the invention

[0004] In order to facilitate the testing of status changes of components on a circuit board, the present application provides a component detection method, system and flying probe test equipment based on machine vision.

[0005] In the first aspect, the present application provides a component detection method based on machine vision, which adopts the following technical solution:

[0006] A component detection method based on machine vision, comprising:

[0007] Acquire a preset standard image sequence of a standard circuit board;

[0008] Acquire a state image sequence of a circuit board to be inspected; wherein the circuit board to be inspected is of the same type as a standard circuit board;

[0009] Construct component status detection model;

[0010] The standard image sequence and the state image sequence are input into the component state detection model, and the component state detection model extracts features from the standard image sequence and the state image sequence to obtain template features and features to be detected; and generates a state template based on the template features, and generates a recognition result based on the features to be detected; wherein the state template includes a standard position frame of each component and a standard state attribute corresponding to a preset timestamp; and the recognition result includes a recognition position frame of each component, state information, and a timestamp of the state change moment;

[0011] According to the recognition position frame in the recognition result within the preset time window, the standard position frame is calibrated to obtain the optimized position frame and an optimized state template is generated based on the optimized position frame;

[0012] The recognition result is compared with the optimized state template to generate a component detection result.

[0013] Optionally, the component state detection model includes a U-net module, a discriminator module and a timing control module; and constructing the component state detection model specifically includes:

[0014] Produce a training data set including multiple groups of sample images, annotated images obtained by annotating the sample images, and time stamp labels; wherein the annotated images include a position frame of a set color value at a component; and the time stamp label is the time when the color value changes;

[0015] Use the U-net module to segment and color each group of sample images to generate the predicted image of the group;

[0016] analyzing the timestamp information of the group based on the predicted image using a timing control module;

[0017] Using a discriminator to verify the authenticity of the predicted image and the sample image corresponding thereto, respectively, to generate a first probability that the labeled image is a true image and a second probability that the predicted image is a true image;

[0018] Constructing a discriminant loss function of the discriminator according to the first probability and the second probability;

[0019] According to the pixel difference between the predicted image and the annotated image, a pixel loss index is constructed, and the generation loss function of the U-net module is constructed based on the pixel loss index and the discriminant loss function;

[0020] Construct a time series loss function based on timestamp labels and timestamp information;

[0021] Iterative training is performed according to the timing loss function, the discrimination loss function and the generation loss function to obtain the component status detection model.

[0022] Optionally, the preparing of a training data set including a plurality of groups of sample images, annotated images obtained by annotating the sample images, and timestamp labels specifically includes:

[0023] Acquire a plurality of sample images obtained by photographing a plurality of circuit boards respectively;

[0024] Annotating the position frames and state attributes of components in a plurality of sample images, and setting the color value of the position frame according to the state attribute to obtain annotated images;

[0025] The labeled images are grouped based on the circuit board type, and the sample images in each group are randomly sorted to obtain a labeled image sequence;

[0026] Recording a sequence value of annotated images in the annotated image sequence where the color value changes, and calculating a timestamp tag in the annotated image based on the sequence value;

[0027] The training data set is obtained based on each group of sample images, annotated images and timestamp labels.

[0028] Optionally, calibrating the standard position frame according to the recognition position frame in the recognition result within the preset time window to obtain the optimized position frame and generating the optimized state template based on the optimized position frame specifically includes:

[0029] Obtain the center coordinates and length and width information of each recognition position frame within the preset time window, and calculate the center average value of the center coordinates and the length and width average value of the length and width information within the preset time window;

[0030] It is determined whether the center average value and the length and width information exceed the corresponding thresholds. If so, the standard position frame is calibrated according to the center average value and the length and width information to obtain the optimized position frame; if not, the standard position frame is used as the optimized position frame.

[0031] Optionally, comparing the recognition result with the optimized state template to generate a component detection result specifically includes:

[0032] Based on the preset timestamp of the optimized state template, the recognition results of adjacent frames are intercepted as the target results;

[0033] Determine whether the recognition position frame and the standard position frame in the target result match;

[0034] If they do not match, a prompt is generated indicating that the circuit board to be tested does not match the template;

[0035] If they match, determine whether the status information in the target result is consistent with the standard status attributes;

[0036] If they are consistent, determine whether the image corresponding to the current target result and the image pointed to by the timestamp in the current target result are the same frame;

[0037] If so, the generated component detection result is qualified; if not, the generated component detection result is unqualified.

[0038] Optionally, the determining whether the recognition position box in the target result matches the standard position box specifically includes:

[0039] Obtain the center coordinates of all recognized positions in the target result and the center coordinates of all standard position boxes in the standard template;

[0040] The center coordinates are respectively compared with the preset reference points to generate the offset angle and distance of each identification position frame and the standard position frame relative to the preset reference points;

[0041] Compare the offset angle and distance of each recognition position frame with the standard position frame to match the recognition position frame with the standard position frame one by one;

[0042] The intersection-and-union ratio of the matched recognition position box and the bounding box of the standard position is calculated, and based on the intersection-and-union ratio of the bounding boxes, it is determined whether the recognition position box and the standard position box in the target result match.

[0043] Optionally, the component includes one or more of a light emitting diode, a fuse, a relay, a button, a seven-segment digital tube and a liquid crystal display.

[0044] In a second aspect, the present application provides a component detection system based on machine vision, which adopts the following technical solution:

[0045] A component detection system based on machine vision, comprising:

[0046] A template acquisition module is used to obtain a standard image sequence of a preset standard circuit board;

[0047] A data acquisition module, used to obtain a state image sequence of a circuit board to be inspected; wherein the circuit board to be inspected is of the same type as a standard circuit board;

[0048] A model building module, used to build a component status detection model;

[0049] A model output module is used to input the standard image sequence and the state image sequence into the component state detection model, and the component state detection model extracts features from the standard image sequence and the state image sequence to obtain template features and features to be detected; and generates a state template based on the template features, and generates a recognition result based on the features to be detected; wherein the state template includes a standard position frame of each component and a standard state attribute corresponding to a preset timestamp; and the recognition result includes a recognition position frame of each component, state information, and a timestamp of the state change moment;

[0050] An optimization module, configured to calibrate the standard position frame according to the recognition position frame in the recognition result within a preset time window, obtain an optimized position frame, and generate an optimized state template based on the optimized position frame;

[0051] The inspection result generating module is used to compare the recognition result with the optimized state template to generate the component inspection result.

[0052] In a third aspect, the present application provides a flying probe test device, which adopts the following technical solution:

[0053] A flying probe test device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program according to any one of the above methods.

[0054] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:

[0055] A computer-readable storage medium includes a computer program that can be loaded by a processor and executed in any of the above methods.

[0056] In summary, this application includes the following beneficial technical effects:

[0057] First, a standard image sequence of a preset standard circuit board is obtained to record the state of the standard circuit board at different time points. At the same time, a state image sequence of the circuit board to be detected is obtained to reflect the current actual situation of the circuit board to be detected. A component state detection model is constructed to automatically extract features from the input image sequence, thereby obtaining template features and features to be detected. A state template and recognition results are generated based on the template features and features to be detected. The state template contains the standard position frame of each component and the standard state attributes of the corresponding timestamp, while the recognition result contains the recognition position frame, state information and the timestamp of the state change moment of each component. Then, according to the recognition position frame in the recognition result within the preset time window, the standard position frame is calibrated to optimize the accuracy of the position frame and improve the accuracy of subsequent comparison. Finally, the recognition result is compared with the optimized state template to generate the component detection result, and because the method of image sequence analysis is combined, the state change of the component can be accurately captured, thereby achieving the effect of detecting the state change of the component on the circuit board. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flow chart of a component detection method based on machine vision according to one embodiment of the present application.

[0059] Figure 2 This is a flow chart of a method for training a component status detection model according to one embodiment of the present application.

[0060] Figure 3 This is a flow chart of a method for producing a training set according to one embodiment of the present application.

[0061] Figure 4 This is a flow chart of a method for optimizing position box generation according to one embodiment of the present application.

[0062] Figure 5 This is a flow chart of a method for generating component detection results according to one embodiment of the present application.

[0063] Figure 6 It is a block diagram of a component detection system based on machine vision according to one embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0065] The present application embodiment discloses a component detection method based on machine vision. Figure 1 , a component detection method based on machine vision, comprising:

[0066] Step S101: obtaining a preset standard image sequence of a standard circuit board;

[0067] Step S102: Acquire a state image sequence of the circuit board to be inspected;

[0068] The types of the circuit board to be tested and the standard circuit board are consistent. It should be understood that when a flying probe tester tests a circuit board, it is usually necessary to test the circuit boards generated in batches. Therefore, the types of the circuit board to be tested and the standard circuit board are consistent, which means that the circuit boards have the same appearance and function and are produced according to the same circuit layout and manufacturing process.

[0069] Specifically, the standard image sequence and the state image sequence may be each frame in a video, or may be composed of partial images extracted from a video and arranged in sequence.

[0070] Step S103: constructing a component status detection model;

[0071] Step S104: inputting the standard image sequence and the state image sequence into the component state detection model, the component state detection model extracts features from the standard image sequence and the state image sequence to obtain template features and features to be detected; and generates a state template based on the template features, and generates a recognition result based on the features to be detected;

[0072] The state template includes a standard position frame of each component and a standard state attribute corresponding to a preset timestamp; the recognition result includes an identification position frame, state information, and a timestamp of the state change of each component. The identification position frame and state information are helpful to identify the position and state information of the component. The timestamp information is helpful to track and analyze the state change process of the component.

[0073] The component state detection model includes a U-net module, a discriminator module and a timing control module connected in sequence.

[0074] Specifically, the U-Net module is a deep learning network model, which consists of an encoder (downsampling) and a decoder (upsampling). The encoder part extracts the features of the image layer by layer, while the decoder part is responsible for restoring the detailed information of the image to achieve accurate extraction of image features.

[0075] Step S105: calibrating the standard position frame according to the recognition position frame in the recognition result within the preset time window to obtain the optimized position frame and generating an optimized state template based on the optimized position frame;

[0076] It should be noted that the recognition results within the preset time window need to be recognition results of circuit boards with qualified component detection results. For example, in the initial stage of circuit board detection, the recognition results within the preset time window can be entered by manual proofreading. After the component detection results are generated in the subsequent step S106, the corresponding recognition results can be directly screened out from the circuit boards with qualified component detection results as the recognition results within the preset time window.

[0077] Step S106: Compare the recognition result with the optimized state template to generate a component detection result.

[0078] The components include one or more of a light emitting diode, a fuse, a relay, a button, a seven-segment digital tube and a liquid crystal display.

[0079] In the above implementation, first, a standard image sequence of a preset standard circuit board is obtained to record the state of the standard circuit board at different time points. At the same time, a state image sequence of the circuit board to be detected is obtained to reflect the current actual situation of the circuit board to be detected. A component state detection model is constructed to automatically extract features from the input image sequence, thereby obtaining template features and features to be detected. A state template and a recognition result are generated based on the template features and features to be detected. The state template includes a standard position frame of each component and a standard state attribute corresponding to a timestamp, and the recognition result includes an identification position frame, state information, and a timestamp of the state change moment of each component. Then, according to the identification position frame in the recognition result within the preset time window, the standard position frame is calibrated to optimize the accuracy of the position frame and improve the accuracy of subsequent comparisons. Finally, the recognition result is compared with the optimized state template to generate a component detection result, and because the method of analyzing the image sequence is combined, the state change of the component can be accurately captured, thereby achieving the effect of detecting the state change of the component on the circuit board.

[0080] Reference Figure 2 As an implementation of step S103, step S103 specifically includes:

[0081] Step S1031: creating a training data set including a plurality of groups of sample images, annotated images obtained by annotating the sample images, and timestamp labels;

[0082] The annotated image includes a location box where the color value is set at the component; the timestamp label is the time when the color value changes;

[0083] Step S1032: Use the U-net module to segment and color each group of sample images to generate a predicted image for the group.

[0084] It should be understood that the U-net module segments and then fills the sample image with color, which helps to quickly obtain the represented state attributes based on the color value in the box.

[0085] Step S1033: Utilize the timing control module to analyze the timestamp information of the group based on the predicted image.

[0086] It should be understood that the timing control module records the coloring part of each sample image in each group of sample images by the U-net module and analyzes it in the time dimension to output the timestamp information of the time when the color value changes. The change of color value indicates that the state attribute has changed, that is, the timestamp information represents the time when the state attribute changes.

[0087] Step S1034: using the discriminator to respectively verify the authenticity of the predicted image and the sample image corresponding thereto, so as to generate a first probability that the annotated image is a true image and a second probability that the predicted image is a true image.

[0088] As an example, the discriminator is a Markov discriminator (PatchGAN), which is a discriminator of a convolutional neural network-based generative adversarial network (GAN) and is used to discriminate whether the predicted image is generated by the generator.

[0089] Step S1035: construct a discriminant loss function of the discriminator according to the first probability and the second probability.

[0090] Specifically, the discriminant loss function of the discriminator is:

[0091] Where: is the probability that the discriminator considers the labeled image to be a true image; is the probability that the discriminator considers the predicted image to be the true image.

[0092] Step S1036: constructing a pixel loss index according to the pixel difference between the predicted image and the annotated image, and constructing a generation loss function of the U-net module based on the pixel loss index and the discriminant loss function;

[0093] Step S1037: construct a timing loss function according to the timestamp label and timestamp information.

[0094] Step S1038: Perform iterative training according to the timing loss function, the discrimination loss function and the generation loss function to obtain a component state detection model.

[0095] In the above implementation, first, the U-net module is used to accurately segment the components on the circuit board by learning the features in the sample image, and a position frame with a set color value is generated for each component, thereby realizing the segmentation of the image; then, the timing control module analyzes the specific moment of the component state change by predicting the component state information in the image. The introduction of the timing control module enables not only the static state of the component to be identified, but also its dynamic change process to be tracked. The predicted image and the sample image are verified for authenticity by the discriminator, and the difference between the two is compared to generate a probability value representing the authenticity of the predicted image. This helps the model to continuously optimize itself during the training process and improve the accuracy of the prediction. During the training process, the optimization of the model is guided by a variety of loss functions. After iterative training, a component state detection model that can quickly process a circuit board image sequence, accurately identify the position and state information of the component, and track its dynamic change process is obtained.

[0096] Reference Figure 3 As an implementation of step S1031, step S1031 specifically includes:

[0097] Step S10311: Acquire multiple sample images obtained by photographing multiple circuit boards respectively.

[0098] It should be understood that the photographing of each circuit board should be carried out in conjunction with the corresponding test process of the flying probe test equipment, so that the sample image can represent the entire process of component transformation during the test of the flying probe test equipment.

[0099] Step S10312: marking the position boxes and state attributes of the components in the plurality of sample images, and setting the color value of the position box according to the state attribute to obtain a marked image;

[0100] In this embodiment, each state attribute has two, and in other embodiments, the state attribute may be more than two. For example, when the component is in the first state attribute, the position box is set to the first color value; when the component is in the second state attribute, the position box is set to the second color value.

[0101] Step S10313: grouping the annotated images based on the circuit board type, and randomly sorting the sample images in each group to obtain an annotated image sequence;

[0102] It should be understood that in the actual detection process, the state of the component will change sequentially according to the test logic of the flying probe test equipment, but in the training process, the images can be disrupted into a random order so that the model can better capture the changes in the component state.

[0103] Step S10314: recording the sequence value of the annotated image in the annotated image sequence where the color value has changed, and calculating the time stamp tag in the annotated image based on the sequence value.

[0104] The sequence value is the sequence number of the annotated image in the standard image sequence. The current timestamp can be calculated based on the preset interval length of each annotated image frame.

[0105] Step S10315: Based on each group of sample images, annotated images and timestamp labels, the training data set is obtained.

[0106] Reference Figure 4 As an implementation of step S105, step S105 specifically includes:

[0107] Step S1051: obtaining the center coordinates and length and width information of each identification position frame within a preset time window, and calculating the center average value of the center coordinates and the length and width average value of the length and width information within the preset time window;

[0108] Step S1052: Determine whether the center average value and the length and width information exceed the corresponding thresholds. If so, execute step S1053; if not, execute step S1054.

[0109] Step S1053: calibrating the standard position frame according to the center average value and the length and width information to obtain the optimized position frame;

[0110] Step S1054: Using the standard position frame as the optimized position frame.

[0111] In the above implementation, by calculating the average value of the center coordinates and length and width information of multiple identification position frames within a preset time window, the position frame fluctuation caused by a single recognition error can be reduced. When the calculated center average value and length and width average value exceed the preset threshold, it means that there is a large difference or offset between the corresponding position frames. The standard position frame is adjusted based on the average position, so that the final optimized position frame is closer to the position and size of the real target, thereby improving the recognition accuracy.

[0112] Reference Figure 5 As an implementation of step S106, step S106 specifically includes:

[0113] Step S1061: based on the preset timestamp of the optimized state template, intercept the recognition results of adjacent frames as the target results;

[0114] Step S1062: Determine whether the recognition position box and the standard position box in the target result match; if not, proceed to step S1063; if they match, proceed to step S1064.

[0115] As an implementation of step S1062, step S1062 specifically includes: step S10621: obtaining the center coordinates of all the recognition positions in the target result and the center coordinates of all the standard position frames in the standard template; step S10622: comparing the center coordinates with the preset reference points respectively, and generating the offset angle and distance of each recognition position frame and the standard position frame relative to the preset reference point; step S10623: traversing and comparing the offset angles and distances of each recognition position frame and the standard position frame to match the recognition position frame and the standard position frame one-to-one; step S10624: calculating the intersection-and-union ratio of the bounding boxes of the matched recognition position frame and the standard position, and judging whether the recognition position frame and the standard position frame in the target result match based on the intersection-and-union ratio of the bounding boxes.

[0116] Step S1063: generating a prompt indicating that the circuit board to be tested does not match the template;

[0117] It should be understood that when the identification position box and the standard position box in the target result of the circuit board to be tested are inconsistent, it means that the component position in the circuit board to be tested is inconsistent with the component position of the standard circuit board. The most likely reason at this time is that the standard circuit board and the circuit board to be tested are of different types, which leads to the mismatch between the optimized state template and the current circuit board to be tested.

[0118] Step S1064: Determine whether the state information in the target result is consistent with the standard state attribute; if consistent, execute step S1065. If inconsistent, the state information in multiple target results is inconsistent, then execute step S1067;

[0119] Step S1065: Determine whether the image corresponding to the current target result and the image pointed to by the timestamp in the current target result are the same frame; if so, execute step S1066; if not, execute step S1067.

[0120] Step S1066: Generate a component inspection result that is qualified;

[0121] Step S1067: Generate a component inspection result that is unqualified.

[0122] It should be understood that each target result will contain timestamp information to characterize the moment when the actual state of the component changes. However, during the circuit board detection process, a specific image frame will be captured or selected as the target result for analysis based on a preset timestamp (i.e., the moment when the component state change is expected to be detected in the standard template). Therefore, if the image corresponding to the current target result is the same frame as the image pointed to by the timestamp in the current target result, it means that the circuit board to be tested is working in accordance with the preset logic, that is, the expected state change occurs at the preset timestamp, and the component detection result is qualified at this time, otherwise it is unqualified. This avoids the situation where the state attributes and identification temperature are correct, but the component state changes at the wrong time.

[0123] In the above implementation, by presetting the timestamp, the image data of the circuit board at the time point when the component state should change can be captured in real time, and used as the target result of the detection. Next, the position frame in the recognition result is matched with the standard position frame, and the position of the component is determined by calculating the offset angle and distance. At the same time, the state information in the recognition result is compared to see if it is consistent with the standard state attribute to determine whether the state of the component is normal. Finally, before generating the final detection result, it is verified whether the image corresponding to the current target result and the image pointed to by the timestamp in the current target result are the same frame, so as to ensure the consistency of the detection result with the actual image, and further improve the reliability of the system.

[0124] In addition, the embodiment of the present application discloses a component detection system based on machine vision. The component detection system based on machine vision can be applied to a flying probe test device, which is a schematic diagram of the architecture of a flying probe test device for implementing the above method provided by an embodiment of the present invention. In this embodiment, the flying probe test device may include a component detection system based on machine vision, a machine-readable storage medium, and a processor.

[0125] In this embodiment, the machine-readable storage medium and the processor may be located in the flying probe test device and the two are separately arranged. The machine-readable storage medium may also be independent of the flying probe test device and accessed by the processor. The component detection system based on machine vision may include multiple functional modules stored in the machine-readable storage medium, such as the various software functional modules included in the component detection system based on machine vision. When the processor executes the computer program corresponding to the software functional module in the component detection system based on machine vision, the component detection system based on machine vision provided by the aforementioned method embodiment is implemented.

[0126] In the present embodiment, the flying probe test equipment may include one or more processors. The processor may process information and / or data related to the service request to perform one or more functions described in the present invention. In some embodiments, the processor may include one or more processing engines (e.g., a single-core processor or a multi-core processor). Just to give an example, the processor may include one or more hardware processors, such as a central processing unit CPU, an application-specific integrated circuit ASIC, a dedicated instruction set processor ASIP, a graphics processor GPU, a physical operation processing unit PPU, a digital signal processor DSP, a field programmable gate array FPGA, a programmable logic device PLD, a controller, a microcontroller unit, a reduced instruction set computer RISC, a microprocessor, etc., or similar or any combination thereof.

[0127] The machine-readable storage medium can store data and / or instructions. In some embodiments, the machine-readable storage medium can store the obtained data or information. In some embodiments, the machine-readable storage medium can store data and / or instructions for the flying probe test device to execute or use, and the flying probe test device can implement the exemplary method described in this application by executing or using the data and / or instructions. In some embodiments, the machine-readable storage medium may include a large-capacity memory, a removable memory, a volatile read-write memory, a read-only memory ROM, etc. or any combination of similar or above examples. Exemplary large-capacity memory may include a disk, an optical disk, a solid-state disk, etc. Exemplary removable memory may include a flash drive, a floppy disk, an optical disk, a memory card, a compressed disk, a magnetic tape, etc. Exemplary volatile read-write memory may include a random access memory RAM. Exemplary random access memory may include dynamic RAM, double-rate synchronous dynamic RAM, static RAM, thyristor RAM, and zero-capacitor RAM, etc. Exemplary ROM may include a masked ROM, a programmable ROM, an erasable programmable ROM, an electronically erasable programmable ROM, a compressed disk ROM, and a digital universal disk ROM, etc.

[0128] The machine vision-based component detection system included in the flying probe test equipment may include one or more software function modules. The software function modules may be programs and instructions stored in the machine-readable storage medium, and these software function modules are used to implement the above-mentioned method when executed by the corresponding processor, for example, when executed by the processor of the drone, they are used to implement the method steps executed by the drone, or when executed by the flying probe test equipment, they are used to implement the method steps executed by the flying probe test equipment.

[0129] Refer to the detailed Figure 6 The present application embodiment discloses a component detection system based on machine vision. A component detection system based on machine vision includes:

[0130] In a second aspect, the present application provides a component detection system based on machine vision, which adopts the following technical solution:

[0131] A template acquisition module is used to obtain a standard image sequence of a preset standard circuit board;

[0132] A data acquisition module, used to obtain a state image sequence of a circuit board to be inspected; wherein the circuit board to be inspected is of the same type as a standard circuit board;

[0133] A model building module, used to build a component status detection model;

[0134] A model output module is used to input the standard image sequence and the state image sequence into the component state detection model, and the component state detection model extracts features from the standard image sequence and the state image sequence to obtain template features and features to be detected; and generates a state template based on the template features, and generates a recognition result based on the features to be detected; wherein the state template includes a standard position frame of each component and a standard state attribute corresponding to a preset timestamp; and the recognition result includes a recognition position frame of each component, state information, and a timestamp of the state change moment;

[0135] An optimization module, configured to calibrate the standard position frame according to the recognition position frame in the recognition result within a preset time window, obtain an optimized position frame, and generate an optimized state template based on the optimized position frame;

[0136] The inspection result generating module is used to compare the recognition result with the optimized state template to generate the component inspection result.

[0137] The component detection system based on machine vision provided in the present application can realize the above-mentioned component detection method based on machine vision, and the specific working process of the component detection system based on machine vision can refer to the corresponding process in the above-mentioned method embodiment.

[0138] It should be noted that in the above embodiments, the description of each embodiment has different emphases, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0139] The present invention also discloses a computer-readable storage medium, which includes a computer program that can be loaded by a processor and executed as in any of the above methods.

[0140] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0141] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0142] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any feature disclosed in this specification (including the abstract and drawings), unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

Claims

1. A component detection method based on machine vision, characterized in that: include: Acquire a preset standard image sequence of a standard circuit board; Acquire a state image sequence of a circuit board to be inspected; wherein the circuit board to be inspected is of the same type as a standard circuit board; Constructing a component state detection model; wherein the component state detection model includes a U-net module, a discriminator module and a timing control module; The constructing of the component status detection model specifically includes: Prepare a training data set including multiple groups of sample images, annotated images obtained by annotating them, and timestamp labels; wherein the annotated images include a position box of a set color value at a component; the timestamp label is the time when the color value changes; use a U-net module to segment and fill each group of sample images to generate a predicted image of the group; use a timing control module to analyze the timestamp information of the group based on the predicted image; use a discriminator to verify the authenticity of the predicted image and the corresponding sample image respectively to generate a first probability that the annotated image is a true image and a second probability that the predicted image is a true image; construct a discriminant loss function of the discriminator based on the first probability and the second probability; construct a pixel loss index based on the pixel difference between the predicted image and the annotated image, and construct a generation loss function of the U-net module based on the pixel loss index and the discriminant loss function; construct a timing loss function based on the timestamp label and the timestamp information; perform iterative training based on the timing loss function, the discriminant loss function, and the generation loss function to obtain a component state detection model; The standard image sequence and the state image sequence are input into the component state detection model, and the component state detection model extracts features from the standard image sequence and the state image sequence to obtain template features and features to be detected; and generates a state template based on the template features, and generates a recognition result based on the features to be detected; wherein the state template includes a standard position frame of each component and a standard state attribute corresponding to a preset timestamp; and the recognition result includes a recognition position frame of each component, state information, and a timestamp of the state change moment; According to the recognition position frame in the recognition result within the preset time window, the standard position frame is calibrated to obtain the optimized position frame and an optimized state template is generated based on the optimized position frame; The recognition result is compared with the optimized state template to generate a component detection result.

2. The method according to claim 1, characterized in that The step of preparing a training data set including a plurality of groups of sample images, annotated images obtained by annotating the sample images, and timestamp labels specifically includes: Acquire a plurality of sample images obtained by photographing a plurality of circuit boards respectively; Annotating the position frames and state attributes of components in a plurality of sample images, and setting the color value of the position frame according to the state attribute to obtain annotated images; The labeled images are grouped based on the circuit board type, and the sample images in each group are randomly sorted to obtain a labeled image sequence; Recording a sequence value of annotated images in the annotated image sequence where the color value changes, and calculating a timestamp tag in the annotated image based on the sequence value; The training data set is obtained based on each group of sample images, annotated images and timestamp labels.

3. The method according to claim 1, characterized in that The step of calibrating the standard position frame according to the recognition position frame in the recognition result within the preset time window to obtain the optimized position frame and generating the optimized state template based on the optimized position frame specifically includes: Obtain the center coordinates and length and width information of each recognition position frame within the preset time window, and calculate the center average value of the center coordinates and the length and width average value of the length and width information within the preset time window; It is determined whether the center average value and the length and width information exceed the corresponding thresholds. If so, the standard position frame is calibrated according to the center average value and the length and width information to obtain the optimized position frame; if not, the standard position frame is used as the optimized position frame.

4. The method according to claim 1, characterized in that: The comparing the recognition result with the optimized state template to generate the component detection result specifically includes: Based on the preset timestamp of the optimized state template, the recognition results of adjacent frames are intercepted as the target results; Determine whether the recognition position frame and the standard position frame in the target result match; If they do not match, a prompt is generated indicating that the circuit board to be tested does not match the template; If they match, determine whether the status information in the target result is consistent with the standard status attributes; If they are consistent, determine whether the image corresponding to the current target result and the image pointed to by the timestamp in the current target result are the same frame; If so, the generated component detection result is qualified; if not, the generated component detection result is unqualified.

5. The method according to claim 4, characterized in that The determining whether the recognition position frame in the target result matches the standard position frame specifically includes: Obtain the center coordinates of all recognized positions in the target result and the center coordinates of all standard position boxes in the standard template; The center coordinates are respectively compared with the preset reference points to generate the offset angle and distance of each identification position frame and the standard position frame relative to the preset reference points; Compare the offset angle and distance of each recognition position frame with the standard position frame to match the recognition position frame with the standard position frame one by one; The intersection-and-union ratio of the matched recognition position box and the bounding box of the standard position is calculated, and based on the intersection-and-union ratio of the bounding boxes, it is determined whether the recognition position box and the standard position box in the target result match.

6. The method according to claim 1, characterized in that The components include one or more of a light emitting diode, a fuse, a relay, a button, a seven-segment digital tube and a liquid crystal display screen.

7. A component detection system based on machine vision, characterized in that: include: A template acquisition module is used to obtain a standard image sequence of a preset standard circuit board; A data acquisition module, used to obtain a state image sequence of a circuit board to be inspected; wherein the circuit board to be inspected is of the same type as a standard circuit board; A model building module is used to build a component state detection model; wherein the component state detection model includes a U-net module, a discriminator module and a timing control module; the construction of the component state detection model specifically includes: preparing a training data set including multiple groups of sample images, annotated images obtained by annotating them, and timestamp labels; wherein the annotated images include a position box of a set color value at a component; the timestamp label is the time when the color value changes; using the U-net module to segment and color each group of sample images to generate a predicted image of the group; using the timing control module to analyze the time of the group based on the predicted image stamp information; use the discriminator to verify the authenticity of the predicted image and the corresponding sample image respectively to generate a first probability that the labeled image is a true image and a second probability that the predicted image is a true image; construct a discriminant loss function of the discriminator according to the first probability and the second probability; construct a pixel loss index according to the pixel difference between the predicted image and the labeled image, and construct a generation loss function of the U-net module based on the pixel loss index and the discriminant loss function; construct a timing loss function according to the timestamp label and the timestamp information; perform iterative training according to the timing loss function, the discriminant loss function and the generation loss function to obtain a component state detection model; A model output module is used to input the standard image sequence and the state image sequence into the component state detection model, and the component state detection model extracts features from the standard image sequence and the state image sequence to obtain template features and features to be detected; and generates a state template based on the template features, and generates a recognition result based on the features to be detected; wherein the state template includes a standard position frame of each component and a standard state attribute corresponding to a preset timestamp; and the recognition result includes a recognition position frame of each component, state information, and a timestamp of the state change moment; An optimization module, configured to calibrate the standard position frame according to the recognition position frame in the recognition result within a preset time window, obtain an optimized position frame, and generate an optimized state template based on the optimized position frame; The inspection result generating module is used to compare the recognition result with the optimized state template to generate the component inspection result.

8. A flying probe test device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The method comprises storing a computer program which can be loaded by a processor and execute the method according to any one of claims 1 to 6.

Images