Single board fault detection method

Abstract

The embodiment of the invention provides a single board fault detection method, and the method comprises the steps: obtaining a pre-screened target image of a to-be-detected single board and the detection information of the to-be-detected single board, and determining the global optical flow change of the target image based on an optical flow method, and determining a fault type of the to-be-detected single board according to the global optical flow change and the detection information of the to-be-detected single board, and determining a fault position of the to-be-detected single board according to the global optical flow change and the fault type. According to the embodiment of the invention, the problem of how to effectively realize the visual maintenance of the single board in the related technology can be solved, the visual operation and maintenance and fault detection of the single board are realized, and the positioning speed and fault processing speed of hardware faults are improved.

Description

Technical Field

[0001] This application relates to the field of hardware fault detection, and more specifically, to a single-board fault detection method. Background Technology

[0002] A single-board unit, also known as a printed circuit board (PCB), is a core component of electronic products. In the field of mobile communication networks, with continuous technological advancements, the application scope of single-board units is expanding daily. From early computer hardware to various electronic products today, single-board units are ubiquitous. They can be found in power supplies, base stations, sensors, and many other devices.

[0003] As the application areas of single-board computers continue to expand, the issue of single-board fault maintenance is becoming increasingly important. Although single-board computers have many advantages, these advantages also bring challenges. Single-board failures often lead to the paralysis of the entire system, causing great inconvenience to users.

[0004] Currently, single-board monitoring is still in the software monitoring stage, monitoring the board by monitoring changes in the values ​​of relevant components through software. However, the software monitoring part is also part of the single-board components, and software monitoring suffers from the limitation of not being able to perform self-checks, thus failing to meet all monitoring needs. In addition, single-board hardware-level faults, such as component displacement, pin damage, as well as single-board short circuits, leakage current, and single-board design defects, cannot be directly monitored by sensing components and require fault detection at the visualization level. However, due to the fine and complex nature of single boards, single-board visualization often comes with problems such as high cost and easy damage to monitoring equipment.

[0005] Therefore, there is still no good solution to the problem of how to effectively achieve visual maintenance of single boards in related technologies. Summary of the Invention

[0006] This application provides a single-board fault detection method to at least solve the problem of how to effectively achieve visual maintenance of single boards in related technologies.

[0007] According to one embodiment of this application, a single-board fault detection method is provided. The method includes: acquiring a target image of a pre-selected single-board to be detected and detection information of the single-board to be detected, wherein the number of frames of the target image is greater than or equal to 2; determining the global optical flow change of the target image based on optical flow method, wherein the global optical flow change includes the optical flow change of each pixel in the target image; determining the fault type of the single-board to be detected based on the global optical flow change and the detection information of the single-board to be detected; and determining the fault location of the single-board to be detected based on the global optical flow change and the fault type.

[0008] According to yet another embodiment of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein the computer program is executed by a processor to perform the steps in any of the above method embodiments.

[0009] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0010] According to yet another embodiment of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0011] This application presents a single-board fault detection method. By acquiring the target image of a pre-selected single-board to be tested and the detection information of the single-board to be tested, the global optical flow change of the target image is determined based on the optical flow method. The fault type of the single-board to be tested is determined based on the global optical flow change and the detection information of the single-board to be tested. The fault location of the single-board to be tested is determined based on the global optical flow change and the fault type. This method can solve the problem of how to effectively realize the visual maintenance of single boards in related technologies, realize the visual operation and maintenance and fault detection of single boards, and improve the speed of hardware fault location and fault handling. Attached Figure Description

[0012] Figure 1 This is a hardware structure block diagram of the single-board fault detection method according to an embodiment of this application;

[0013] Figure 2 This is a flowchart of a single-board fault detection method according to an embodiment of this application;

[0014] Figure 3 This is an overall flowchart of single-board fault detection in the embodiments of this application;

[0015] Figure 4 This is a schematic diagram of single-board image cropping and position correction in an embodiment of this application;

[0016] Figure 5 This is a schematic diagram of the diffusion model in the embodiments of this application;

[0017] Figure 6 This is a schematic diagram of the visualization of optical flow changes in the embodiments of this application;

[0018] Figure 7 This is a schematic diagram of fault location in an embodiment of this application;

[0019] Figure 8This is a schematic diagram of the single-board fault detection system according to the embodiments of this application. Detailed Implementation

[0020] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0022] The methods and embodiments provided in this application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of the single-board fault detection method according to an embodiment of this application, as shown below. Figure 1 As shown, a hardware board may include one or more ( Figure 1 Only one is shown in the diagram. A processor 12 (which may include, but is not limited to, a microprocessor MCU or programmable logic device, etc.) and a memory 14 for storing data are also shown. The computer terminal may further include a transmission device 16 for communication functions and an input / output device 18. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0023] The memory 14 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the single-board fault detection method in this embodiment. The processor 12 executes various functional applications and the single-board fault detection method by running the computer program stored in the memory 14, thus implementing the aforementioned method. The memory 14 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 14 may further include memory remotely located relative to the processor 12, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0024] The transmission device 16 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a telecommunications provider. In one example, the transmission device 16 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 16 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0025] One embodiment of this application provides a single-board fault detection method. Figure 2 This is a flowchart of a single-board fault detection method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0026] Step S202: Obtain the target image of the pre-selected board to be detected and the detection information of the board to be detected, wherein the number of frames of the target image is greater than or equal to 2;

[0027] Step S204: Determine the global optical flow change of the target image based on the optical flow method, wherein the global optical flow change includes the optical flow change of each pixel in the target image;

[0028] Step S206: Determine the fault type of the board under test based on the global optical flow change and the detection information of the board under test;

[0029] Step S208: Determine the fault location of the board to be tested based on the global optical flow change and the fault type.

[0030] In this embodiment, the board to be tested can be a board that has been identified as having a hardware fault after preliminary screening. However, the type and location of the hardware fault cannot be directly determined. This method embodiment can identify and locate the hardware fault of the board, reducing the maintenance pressure of hardware fault detection.

[0031] In this embodiment, the detection information of the single board may include, but is not limited to, at least one of the following: single board current, temperature, overall module temperature, operating power, operating ambient temperature, etc. The detection information may be reported by the single board itself, or generated by software or sensors. The detection information can be extracted from the relevant data of the single board through a single board diagnostic module in related technologies.

[0032] In this embodiment, the target image of the board to be inspected can be directly acquired by an imaging device, or it can be acquired by a low-quality imaging device and then processed to obtain a certain image high-definition image.

[0033] In this embodiment of the application, by using the above steps S202 to S208, the problem of how to effectively realize the visual maintenance of a single board in the related technology can be solved, realizing the visual operation and maintenance and fault detection of the single board, and improving the speed of hardware fault location and fault handling.

[0034] In this embodiment, the working environment of the single-board imaging firmware can meet the constraints of optical flow, which is the premise for this method embodiment to realize single-board fault detection and location based on optical flow.

[0035] Optical flow refers to the movement of target pixels in an image caused by the movement of objects or the camera in two consecutive frames. The prerequisites for applying optical flow include: constant brightness, temporal persistence (minor movement), and spatial consistency. Constant brightness means that the brightness of a point does not change over time. Temporal persistence means that changes in time do not cause drastic changes in position, allowing for partial derivatives of grayscale with respect to position. Spatial consistency means that adjacent points on the same surface in a scene have similar movements, and their projections onto the image plane are relatively close. Neighboring points in a scene are also projected as neighboring points on the image, and these neighboring points have consistent velocities. Constant brightness and temporal persistence are fundamental assumptions of general optical flow methods, while spatial consistency is a unique assumption of the LK optical flow method.

[0036] In some embodiments, optical flow methods may include LK (Lucas-Kanade) optical flow. Optical flow methods can be implemented based on convolutional neural networks or other machine learning models, and this application does not impose any limitations on this.

[0037] In an exemplary embodiment, the types of single-board hardware failures may include, but are not limited to, at least one of the following: circuit board corrosion, component displacement, component overheating leading to burnout, soldering defects, pin damage, etc., and this application does not impose any limitations on these.

[0038] In some embodiments, step S206 may include: inputting the global optical flow change and the detection information of the board to be detected into a pre-trained classification model to obtain the fault type.

[0039] In an exemplary embodiment, the classification model may include, but is not limited to, logistic regression, decision trees, clustering, Naive Bayes, neural networks, etc. When training the classification model, single-board images and detection information for various fault types can be pre-collected to determine the optical flow variation information of the single-board images. The fault type, detection information, and global optical flow variation are then used as a set of training data to train the classification model.

[0040] In an exemplary embodiment, the detection information can be converted into an N-dimensional feature vector, the optical flow change can be converted into an M-dimensional feature vector, and the combination can be used to obtain an N+M-dimensional feature vector. The N+M-dimensional feature vector is used as the input feature and the fault type is used as the target output. The classification model is trained to learn the complex mapping between the detection information, the optical flow change and the fault type, so that the fault type within the corresponding range can be found based on the input detection information and the optical flow change during fault detection.

[0041] In an exemplary embodiment, the loss function of the classification model can be the multiclass cross-entropy (CCE) loss function, used to measure the difference between the model's predicted probability distribution and the true labels. In multiclass classification problems, each sample typically belongs to a specific class, so the output is a probability distribution representing the probability of each class. CCE can be expressed as: Where n is the number of samples, true_i is the true fault type of the i-th sample, and pred_i is the predicted fault type of the i-th sample.

[0042] In some embodiments, step S208 may include the following steps:

[0043] Step S2082: Locate the fault location range corresponding to the fault type, wherein there is a preset mapping relationship between the fault type and the fault location range;

[0044] Step S2084: Determine the first feature matrix based on the fault location range, and determine the second feature matrix based on the global optical flow change;

[0045] Step S2086: Perform a convolution operation on the first feature matrix and the second feature matrix to obtain the third feature matrix;

[0046] Step S2088: The feature region in the third feature matrix with a feature value greater than a preset value is determined as the fault location, wherein the fault location includes multiple pixels in the target image within the fault location range where the optical flow change is greater than 0.

[0047] In this embodiment, the fault location range refers to all possible areas where this type of fault may occur. Based on the fault type, the possible areas where the fault location may occur can be determined, thereby narrowing down the fault location discrimination area. In this embodiment, single-board images of various fault types can be pre-acquired, and the fault location range corresponding to each fault type can be analyzed and statistically analyzed.

[0048] In this embodiment, the convolution operation on the first feature matrix and the second feature matrix is ​​actually to determine the overlapping area between the region where optical flow changes and the fault location range region. For example, the overlapping area of ​​the two can be directly determined as the fault location.

[0049] In this embodiment of the application, by detecting the fault location, the frequency of hardware failures occurring in certain locations of different types of single boards can be analyzed, thereby summarizing the design defects of the corresponding single boards and providing them to the single board manufacturers for adjustment, thereby reducing the single board failure rate.

[0050] In some embodiments, after determining the global optical flow change of the target image based on the optical flow method in step S204, the method further includes: step S205, performing frame interpolation processing on the target image to obtain the target video. Specifically, this may include the following steps:

[0051] Step S2052: Determine at least one frame image based on the global optical flow change between two adjacent frames in the target image;

[0052] Step S2054: Add the at least one frame image between the two adjacent frames to obtain the target video.

[0053] In this embodiment, the imaging device for a single board can acquire images at a lower frequency, and then visualize the single board fault in the form of video through frame interpolation. Furthermore, the image acquisition frequency can be set according to the frame interpolation capability; the stronger the frame interpolation capability, the lower the image acquisition frequency can be. For example, if the video frame rate is 12fps, and two frames can be added between adjacent frames through frame interpolation, then the image acquisition frequency of the imaging device can be set to 4fps.

[0054] In this embodiment, frame interpolation is based on the global optical flow variation between two adjacent image frames to predict the image frames between them. For example, only pixels with optical flow variations can be collected and predicted, while other pixels retain their original image features.

[0055] In some embodiments, after determining the fault location of the board to be tested based on the global optical flow change and the fault type in step S208, the method further includes: step S209, marking the fault location in the target video, and pushing the marked target video when a fault alarm is triggered. This method can provide maintenance personnel with a clear fault location, significantly improving the speed of fault handling and reducing the maintenance burden on maintenance personnel. In addition, hardware faults of different types of boards can be statistically analyzed separately, boards with design defects can be screened out, and feedback can be given to the board manufacturer for design adjustments, thereby reducing the board failure rate and ensuring a stable operating environment for the boards.

[0056] In some embodiments, after determining the type of board failure, a corresponding failure solution can be found and provided to maintenance personnel through a visual interface. This method can further improve the speed of fault handling and reduce the maintenance workload of maintenance personnel.

[0057] In this embodiment of the application, the basic principle of the optical flow method is as follows:

[0058] Based on the assumptions of constant brightness and duration, the following equation can be obtained:

[0059]

[0060] Where I is optical flow, x and y are the movements along the coordinate axes in the corresponding coordinate system, and t is the time interval of optical flow.

[0061] To solve the above equations, the LK optical flow method introduces the assumption of spatial consistency, that is, assuming a space of size n = m 2 Within the window, the optical flow of the image is a constant value, represented by a matrix as follows:

[0062]

[0063] This matrix can be represented as: Using the least squares method, we can obtain: Finally, the optical flow (velocity vector) can be obtained by solving for: in, This refers to the optical flow corresponding to the LK algorithm.

[0064] In this embodiment, spatial consistency is a specific assumption of LK optical flow. Therefore, the optical flow method in this embodiment can be the LK optical flow method or related improved algorithms, such as the LK pyramid optical flow method. However, this application is not limited to this; other types of optical flow methods can also be used to perform optical flow analysis on single-board images.

[0065] In some embodiments, step S204, which determines the global optical flow change of the target image based on optical flow, may include:

[0066] Step S2042: Input the target image into a pre-trained optical flow variation model to obtain the global optical flow variation of the target image. The optical flow variation model includes a convolutional neural network model based on the LK optical flow method or a convolutional neural network model based on the LK pyramid optical flow method.

[0067] In one exemplary embodiment, the convolutional neural network model based on the LK optical flow method may include, but is not limited to, the FlowNetC model.

[0068] In an exemplary embodiment, if the board is a multilayer board, the LK pyramid optical flow method can be used to process the different layers of the board one by one.

[0069] In some embodiments, before step S2042, which inputs the target image into a pre-trained optical flow variation model to obtain the global optical flow variation of the target image, the method further includes step S2040, training the optical flow variation model. Specifically, this may include the following steps:

[0070] Step S2040-1: Simulate the sample optical flow change caused by component movement when a single board fails, and generate a sample dataset, wherein the sample dataset includes multiple single board images with optical flow changes.

[0071] Step S2040-2: Extract the image features of each board image in the sample dataset using a convolutional neural network to obtain multiple board image feature maps;

[0072] Step S2040-3: Train the optical flow variation model based on feature points with feature differences in the feature maps of the multiple single-board images.

[0073] In this embodiment, since the optical flow of a single board is difficult to obtain directly, artificially generated optical flow changes can be used to generate a training dataset. For example, the optical flow changes caused by component movement when a single board malfunctions can be simulated. In this embodiment, a convolutional neural network is used to extract image features from the images before and after the optical flow changes. The locations on the two feature maps where there are feature differences are the artificially generated optical flow change points. When training the model, it is not necessary to traverse and calculate the feature information at all locations on the feature map; only the locations where there are optical flow changes (i.e., feature points with feature differences) need to be used, which can significantly save the time consumed by model training.

[0074] In some embodiments, step S204, which determines the global optical flow change of the target image based on optical flow, may include: step S2044, determining the feature correlation between two adjacent frames of the target image according to the following method, and using the feature correlation as the global optical flow change:

[0075] c(x1,x2)=∑ o∈[-k,k]×[-k,k] <f1(x1+o),f2(x2+o)> ;

[0076] Where c represents the feature correlation, f1(x1+o) and f2(x2+o) are the pixel distributions of x1 and x2 in the two adjacent frames respectively, k is the preset size of the basic square region in the two adjacent frames, and o is the position of any pixel in the basic square region.

[0077] In this embodiment, to improve the matching performance of the network, a "cross-correlation layer" is designed to artificially mimic the standard matching process. This layer first extracts features, then calculates the correlation between the features, and treats the feature correlation as an optical flow change. Image features can be extracted using a convolutional neural network. This application uses a k×k basic square region as a basic unit (patch) for image processing, and then determines the global optical flow change corresponding to all pixels in the image by traversing this region.

[0078] In some embodiments, prior to step S202, the method further includes:

[0079] Step S200-1: Collect detection information from multiple boards;

[0080] Step S200-2: Select boards with hardware abnormalities from the detection information of the multiple boards using the isolated forest algorithm as the boards to be detected.

[0081] In this embodiment, the Isolation Forest algorithm is used to initially screen the detection information of the single board, which can filter out data information of hardware anomalies on the single board. This data is often a fault that cannot be handled by software methods. The difference between the Isolation Forest algorithm and other anomaly detection algorithms is that it can isolate anomalies from all samples. During data processing, it does not need to calculate indicators related to distance and density, which can significantly improve data processing speed and reduce system overhead. Furthermore, through preliminary screening, only single boards with confirmed hardware faults are subjected to subsequent optical flow processing, which can avoid performing related optical flow processing on fault-free single boards, further reducing system overhead and improving hardware fault identification efficiency.

[0082] In some embodiments, prior to step S202, the method may further include:

[0083] Step S200-3: Acquire initial images of multiple single boards, wherein the multiple single boards include the single board to be detected;

[0084] Step S200-4: Perform preliminary processing and recognition on the initial image of the board to be detected to obtain a first image and the board type of the board to be detected;

[0085] Step S200-5: Perform resolution restoration on the first image to obtain a second image, wherein the resolution of the second image is greater than the resolution of the initial image;

[0086] Step S200-6: The second image is redrawn according to the single board type and the pre-trained diffusion model to obtain the target image, wherein the clarity of the target image is higher than that of the second image, and the pre-trained diffusion model is used to repair the image display details and color data of the second image.

[0087] In this embodiment, the board to be tested is a board that has been identified as having hardware faults through preliminary screening. This application only needs to perform subsequent processing on the images of the boards to be tested identified after preliminary screening, and the images of other normal boards or boards with software abnormalities can be directly discarded.

[0088] In this embodiment, a low-quality imaging device can be used for the single-board imaging, and the initial image can have lower clarity and resolution, reducing the cost of single-board visualization and maintenance. Then, through image processing, the pixels and clarity of the low-quality image are improved to meet the basic processing requirements of the optical flow method, thereby improving the accuracy of single-board fault identification.

[0089] In some embodiments, preliminary image processing may include, but is not limited to, board type recognition, image cropping, and image position correction.

[0090] In an exemplary embodiment, step S200-4 may include the following steps:

[0091] Step S200-41: Extract the convolution kernel features of the initial image of the board to be detected using a convolutional neural network;

[0092] Step S200-42: Identify the board type from the initial image of the board to be detected based on the convolution kernel features;

[0093] Step S200-43: Based on the convolution kernel features, extract pixel line features with changes in edge contrast from the initial image of the board to be detected;

[0094] Step S200-44: Crop the initial image of the board to be detected according to the pixel line features, and perform position correction on the cropped image to obtain the first image.

[0095] In this embodiment, the board type may include the board model. For example, text / symbol information can be extracted from the board image through feature extraction to identify the corresponding board model; alternatively, the overall layout of each component in the board can be learned based on image features to identify the board type. In this embodiment, since the component layouts of different types of boards vary significantly, the board type can be used as the basis for subsequent fault identification. Corresponding classification models, diffusion models, etc., can be trained for each board type to improve the accuracy of identification.

[0096] In some embodiments, step S200-5 may include: inputting the first image into a preset resolution restoration model to obtain a second image that meets the preset resolution requirements.

[0097] In this embodiment, the resolution restoration model can be implemented using the following algorithms: Super-Resolution Convolutional Neural Network (SRCNN), Super-Resolution Generative Adversarial Network (SRGAN), and Pixel Recursive Neural Network (Pixel RNN), but this application is not limited to these. The resolution restoration model can improve the resolution of the single-board image; therefore, this application has lower pixel requirements for the imaging device, thereby reducing the cost of the image acquisition equipment.

[0098] In some embodiments, step S200-6 may include the following steps:

[0099] Steps S200-65: The second image and the board type are converted into image feature vectors and text feature vectors through the Contrastive Language-Image Pre-Training (CLIP) model.

[0100] Steps S200-66: During the forward diffusion process, multiple downsampling operations are performed based on the image feature vector, and the corresponding convolution kernel parameters are recorded to obtain an intermediate feature map;

[0101] Step S200-67: During the backdiffusion process, the intermediate feature map is upsampled multiple times according to the text feature vector and the convolution kernel parameters to obtain the target image.

[0102] In this embodiment, CLIP is a combination of an image encoder and a text encoder. Its training process can be simplified to adding text descriptions to an image. For example, the second image (image information) and the board type (text information) can be encoded using the image encoder and the text encoder respectively. Then, cosine similarity is used to characterize whether they match. Subsequently, the loss function is used to calculate the loss of information due to the encoding, and the relevant code parameters are continuously updated to obtain new embedding vectors, namely image feature vectors and text feature vectors.

[0103] In this embodiment, the forward diffusion process gradually adds noise to the image, converting it into full noise, while the reverse diffusion process gradually removes noise from the full noise, at least completely removing it to recover a clear image. The diffusion model can improve image clarity; therefore, this application has lower requirements for the clarity of the imaging device, thereby reducing the cost of the image acquisition equipment.

[0104] In this embodiment, the image size is gradually reduced during the forward diffusion process and gradually enlarged during the reverse diffusion process until the image is restored to its original size.

[0105] In one exemplary embodiment, the diffusion model may employ a U-Net convolutional neural network, but this application is not limited thereto. In this embodiment, the board type is a key cue word in the image restoration process. Using the board type as a supervisory condition for the diffusion model can better adapt to the image restoration and redrawing of the current board, greatly improving the convenience of board monitoring on the visualization page.

[0106] In some embodiments, prior to step S200-6, the method may further include: training a diffusion model. Specifically, this may include the following steps:

[0107] Step S200-61: Obtain sample images of multiple training samples and label information of the sample images, wherein the label information includes the board type and board operating status;

[0108] Step S200-62: Convert the sample image and the label information into sample image vector and sample text vector using the CLIP model;

[0109] Step S200-63: Set a random time step for each training sample and convert the random time step into a time step vector;

[0110] Steps S200-64: Train the diffusion model based on the sample image vector, the sample text vector, and the time step vector.

[0111] In this embodiment, the single-board operating status can include: normal and hardware fault. Furthermore, hardware faults can be further subdivided according to fault type, such as component displacement and burnout. Incorporating the single-board operating status into the training process of the diffusion model can maximize the preservation of image details at the fault location during image reconstruction, preventing the loss of key features during image reconstruction.

[0112] The single-board fault detection method in this application embodiment has the following beneficial effects: it can achieve high-precision visualized fault operation and maintenance effects by using low-pixel imaging equipment and low-frequency data acquisition, significantly reducing the cost of single-board visualized operation and maintenance and solving the problem of short service life of visualized hardware. It adds a visualization dimension to single-board operation and maintenance diagnosis, allowing direct marking of fault locations in images or videos, greatly improving the speed of hardware fault location and fault handling. For hardware with small displacement and stable lighting environments, fault detection is achieved through optical flow, making it applicable to a wide range of scenarios.

[0113] Figure 3This is an overall flowchart of single-board fault detection in the embodiments of this application, such as... Figure 3 As shown, the process may include the following steps:

[0114] Step S1, Data Acquisition;

[0115] Step S2: Data repair and filtering;

[0116] Step S3: Image re-enhancing;

[0117] Step S4, optical flow processing;

[0118] Step S5, fault detection;

[0119] Step S6: Fault recovery and model improvement.

[0120] In this embodiment, the data in step S1 may include single-board detection data and single-board visualization data.

[0121] In step S1, data such as board current, temperature, overall module temperature, operating power, and ambient temperature can be obtained and extracted through detection modules or related sensors. Alternatively, the detection data of the board can be actively retrieved via command line on the network management page. The data obtained in step S1 are fault characteristic values ​​for board fault location.

[0122] In step S1, a low-quality imaging device can be set up. The image acquisition frequency is set according to the model complexity, system computing power, and actual conditions to acquire low-quality images of the board in real time. The acquired image data can be compressed and packaged together with the board's detection data before being sent to the next processing node, such as the network management system, for further data processing.

[0123] In this embodiment, as long as the device can obtain the relevant detection data and visualization data of the single board, the method in this application embodiment can be used to detect single board faults.

[0124] In this embodiment, step S2 is to filter the data obtained in step S1 and restore the image data to a data format that is convenient for optical flow analysis.

[0125] In step S2, the boards first need to be screened. The detection data is filtered using the Isolation Forest algorithm in machine learning to identify boards with hardware malfunctions, which are often beyond the capabilities of software solutions. This screening process helps to preliminarily identify boards with hardware faults. Next, convolutional neural networks can be used to identify and crop the image data. Firstly, the convolutional kernel can be used to identify and extract the edge positions of the boards to determine the cropping direction. Secondly, the convolutional kernel can be used to extract image features to identify the board type.

[0126] Figure 4 This is a schematic diagram of single-board image cropping and position correction in an embodiment of this application, as shown below. Figure 4 As shown, pixel line features with significant changes in edge contrast can be extracted based on the convolution kernel to obtain the cropping vector direction. The image can be cropped from the point where the direction changes, thus cropping out the current board. By rotating the image, the pixel line features can be aligned to the horizontal / vertical direction, thereby correcting the position of the board.

[0127] In some embodiments, since the visualization data in step S1 is acquired through a low-quality imaging device, its resolution may be low and unable to meet the resolution requirements of subsequent model processing. In this case, in step S2, it is also necessary to perform resolution restoration on the cropped and position-corrected image to obtain image data that meets the resolution requirements. This step can be achieved by feeding the cropped and position-corrected image into a resolution restoration algorithm. For example, the following algorithms can be used: Super-Resolution Convolutional Neural Network (SRCNN), Super-Resolution Generative Adversarial Network (SRGAN), Pixel Recurrent Neural Network (Pixel RNN), etc. This algorithm can restore the image resolution to the resolution required by the model in step S3, which is the first step in the process of using algorithms to achieve high-definition processing of low-quality images.

[0128] In step S3, the image with the resolution repaired in step S2 and the identified board type can be used as feature parameters to input into the pre-trained diffusion model to repair the details and color data of the image display, without erasing details of the fault causes such as board component displacement and burnout, thus improving the low-quality imaging.

[0129] In step S3, the repaired image and board type can be encoded using CLIP before being fed into the diffusion model. The diffusion model can employ a Unet convolutional neural network, which uses a convolutional neural network to extract image feature information, implement forward and reverse diffusion processes, and restore image details and content under a given loss function, thereby achieving image retouching.

[0130] Figure 5 This is a schematic diagram of the diffusion model in the embodiments of this application, such as... Figure 5 As shown, the diffusion model consists of the following two parts:

[0131] The forward diffusion process adds noise to an image and extracts image features.

[0132] The reverse diffusion process removes noise from an image and restores the original image data from its features.

[0133] In this embodiment, the forward diffusion process can be understood as a downsampling process, which extracts features by reducing the image size and increasing the number of image channels. The forward diffusion process gradually adds noise to the image data in multiple time steps, with each step making the image data closer to a pure noise state.

[0134] In this embodiment, the back-diffusion process can be understood as an up-and-down process that gradually restores the image size based on the feature information from the forward diffusion process. For example... Figure 5 As shown, the input image size is 128×128. Through multiple forward diffusion processes, a 4×4 intermediate feature is obtained. Then, through multiple backward diffusion processes, the image size is gradually restored to 128×128. When training the diffusion model, the model needs to learn how to backward recover the original image data from a noisy state, such as by using a neural network to predict the backward diffusion process of the data in each step. In each step of backward diffusion, the neural network needs to predict the noise components of the current noise state. By gradually reducing the noise, a high-resolution image close to the original image data is finally recovered.

[0135] In this embodiment, the resolution of the input image of the diffusion model needs to meet a preset requirement, but this application does not limit the specific value of the resolution and can adjust it according to the model training situation. If the original image of the board does not meet the resolution requirement, the image needs to be re-resolution repaired through the above step S2. If the original image of the board meets the resolution requirement, the resolution repair processing step can be skipped.

[0136] In practical applications, the boards produced by various manufacturers undergo extensive quality testing before being put on the market. This allows us to obtain a large number of images of normal boards and boards in abnormal states from the manufacturers, and to label these boards with "board type + board operating status". Due to the large number and styles of boards, it is easy to obtain a dataset of related board images. For example, the dataset can contain at least 400 million board images with similar resolution and their label information.

[0137] When training the diffusion model, directly using the original images may cause fault locations to be directly erased during the model's retouching process. Therefore, the CLIP model needs to be used, trained on a dataset containing single-board images and corresponding "single-board type + single-board operating status" labels. The CLIP model is a combination of an image encoder and a text encoder, and its training process can be simplified to adding text descriptions to images. First, the dataset is encoded using the image encoder and the text encoder respectively. Then, cosine similarity is used to characterize whether there is a match. Subsequently, a loss function is used to calculate the information loss of the encoding, and the relevant code parameters are continuously updated to obtain new image feature vectors and text feature vectors.

[0138] In one exemplary embodiment, during the training of the diffusion model, image feature vectors and text feature vectors (embeddings) generated by the CLIP model can be fed into the Unet convolutional neural network. First, a downsampling process is performed, reducing the image size and increasing the number of image channels to extract features. In each training round, a random time step 't' is selected for each training sample corresponding to the image and text feature vectors. Gaussian noise corresponding to time step 't' is applied to the image, and then the time step is converted into a corresponding time step feature vector (embedding). The image, text, and time step embedding are then fed into the Unet convolutional neural network to extract image features. Subsequently, upsampling is used to gradually restore the original image size. During the upsampling process, the input features at each step are not only the feature map output from the previous step but also include feature information corresponding to the same image size during the forward diffusion process. The diffusion model records the convolution kernel parameters during the image restoration process (i.e., the back diffusion process) and associates them with the label "board type + board running status". The subsequent image re-enhancing process can determine the specific image restoration process based on the label information corresponding to the input image, and then redraw based on the input image to achieve high-quality restoration of the input image and obtain a high-definition image.

[0139] When training a diffusion model, the information loss between the restored image and the original image can be calculated using a loss function. By adjusting the hyperparameters, a well-trained diffusion model can be obtained when the value of the loss function is less than the acceptable maximum loss threshold.

[0140] In an exemplary embodiment, the hyperparameters of the diffusion model may include a noise policy, a learning rate, and the number of iterations. Forward diffusion only adds noise to the image; the noise addition is random, and the step size of the noise samples can be determined by the noise policy in the hyperparameters, such as a variance mechanism. The number of iterations for backdiffusion can be selected based on the mean difference between the generated images using the negative log-likelihood (NLL), with the value closest to 1 being determined as the optimal number of iterations.

[0141] In this embodiment, by adding the board type to the supervision conditions of the diffusion model, the diffusion model under conditional supervision can better adapt to the image restoration and redrawing of the current board, which greatly improves the convenience of board monitoring on the visualization page.

[0142] In this embodiment, step S4 may specifically include: obtaining the global optical flow change of the high-definition image using optical flow method, performing frame interpolation on the high-definition image based on the optical flow change to generate a fault failure video, thereby realizing the display of the fault video, and then transferring the optical flow to step S5 for further processing. Since the image data acquisition frequency in step S1 is low, frame interpolation is still required to generate a smooth fault video for visualization. The image acquisition frequency in step S1 can be set according to the video frame interpolation capability; the stronger the frame interpolation capability, the fewer image frames need to be acquired per second.

[0143] In step S4, using the high-resolution image obtained in step S3, the global optical flow changes of the high-resolution image can be obtained using the optical flow method. The concept of optical flow has been explained in detail in the preceding content and will not be repeated here.

[0144] In this embodiment, the LK optical flow method can be used to generate the corresponding optical flow variation model. By inputting the optical flow information of each pixel in the high-definition image, an optical flow variation (wrapped) map can be generated, realizing the visualization of optical flow information.

[0145] Figure 6 This is a schematic diagram of the visualization of optical flow changes in the embodiments of this application, such as... Figure 6 As shown, the motion of each pixel in the optical flow diagram can be represented by a vector (arrow), allowing the observer to intuitively see the motion of objects in the image.

[0146] In an optical flow diagram, the Focus of Expansion (FOE) is the point where all outward-radiating optical flow vectors converge. It is related to the observer's line of sight and usually corresponds to the observer's direction of motion. An optical flow diagram is a two-dimensional velocity field representing the motion of objects in an image sequence. The motion of each pixel can be represented by a vector, whose direction and magnitude represent the direction and velocity of the object's motion, respectively.

[0147] In this embodiment, in step S4, all pixels with optical flow changes can be collected, and the predicted optical flow changes can be used to obtain the next frame image. Then, it can be used to predict the images of several subsequent frames. The number of predicted frames is the acquisition interval of the acquisition device. Therefore, the stronger the model prediction capability, the lower the acquisition frequency of the acquisition device, which can reduce device wear.

[0148] In some embodiments, when multiple PCBs exist, the LK optical flow method for a single layer often fails to achieve satisfactory results for deeper layers of optical flow. Therefore, for multi-layer PCB images, the LK pyramid optical flow method can be used to identify global optical flow changes. In this embodiment, a pyramid image of the original image can be generated based on the hierarchical structure of the PCBs, and the optical flow can be solved layer by layer to continuously and accurately determine the global optical flow changes. One pixel in the upper pyramid (low resolution) can represent two pixels in the lower pyramid.

[0149] In this embodiment, step S5 may specifically include: determining the fault (fault type and fault location) based on the detection information and global optical flow changes; determining fault-related information through machine learning models (such as classification models); and circling the fault location on the visualization page to notify maintenance personnel to resolve the fault.

[0150] In step S5, the single-board video, optical flow changes, and the detection information of the single board filtered in step S2 obtained in step S4 are used to classify and judge the fault type of the current single board through a machine learning model. Then, based on the fault type, the possible fault location range is determined. The fault location is used as a feature and convolved with the optical flow changes to determine the fault location. The location of the fault is marked from the single-board video, and relevant information such as fault flags, fault diagnosis results, and fault handling suggestions are fed back to the network management system to prompt the user to handle it in a timely manner.

[0151] Figure 7 This is a schematic diagram of fault location in an embodiment of this application, such as... Figure 7 As shown, the optical flow changes and the fault location range corresponding to the fault type are used as feature maps for convolution processing. The position with a convolution value of 1 in the convolution map is set as the fault location, which can accurately determine the actual location of the fault.

[0152] In one exemplary embodiment, step S5 can also automatically execute relevant automated scripts according to the fault type to achieve part of the fault handling. However, single-board hardware faults usually still require maintenance personnel to handle them offline.

[0153] In this embodiment, step S6 may specifically include: maintenance personnel handling the relevant faults based on the fault location. After the single-board fault is handled, the fault module can re-determine whether the fault has disappeared, and then notify the network management department to restore the alarm and remove the fault flag from the visual video.

[0154] In some embodiments, in step S6, the relevant fault handling methods, corresponding fault locations, and fault causes can be used to retrain the detection information model, continuously improving the corresponding model. For example, this processing flow can be uploaded to a database to continuously improve the database and enhance the overall model's prediction accuracy and versatility. Furthermore, high-failure-rate boards can be statistically analyzed to identify the locations of design defects on the boards and report them to the board manufacturers.

[0155] This application's embodiments achieve visualized single-board operation and maintenance and fault detection, improving the speed of hardware fault location and handling, thereby solving the problem of how to effectively implement visualized maintenance of single boards in related technologies. Furthermore, this application has low requirements for and low usage frequency of imaging equipment, significantly reducing the implementation cost of visualized operation and maintenance and the failure rate of imaging equipment.

[0156] In one exemplary embodiment, when a power amplifier component in a single-board unit is displaced or damaged, the single-board imaging firmware periodically acquires low-quality images and diagnostic messages, even though the bandwidth rate remains normal. After data processing, the images are redrawn in high definition. Based on the visualization information, subsequent optical flow changes are predicted, forming a video reflecting the component displacement or damage fault. By identifying the fault location and pixel changes (optical flow), it is determined that the fault is a power amplifier component, and the power amplifier component is circled, thereby achieving visual annotation of the single-board fault.

[0157] The single-board fault detection method in this application embodiment can also be applied to other hardware fault detection scenarios with small displacement and stable lighting environment, and has a wide range of applications.

[0158] According to another embodiment of this application, a single-board fault detection system is also improved.

[0159] Figure 8 This is a structural schematic diagram of the single-board fault detection system according to an embodiment of this application, such as... Figure 6 As shown, the system may include the following structure:

[0160] The system includes a single-board module 81, a data screening module 82, a visualization module 83, a fault detection module 84, and an alarm management system 85.

[0161] Single-board module 81 includes an image module and a detection data module. The image module acquires low-quality images using low-resolution image hardware, while the detection data module acquires the single-board's operating status and related detection data, such as power, temperature, and chip operating frequency, using the single-board's sensing hardware. The single-board module collects and processes the data before transmitting it to the next module.

[0162] Data screening module 82: This module consists of two parts: data screening and pixel repair and cropping. It uses machine learning to screen valid detection data, and a convolutional neural network to identify the board type and boundary position. The detection data is then transmitted to the fault detection module, and the image data is repaired for resolution before being transmitted to the visualization module.

[0163] Visualization module 83 includes image visualization redrawing, optical flow capture, and optical flow frame interpolation. It generates an image redrawing model using a diffusion model, inputs the board type and image (resolution-corrected image) obtained from the data screening module, generates a high-resolution board image, and uses optical flow to identify global optical flow changes. In this embodiment, the optical flow data has the following uses: using optical flow changes to interpolate frames to form video clips, reducing the acquisition frequency of the imaging module; inputting it into the fault detection module as a feature for fault classification; and using it in the fault detection module to determine the fault location and achieve fault location marking.

[0164] Fault diagnosis module 84: It includes two parts: fault detection and step location marking. First, it detects the fault type based on the machine learning classification algorithm and confirms the fault location range corresponding to the fault type. Combined with optical flow data, it determines the accurate fault location and marks it in the visualized video after frame interpolation. The marked fault video is reported to the network management system to notify relevant operation and maintenance personnel to handle the fault accordingly.

[0165] Alarm Management 85: Used for visualizing fault videos and providing alarm notifications. It can also execute corresponding automated processing scripts based on the fault type to automatically resolve some alarms, or push relevant fault solutions to maintenance personnel based on the fault type, thereby accelerating their fault handling efficiency. After the fault is resolved, the gateway will also notify the board module that the fault has been resolved and clear the corresponding alarm notification.

[0166] The embodiments of this application can achieve low-cost visual monitoring of single boards, mark the fault location of problematic single boards, and diagnose the cause of the fault based on the single board diagnostic data and the visualized fault point, obtain relevant fault solutions, provide operation and maintenance personnel with clear fault location, greatly improve the speed of fault handling, reduce the operation and maintenance personnel's operation and maintenance pressure, and at the same time screen out single boards with design defects to ensure the stable operation environment of single boards.

[0167] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to perform the steps in any of the above method embodiments.

[0168] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0169] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0170] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0171] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0172] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0173] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0174] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A single-board fault detection method, characterized in that, The method includes: Obtain the target image of the pre-selected board to be detected and the detection information of the board to be detected, wherein the number of frames of the target image is greater than or equal to 2; The global optical flow variation of the target image is determined based on the optical flow method, wherein the global optical flow variation includes the optical flow variation of each pixel in the target image; The fault type of the board under test is determined based on the global optical flow change and the detection information of the board under test; The fault location of the board to be tested is determined based on the global optical flow change and the fault type.

2. The method according to claim 1, characterized in that, The fault type of the board under test is determined based on the global optical flow change and the detection information of the board under test, including: The global optical flow change and the detection information of the board to be tested are input into a pre-trained classification model to obtain the fault type.

3. The method according to claim 1, characterized in that, Determining the fault location of the board under test based on the global optical flow change and the fault type includes: Find the fault location range corresponding to the fault type, wherein there is a preset mapping relationship between the fault type and the fault location range; A first feature matrix is ​​determined based on the fault location range, and a second feature matrix is ​​determined based on the global optical flow change; Perform a convolution operation on the first feature matrix and the second feature matrix to obtain the third feature matrix; The feature region in the third feature matrix with a feature value greater than a preset value is determined as the fault location, wherein the fault location includes multiple pixels in the target image within the fault location range where the optical flow change is greater than 0.

4. The method according to claim 1, characterized in that, After determining the global optical flow changes of the target image based on the optical flow method, the method further includes: At least one frame of image is determined based on the global optical flow change between two adjacent frames of the target image; The at least one frame is added between the two adjacent frames to obtain the target video.

5. The method according to claim 4, characterized in that, After determining the fault location of the board to be tested based on the global optical flow change and the fault type, the method further includes: The fault location is marked in the target video, and the marked target video is pushed when a fault alarm is triggered.

6. The method according to claim 1, characterized in that, Determining the global optical flow changes of the target image based on optical flow methods includes: The target image is input into a pre-trained optical flow variation model to obtain the global optical flow variation of the target image. The optical flow variation model includes a convolutional neural network model based on the LK optical flow method or a convolutional neural network model based on the LK pyramid optical flow method.

7. The method according to claim 6, characterized in that, Before inputting the target image into a pre-trained optical flow variation model to obtain the global optical flow variation of the target image, the method further includes: When a single board malfunctions, the sample optical flow changes caused by component movement are simulated to generate a sample dataset, which includes multiple single board images showing optical flow changes. Image features of each board image in the sample dataset are extracted using a convolutional neural network to obtain multiple board image feature maps. The optical flow variation model is trained based on feature points with differences in the feature maps of the multiple single-board images.

8. The method according to claim 1, characterized in that, Determining the global optical flow changes of the target image based on optical flow methods includes: The feature correlation between two adjacent frames x1 and x2 in the target image is determined according to the following method, and the feature correlation is used as the global optical flow change: c(x1,x2)=∑ o∈[-k,k]×[-k,k] <f1(x1+o),f2(x2+o)>; Where c represents the feature correlation, f1(x1+o) and f2(x2+o) are the pixel distributions of x1 and x2 in the two adjacent frames respectively, k is the preset size of the basic square region in the two adjacent frames, and o is the position of any pixel in the basic square region.

9. The method according to claim 1, characterized in that, The method further includes: Collect testing information from multiple boards; The isolated forest algorithm is used to filter out boards with hardware anomalies from the detection information of the multiple boards as the boards to be tested.

10. The method according to claim 1, characterized in that, The method further includes: Acquire initial images of multiple boards, wherein the multiple boards include the board to be detected; The initial image of the board to be detected is preliminarily processed and identified to obtain a first image and the board type of the board to be detected; The resolution of the first image is restored to obtain a second image, wherein the resolution of the second image is greater than the resolution of the initial image; The second image is redrawn according to the single-board type and the pre-trained diffusion model to obtain the target image, wherein the clarity of the target image is higher than that of the second image, and the pre-trained diffusion model is used to repair the image display details and color data of the second image.

11. The method according to claim 10, characterized in that, The initial image of the board to be detected is preliminarily processed and identified to obtain a first image and the board type of the board to be detected, including: The convolution kernel features of the initial image of the board to be detected are extracted using a convolutional neural network. Based on the convolution kernel features, the board type is identified from the initial image of the board to be detected; Based on the convolution kernel features, pixel line features with changes in edge contrast are extracted from the initial image of the board to be detected; The initial image of the board to be detected is cropped according to the pixel line features, and the cropped image is positionally corrected to obtain the first image.

12. The method according to claim 10, characterized in that, The resolution of the first image is restored to obtain a second image, including: The first image is input into a preset resolution restoration model to obtain a second image that meets the preset resolution requirements.

13. The method according to claim 10, characterized in that, The second image is redrawn according to the single-board type and the pre-trained diffusion model to obtain the target image, including: The second image and the board type are converted into image feature vectors and text feature vectors by learning a pre-trained CLIP model through image-text comparison; During the forward diffusion process, multiple downsampling operations are performed based on the image feature vector, and the corresponding convolution kernel parameters are recorded to obtain an intermediate feature map; During the backdiffusion process, the intermediate feature map is upsampled multiple times based on the text feature vector and the convolution kernel parameters to obtain the target image.

14. The method according to claim 13, characterized in that, The method further includes: Acquire sample images of multiple training samples and label information of the sample images, wherein the label information includes the board type and board operating status; The CLIP model is used to convert the sample images and the label information into sample image vectors and sample text vectors. A random time step is set for each training sample, and the random time step is converted into a time step vector; The diffusion model is trained based on the sample image vector, the sample text vector, and the time step vector.

15. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is executed by a processor to perform the method described in any one of claims 1 to 14.

16. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method as described in any one of claims 1 to 14.

17. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 14.

Images