A three-stage adaptive signal waveform image recognition method

Abstract

The present invention discloses a three-stage adaptive signal waveform image recognition method, comprising: collecting a screenshot of a single-channel signal waveform display of an oscilloscope as a signal waveform image, and performing preprocessing and annotation to form a required data set, and dividing it into a training set and a test set; performing data enhancement on the images in the training set, inputting the enhanced images and corresponding labels into a signal waveform image classification model training to obtain an optimal model; intercepting the optimal model, wherein the connection of the optimal model from the initial convolution unit to the second fully connected neural network layer is retained as a signal waveform image feature extraction model; after obtaining the signal waveform image feature extraction model, completing the recognition of the signal waveform image on the test set. The present invention solves the problem of low detection accuracy caused by complex situations such as the non-uniform line color and position of the signal waveform displayed by the oscilloscope and the small signal amplitude during the signal waveform recognition process.

Description

Technical Field

[0001] The present invention relates to the technical field of signal waveform image recognition, and in particular to a three-stage adaptive signal waveform image recognition method. Background Art

[0002] In the field of electronic system design, the importance of intelligent information recognition technology has become increasingly prominent and has become one of the hot topics in this field. In the design, maintenance and verification of electronic systems, the recognition of the signal waveform of the circuit system port not only helps to analyze the operating status of the system, but also provides an important reference for troubleshooting. Therefore, the circuit port signal waveform has become an important indicator of the working status of the electronic system.

[0003] By introducing intelligent signal recognition technology, the judgment of the operating status of electronic systems has become more efficient and objective, thereby improving the efficiency of electronic system design and maintenance. However, due to the wide variety of actual circuit port signals, relying on manual observation and judgment is not only inefficient, but also easily affected by subjective factors. The application of intelligent signal recognition technology can achieve objective experimental result judgment and improve the efficiency of experimental judgment. Therefore, applying intelligent signal waveform recognition technology to circuit system design can significantly improve the verification and maintenance efficiency of electronic systems.

[0004] Although signal recognition technology has made significant progress in recent years, it has not yet been widely used. This is mainly because signal waveform recognition technology still faces many challenges in practical applications:

[0005] 1. As the main device for signal waveform acquisition in circuit systems, oscilloscopes can convert dynamic signal values ​​into static images after buffering, and more intuitively display the characteristics of signal waveforms. However, the accuracy of signal waveform image recognition will be affected by interference factors such as signal amplitude and display position.

[0006] 2. Due to differences in oscilloscope brands and models, the color of the signal waveform image is not uniform, which affects the accuracy of signal waveform type recognition. Especially when the oscilloscope displays multi-channel signal waveform images at the same time, factors such as inconsistent color and overlapping positions of the signal waveform image will seriously interfere with the recognition quality of the signal waveform image.

[0007] 3. Signal waveforms include not only basic sine waves, triangle waves, square waves, etc., but may also include other waveform images such as analog amplitude modulation signals and digital amplitude modulation signals. This requires that the signal waveform recognition technology has online scalability, so that the types of signal waveform images that can be recognized can be expanded online as needed.

[0008] In summary, although signal waveform recognition technology has great application value and development prospects, it still needs to overcome many difficulties in practical applications before it can be widely used. In actual application scenarios, products using signal waveform image intelligent recognition technology are rare. Therefore, inventing a high-precision, scalable and robust signal waveform image recognition method has high practical application value. Summary of the invention

[0009] The purpose of the present invention is to overcome the shortcomings and disadvantages of the prior art, and propose a three-stage adaptive signal waveform image recognition method, which can effectively solve the problem of low recognition accuracy caused by inconsistent waveform image color due to oscilloscope model differences, too small input signal amplitude, and waveform display position differences and overlaps caused by oscilloscope settings, thereby ensuring a high recognition accuracy rate for signal waveform images. In addition, the method replaces the direct output of signal waveform classification by adopting feature extraction and metric learning, and realizes the online expansion of signal waveform image recognition types.

[0010] To achieve the above object, the technical solution provided by the present invention is: a three-stage adaptive signal waveform image recognition method, comprising the following steps:

[0011] 1) Collect screenshots of the oscilloscope's single-channel signal waveform display as signal waveform images, and preprocess and annotate the images to filter out images of poor quality and images labeled with sine waves, square waves, triangle waves, clutter, and no waveform labels to form the required data set, which is then divided into a training set and a test set;

[0012] 2) Perform data enhancement on the images in the training set, and then input the enhanced images and corresponding labels into the constructed signal waveform image classification model for training to obtain an optimal model after training; wherein, the model includes 1 initial convolution unit C1, 3 residual blocks R1~R3, 3 SimAM attention mechanism layers, 1 full average pooling layer GAP, 2 ReLU activation function layers and 3 fully connected neural network layers F1~F3, and its connection order is: initial convolution unit C1→first SimAM attention mechanism layer→first residual block R1→second SimAM attention mechanism layer→second residual block R2→third SimAM attention mechanism layer→third residual block R3→full average pooling layer GAP→first fully connected neural network layer F1→first ReLU activation function layer→second fully connected neural network layer F2→second ReLU activation function layer→third fully connected neural network layer F3;

[0013] 3) The optimal model obtained after training is intercepted, in which the connection from the initial convolution unit C1 to the second fully connected neural network layer F2 is retained as the signal waveform image feature extraction model;

[0014] 4) After obtaining the signal waveform image feature extraction model, the signal waveform image recognition of the test set is completed in the following three stages: the first stage: the input dual-channel signal waveform image is color filtered CFilter according to the color instruction, so as to separate it into two independent channel signal waveform images, and the two separated signal waveform images are grayed GSale, and the image size is transformed EAmp; the second stage: the signal waveform image feature extraction model is used to extract the eigenvalue vector of the image obtained in the first stage; the third stage: the eigenvalue vector extracted in the second stage is calculated CosSim with the eigenvalue vector of the recorded signal waveform image, and the calculation results are sorted in descending order; finally, according to the sorting result, the type of the recorded signal waveform image corresponding to the first place is selected as the recognition result.

[0015] Further, the step 1) includes the following steps:

[0016] 1.1) Data acquisition: Use professional instrument control software, including Virtual Instrument Software Architecture software, or a smart phone to directly take screenshots or photos of different types of signal waveforms displayed by the oscilloscope; connect the oscilloscope with a signal generator, and the generated signal waveforms include sine waves, square waves, and triangle waves, so as to obtain the oscilloscope's display of the above types of signal waveforms; suspend and ground the oscilloscope's waveform acquisition terminal to obtain the oscilloscope's clutter signal display; define the oscilloscope's screenshot or photo without any signal waveform display as no waveform display; when obtaining the signal waveform image, do not specify the signal's amplitude, frequency, and the position of the waveform displayed on the screen; the captured signal waveform images are all single-channel signal waveform images of the oscilloscope, that is, the image only displays a continuous signal waveform;

[0017] 1.2) Image preprocessing: For each captured image, the signal waveform display area is intercepted and the captured images with poor quality are eliminated, including images with low shooting resolution, severe lighting interference, or blurred and unusable images. After elimination, signal waveform images covering sine waves, square waves, triangle waves, clutter and no waveform types are obtained;

[0018] 1.3) Image annotation: According to the captured signal waveform image type, the pre-processed images are divided into five signal waveform image categories: sine wave signal image, square wave signal image, triangle wave signal image, noise signal image and no waveform signal display image, and LabelImg is used to label the images corresponding to the above categories as "sine", "square", "triangle", "noise" and "none" respectively;

[0019] 1.4) Divide the data set: After the image annotation is completed, all images and the corresponding annotation files are divided into training sets and test sets according to the preset ratio.

[0020] Further, in step 2), the specific situation of the signal waveform image classification model is as follows:

[0021] The initialization convolution unit C1 includes a 3×3 convolution kernel, edge padding length padding=1, stride=1, a convolution layer with 64 output channels, and a ReLU activation function layer, which realizes the feature conversion and extraction of different dimensions of the input single-channel grayscale signal waveform image;

[0022] After the initial convolution unit C1 obtains preliminary feature extraction and conversion, the first SimAM attention mechanism layer is connected to distinguish the importance of the influence of different pixels on the features in the feature map, enhance the representation ability of the signal waveform image features, and make the model pay more attention to the key areas of the signal waveform image features and strengthen the expression of the signal waveform image features; especially for the features extracted by the initial convolution unit C1, the SimAM attention mechanism layer can help the model flexibly allocate the attention weights of the feature map when facing the signal waveform display at different positions, dynamically enhance the feature capture of the signal waveform at different positions, strengthen the expression of key features and effectively suppress interference features, thereby improving the recognition accuracy;

[0023] After the first SimAM attention mechanism layer, three residual blocks R1~R3 ​​are connected continuously to ensure the extraction of convolution operation features and mitigate the problem of detail feature loss caused by the reduction of image size in convolution operation. At the same time, the residual connection also alleviates the gradient vanishing problem of deep network and accelerates the convergence speed of the model. Each residual block includes a main branch and a residual shortcut. The main branch includes 2 convolution layers and 2 batch normalization layers, and the connection order is convolution layer→batch normalization layer→convolution layer→batch normalization layer. At the same time, a residual shortcut is connected between the input of the residual block and the output of the main branch of the residual block, which is composed of convolution layer→batch normalization layer connection. The input of the first residual block R1 is a 64-channel feature map, and the output is a 128-channel feature map. The input of the second residual block R2 is a 128-channel feature map, and the output is a 256-channel feature map. The input of the second residual block R3 is a 256-channel feature map, and the output is a 512-channel feature map.

[0024] The outputs of the first residual block R1 and the second residual block R2 are connected to a SimAM attention mechanism layer respectively. By combining the SimAM attention mechanism layer, the attention weight of the feature can be flexibly adjusted, and the features extracted by the residual block can be specifically distinguished and strengthened, further enhancing the robustness of the feature extraction of signal waveforms at different positions.

[0025] In order to reduce the number of model parameters and speed up the inference of the model, the third residual block R3 is directly input into the full average pooling layer GAP, and the 512 channel feature maps output by the third residual block R3 are abstracted into 512 corresponding average feature values;

[0026] After obtaining 512 average eigenvalues, the 512 average eigenvalues ​​are input into three fully connected neural network layers F1 to F3 for further extraction and mapping, and converted into specific outputs; wherein, the first fully connected neural network layer F1 has an input of an eigenvalue vector of 512 numerical values, outputs an eigenvalue vector of 1024 numerical values, and then is processed by the ReLU activation function layer; the second fully connected neural network layer F2 has an input of an eigenvalue vector of 1024 numerical values, outputs an eigenvalue vector of 512 numerical values, and then is processed by the ReLU activation function layer; the third fully connected neural network layer F3 has an input of an eigenvalue vector of 512 numerical values, outputs an eigenvalue vector of 5 numerical values, and the sizes of these 5 numerical values ​​respectively represent the possibilities of five types of signal waveform images.

[0027] Furthermore, in step 2), the data enhancement methods include: image grayscale, image translation, image shearing, image stretching and image noise injection, and image size conversion to 320×640.

[0028] Further, in step 3), the signal waveform image feature extraction model uses the 512-valued feature value vector output by the second fully connected neural network layer F2 as the signal waveform image feature value expression.

[0029] Further, in step 4), the first stage is called the multi-channel signal waveform image separation and enhancement stage, and its color instruction format is: JSON key-value pair of {"color1": "R(8bit),G(8bit),B(8bit)", "color2":"R(8bit),G(8bit),B(8bit)"}, where color1 and color2 represent the colors displayed by the two channel signal waveforms; its image separation and enhancement includes the following steps:

[0030] 4.1.1) By parsing the color command in JSON format, extract the "values" corresponding to the "keys" of color1 and color2 in the color command, that is, the red, green and blue colors displayed by the signal waveform of each channel. Each channel is 8 bits wide and the value range is 0 to 255;

[0031] 4.1.2) Obtain the color values ​​of color1 and color2 according to step 4.1.1) and perform color filtering on the input waveform signal image respectively, separate it into images showing the signal waveform of a single channel and save it; wherein, the color filtering method is as follows:

[0032] Set the red, green, and blue color values ​​corresponding to color1 of the received signal waveform image color instruction to R, G, and B respectively. The red, green, and blue color values ​​of a pixel at a certain position of the input signal waveform image are r, g, and b respectively. Calculate the total color variation value according to the Euclidean distance :

[0033] ;

[0034] right Perform a range relaxation of 200 as a condition for preserving pixel colors;

[0035] Based on the above settings, the color filtering process is as follows:

[0036] Traverse all position pixels of the input signal waveform image and perform the following operations on each position pixel:

[0037] if , retain the color value of the pixel at that position; otherwise, set the color of the pixel at that position to r=0, g=0, b=0, that is, black;

[0038] The traversed position pixels are formed into a new image and saved; similarly, the color filtering process of color2 is the same as color1, and a new image is also formed and saved;

[0039] 4.1.3) The two signal waveform images saved by color filtering are grayed out respectively, and then the sizes of the two signal waveform images are adjusted to 320×640, so that the image height size is stretched, so that the details of the signal waveform images with small amplitudes are magnified, and the display of the signal waveform details is expanded.

[0040] Further, in step 4), the second stage is called the signal waveform image feature extraction stage, which includes sequentially inputting the two signal waveform images processed in step 4.1.3) into the signal waveform image feature extraction model to sequentially obtain 512 eigenvalue vectors of the two signal waveform images.

[0041] Further, in step 4), the third stage is called the feature comparison stage, which includes: Represents the 512 eigenvalue vectors of the first signal waveform image output by the signal waveform image feature extraction model, represents the eigenvalue of the stored nth signal waveform image, then the cosine similarity calculation process is:

[0042] ;

[0043] In the formula, Represents the similarity comparison result with the stored nth signal waveform image feature value, represents the dot product operation of two vectors, and Represents vectors and The Euclidean norm of .

[0044] Further, in step 4), it is necessary to have a eigenvalue vector of the recorded signal waveform image. When recording the eigenvalue vector of the signal waveform image, an SQLite database is used as a storage container for the recorded signal waveform image eigenvalue vector, and the storage includes: type labels corresponding to the signal waveform image, including "sine", "square", "triangle", "noise" and "none", and 512 eigenvalue vectors of the signal waveform image; the SQLite database can also store other types of signal waveform image features to achieve the expansion of the signal waveform image recognition type; the eigenvalue vector of the recorded signal waveform image uses the same feature extraction model in the signal waveform image feature extraction stage. By using the same feature extraction model, the consistency and accuracy of feature extraction when processing and identifying different signal waveform images are ensured, thereby ensuring the stability and reliability of the recognition result; when recording and storing the signal waveform image eigenvalue vector, the input is a single-channel signal waveform image, that is, the signal waveform image only displays a single signal waveform, and the recorded signal waveform type has been determined.

[0045] Further, in step 4), the specific process of pre-entering the signal waveform image eigenvalue vector includes: ① gray-scale the entered signal waveform image and adjust the image size to 320×640; ② input the gray-scaled and resized image into the signal waveform image feature extraction model to extract 512 numerical eigenvalue vectors of the signal waveform image; ③ enter the extracted 512 eigenvalue vectors and the determined type labels of the signal waveform image, including the "sine", "square", "triangle", "noise", "none", "AM" and "ASK" labels corresponding to the sine wave signal image, square wave signal image, triangle wave signal image, noise signal image, no waveform signal display image, analog amplitude modulation wave signal image and digital amplitude modulation wave signal image, respectively, into the SQLite database.

[0046] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0047] 1. The data acquisition of the present invention is completed by taking pictures of the oscilloscope screen with a smart phone and taking screenshots of the oscilloscope screen with dedicated software. These image data include the display of sine waves, square waves, triangle waves, clutter signal waveform images and images without waveforms, providing data support for the research on signal waveform image feature expression and recognition classification.

[0048] 2. The present invention realizes filtering of signal waveform images of different colors in the oscilloscope display image through color instructions, so that the oscilloscope display of any brand and model can be matched according to actual conditions.

[0049] 3. By changing the size of the signal waveform image to 320×640, the signal waveform amplitude display is enhanced, and the display details of the small-amplitude signal waveform image are magnified.

[0050] 4. The present invention not only ensures feature extraction through convolutional layers and residual block structures, but also alleviates the problem of loss of detail features caused by reducing the image size due to convolution operations; at the same time, the residual connection also effectively alleviates the gradient vanishing problem of deep networks and accelerates the convergence speed of the model.

[0051] 5. Combined with the SimAM attention mechanism layer, without adding additional parameters, the attention weights of the feature map can be flexibly allocated to dynamically enhance the feature capture of signal waveforms at different positions, thereby strengthening the expression of key features and effectively suppressing interference, thereby improving recognition accuracy.

[0052] 6. The present invention converts the signal waveform classification task into a feature extraction task, and realizes the online expansion of any type of signal waveform image recognition. That is, by entering and saving the signal waveform image and comparing the feature similarity, it is possible to enter other signal waveform features besides sine waves, square waves and triangle waves according to actual needs, thereby realizing the online expansion of signal waveform image type recognition.

[0053] 7. After experimental verification, when the characteristic values ​​of each type of signal waveform image were entered 20 times, the recognition accuracy of the present invention reached 97.8%; further, after the signal waveform image was expanded and the characteristic values ​​of analog amplitude modulation wave signal image and digital amplitude modulation wave signal image type were added, the accuracy reached 97.3%. In addition, the signal waveform image classification model of the present invention is only 102MB in size and has an inference speed of only 61ms, which can be deployed on embedded platforms such as handheld instruments. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flowchart of the overall framework of the method of the present invention; in the figure, CFilter is color filtering, GSale is image grayscale processing, EAmp is image size 320×640 transformation processing, C1 is the initial convolution unit, R1, R2 and R3 are residual blocks, GAP is the full average pooling layer, SimAM is the SimAM attention mechanism layer, F1, F2 and F3 are fully connected neural network layers, ReLU is the ReLU activation function layer, and CosSim represents cosine similarity calculation.

[0055] Figure 2 It is a structural diagram of the signal waveform image classification model and the signal waveform image feature extraction model. In the figure, CONV is the convolution layer; BN is the Batch normalization layer; Shorcut is the residual shortcut; ReLU is the ReLU activation function layer; SimAM is the SimAM attention mechanism layer, GAP is the full average pooling layer; F1, F2, and F3 are fully connected neural network layers.

[0056] Figure 3 Filter flow chart for color.

[0057] Figure 4 This is the experimental effect diagram of the signal waveform image feature extraction model; in the figure, sine represents a sine wave signal image, square represents a square wave signal image, and triangle represents a triangular wave signal image.

[0058] Figure 5 This is the effect diagram of the extended waveform type recognition experiment; in the figure, AM represents the analog amplitude modulation wave signal image, ASK represents the digital amplitude modulation wave signal image, and square represents the square wave signal image. DETAILED DESCRIPTION

[0059] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0060] like Figures 1 to 5 As shown, this embodiment discloses a three-stage adaptive signal waveform image recognition method, the specific situation of which is as follows:

[0061] 1) Establishment of signal waveform image data set: Use professional software or mobile phone to take screenshots of the oscilloscope screen display to obtain display images of sine wave, square wave, triangle wave, noise and no waveform. Among them, no waveform refers to the image screenshot without any signal waveform displayed on the oscilloscope screen. After image preprocessing, the above display images are labeled as sine wave, square wave, triangle wave, noise and no waveform categories using LabelImg, and then the images are used for model training and testing respectively;

[0062] The details of constructing the signal waveform image dataset are as follows:

[0063] 1.1) Data acquisition: Use professional instrument control software, including Virtual Instrument Software Architecture software, or a smart phone to directly take screenshots or photos of different types of signal waveforms displayed by the oscilloscope. Among them, the oscilloscope is connected with a signal generator, and the generated signal waveforms include sine wave, square wave, and triangle wave outputs, so as to obtain the oscilloscope's display of the above types of signal waveforms; the waveform acquisition terminal of the oscilloscope is suspended and grounded to obtain the oscilloscope's clutter signal display; the screenshot or photo of the oscilloscope without any waveform display is defined as no waveform display. When obtaining the signal waveform image, the amplitude, frequency, and screen display position of the signal are not specified to enhance the robustness of the model; further, the captured signal waveform images are all signal waveform images of a single channel of the oscilloscope, that is, the image only displays a continuous signal waveform image.

[0064] 1.2) Image preprocessing: For each captured image, the signal waveform display area is captured and images with poor quality are eliminated, including images with too low shooting resolution, severe lighting interference, or blurred and unusable images, to prevent images of poor quality from having a negative impact on the model. The 10,000 images obtained after elimination cover signal waveform images of sine waves, square waves, triangle waves, clutter, and no waveform.

[0065] 1.3) Image annotation: According to the captured signal waveform image type, the pre-processed images are divided into five signal waveform image categories: sine wave signal image, square wave signal image, triangle wave signal image, noise signal image and no waveform signal display image, and LabelImg is used to label the images corresponding to the above categories as "sine", "square", "triangle", "noise" and "none" respectively;

[0066] 1.4) Divide the data set: After the image annotation is completed, all images and the corresponding annotation files are divided into training set and test set in a ratio of 8:2.

[0067] 2) The images in the training set are sent to the constructed signal waveform image classification model for training, where the model includes 1 initial convolution unit C1, 3 residual blocks R1~R3, 3 SimAM attention mechanism layers, 1 full average pooling layer GAP, 2 ReLU activation function layers and 3 fully connected neural network layers F1~F3, and its connection order is: initial convolution unit C1→first SimAM attention mechanism layer→first residual block R1→second SimAM attention mechanism layer→second residual block R2→third SimAM attention mechanism layer→third residual block R3→full average pooling layer GAP→first fully connected neural network layer F1→first ReLU activation function layer→second fully connected neural network layer F2→second ReLU activation function layer→third fully connected neural network layer F3. Before training, the training set is first enhanced, and then the enhanced image and the corresponding label are input into the signal waveform image classification model. The signal waveform image classification model obtains the input signal waveform image to obtain the 5-value vector of the third fully connected neural network layer F3; then, the 5-value vector is input into the Softmax function, and the cross entropy loss function is used to calculate the loss value between the Softmax function output and the image label value; further, the stochastic gradient descent method is used, and after multiple iterations, the loss value is minimized and stabilized, thereby obtaining the optimal signal waveform image classification model.

[0068] The data enhancement methods include: image grayscale, image translation, image shearing, image stretching, image noise injection, and image size conversion to 320×640.

[0069] The data-augmented image is input into the signal waveform image classification model, where the initialization convolution unit C1 of the model includes a 3×3 convolution kernel, edge padding length padding=1, stride=1, a convolution layer with 64 output channels and a ReLU activation function layer to realize feature conversion and extraction of different dimensions for the input single-channel grayscale signal waveform image; in addition, the convolution kernel size is set to 3×3 to prevent the loss of image detail information due to too fast reduction in image size.

[0070] After the initial convolution unit C1 obtains preliminary feature extraction and conversion, the first SimAM attention mechanism layer is connected to distinguish the importance of the influence of different pixels in the feature map on the features, enhance the representation ability of the signal waveform image features, and make the model pay more attention to the key areas of the signal waveform image features and strengthen the expression of the signal waveform image features; especially for the features extracted by the initial convolution unit C1, the SimAM attention mechanism layer can help the model flexibly allocate the attention weights of the feature map when facing signal waveform images at different positions, dynamically enhance the feature capture of signal waveforms at different positions, strengthen the expression of key features and effectively suppress interference features, thereby improving the recognition accuracy.

[0071] After the first SimAM attention mechanism layer, three residual blocks R1 to R3 are connected continuously to ensure the extraction of convolution operation features and mitigate the problem of detail feature loss caused by the reduction of image size in convolution operation. At the same time, the residual connection also alleviates the gradient vanishing problem of deep networks and accelerates the convergence of the model. Each residual block includes a main branch and a residual shortcut. The main branch includes 2 convolutional layers and 2 batch normalization layers, and the connection order is convolutional layer → batch normalization layer → convolutional layer → batch normalization layer. At the same time, a residual shortcut connection Shortcut is connected between the residual block input and the main branch output of the residual block, which is composed of convolutional layer → batch normalization layer connection. Let Xn represent the feature map input to the nth residual block, Cn->m represent the convolution operator of the mth convolution layer in the main branch of the nth residual block, CSn represent the convolution layer of the residual shortcut in the nth residual block, BN(·) represent the batch normalization layer operator, R(·) represent the ReLU activation function, then the output feature fusion of the nth residual block is represented as:

[0072] Yn->1 = BN( BN( Xn*Cn->1) *Cn->2);

[0073] Yn->1 = BN(BN( Xn*Cn->1) *Cn->2);

[0074] Yn->2 = BN(Xn*CSn);

[0075] Yn= R(Yn->1+ Yn->2);

[0076] In the formula, Yn->1 and Yn->2 represent the output of the main branch and the residual shortcut in the nth residual block, respectively, and Yn represents the output of the feature fusion of the nth residual block. The main branch of each residual block includes two convolutional layers, namely Cn->1 and Cn->2; among them, the first convolution and the second convolution are processed by batch normalization. In the residual shortcut, the data input to the residual block is processed by one convolutional layer and batch normalization in turn, and then the output results of the main branch and the shortcut branch are added element by element at the corresponding channel position. Furthermore, the result of the addition is processed by a ReLU activation function layer to finally obtain the output result of the residual block. Among them, Cn->1 of each residual block is a 3×3 convolution kernel, stride=2 and padding=1, Cn->2 is a 3×3 convolution kernel, stride=1 and padding=1; CSn of the residual shortcut connection Shortcut is a 1×1 convolution kernel, stride=2. The signal waveform image classification model proposed in this embodiment includes three residual blocks R1~R3. Therefore, the values ​​of n are 1, 2, and 3, representing the first residual block R1, the second residual block R2, and the third residual block R3, respectively. The first residual block R1 inputs a feature map of 64 channels and outputs a feature map of 128 channels; the second residual block R2 inputs a feature map of 128 channels and outputs a feature map of 256 channels; the third residual block R3 inputs a feature map of 256 channels and outputs a feature map of 512 channels.

[0077] Furthermore, the outputs of the first residual block R1 and the second residual block R2 are connected to the SimAM attention mechanism layer respectively. By combining the SimAM attention mechanism layer, the attention weight of the feature can be flexibly adjusted, the features extracted by the residual block can be differentiated and strengthened in a targeted manner, and the robustness of the feature extraction of signal waveforms at different positions can be enhanced.

[0078] Furthermore, in order to reduce the number of model parameters and speed up the inference speed of the model, the third residual block R3 is directly input into the full average pooling layer GAP, and the 512 channel feature maps output by the third residual block R3 are mapped to 512 corresponding average feature values.

[0079] Furthermore, after obtaining 512 average eigenvalues, the 512 average eigenvalues ​​are input into three fully connected neural network layers F1 to F3 for further extraction and mapping, and converted into specific outputs. Among them, the first fully connected neural network layer F1 has an input of an eigenvalue vector of 512 values, an output of an eigenvalue vector of 1024 values, and then processed by the ReLU activation function layer; the second fully connected neural network layer F2 has an input of an eigenvalue vector of 1024 values, an output of an eigenvalue vector of 512 values, and then processed by the ReLU activation function layer; the third fully connected neural network layer F3 has an input of an eigenvalue vector of 512 values, and an output of an eigenvalue vector of 5 values, and the sizes of these 5 values ​​represent the possibility of five types of signal waveform images.

[0080] 3) After obtaining the optimal signal waveform image classification model, the signal waveform image classification model is truncated, wherein the connection from the initial convolution unit C1 to the second fully connected neural network layer F2 is retained as the signal waveform image feature extraction model; wherein the signal waveform image feature extraction model uses the 512-valued eigenvalue vector output by the second fully connected neural network layer F2 as the signal waveform image eigenvalue expression.

[0081] 4) After obtaining the signal waveform image feature extraction model, the signal waveform image recognition of the test set is completed in the following three stages: the first stage: the input dual-channel signal waveform image is color filtered CFilter according to the color instruction, so as to separate it into two independent channel signal waveform images, and the two separated signal waveform images are grayed GSale, and the image size is transformed to 320×640 EAmp; the second stage: the signal waveform image feature extraction model is used to extract the eigenvalue vector of the image obtained in the first stage; the third stage: the eigenvalue vector extracted in the second stage is calculated CosSim with the eigenvalue vector of the recorded signal waveform image, and the calculation results are sorted in descending order; finally, according to the sorting result, the type of the recorded signal waveform image corresponding to the first place is selected as the recognition result; the specific situation is as follows:

[0082] 4.1) The first stage is called the multi-channel signal waveform image separation and enhancement stage: The input of this stage is the signal waveform image and the waveform color instruction in the JSON key-value pair format. Among them, the color instruction JSON key-value pair format is: {"color1":"R(8bit),G(8bit),B(8bit)", "color2": "R(8bit),G(8bit),B(8bit)"}, where color1 and color2 represent the colors of the two channel signal waveform displays. After receiving the signal waveform image and color instruction, the input image will be color filtered according to the values ​​corresponding to color1 and color2, and the signal waveform display corresponding to the colors of color1 and color2 will be extracted respectively, and saved as two independent images. Then, the two images are grayed and resized to 320×640, so that the image height size is stretched, so that the details of the signal waveform image with small amplitude are enlarged, and the display of the enlarged signal waveform details is enlarged.

[0083] Among them, the specific steps of the multi-channel signal waveform image separation and enhancement stage are as follows:

[0084] 4.1.1) By parsing the color command in JSON format, the "values" corresponding to the "keys" color1 and color2 in the color command are extracted respectively, that is, the red (R), green (G), and blue (B) colors displayed by the signal waveform of each channel. Each channel is 8 bits wide and the value range is 0 to 255.

[0085] 4.1.2) Compare the input waveform signal image according to the color values ​​of color1 and color2 to perform pixel color filtering, so as to separate the signal waveform images into two different color display channels for saving. Further, the color filtering method is as follows:

[0086] Set the red, green, and blue color values ​​corresponding to color1 of the received signal waveform image color instruction to R, G, and B respectively. The red, green, and blue color values ​​of a pixel at a certain position of the input signal waveform image are r, g, and b respectively. Calculate the total color variation value according to the Euclidean distance :

[0087] ;

[0088] right A range relaxation of 200 is performed as a condition for preserving pixel colors.

[0089] Further, based on the above settings, the color filtering process is as follows:

[0090] Traverse all position pixels of the input signal waveform image and perform the following operations on each position pixel:

[0091] if , retain the color value of the pixel at that position; otherwise, set the color of the pixel at that position to r=0, g=0, b=0, that is, black.

[0092] The traversed position pixels are formed into a new image and saved.

[0093] Similarly, the color filtering process of color2 is the same as that of color1, and a new image is also formed and saved.

[0094] 4.1.3) The two signal waveform images saved by color filtering are grayed out respectively, and then the sizes of the two images are transformed into 320×640, so that the image height size is stretched, so that the details of the signal waveform image with small amplitude are magnified, and the display of the details of the signal waveform with small amplitude is expanded.

[0095] 4.2) The second stage is called the signal waveform image feature extraction stage: the two signal waveform images processed in step 4.1.3) are sequentially input into the signal waveform image feature extraction model to sequentially obtain 512 eigenvalue vectors of the two signal waveform images.

[0096] 4.3) The third stage is called the feature comparison stage: after obtaining the eigenvalues ​​of the two signal waveform images, the cosine similarity of these two eigenvalue vectors is calculated with the eigenvalue vectors of the waveform signal images that have been recorded and stored, and the settlement results are sorted from large to small; finally, according to the sorting results, the type of recorded signal waveform image that ranks first is selected as the recognition result.

[0097] Furthermore, the specific steps of the feature comparison stage are as follows:

[0098] by Represents the 512 eigenvalue vectors of the first signal waveform image output by the signal waveform image feature extraction model, Represents the eigenvalue of the stored nth signal waveform image, and the cosine similarity calculation process is as follows:

[0099] ;

[0100] In the formula, Represents the similarity comparison result with the nth signal waveform image feature value that has been recorded. represents the dot product operation of two vectors, and Represents vectors and The Euclidean norm of .

[0101] All you will get Sort by size, select the first The corresponding input signal waveform image types, that is, the corresponding "sine", "square", "triangle", "noise" and "none" labels are output as the recognition result of the first signal waveform image.

[0102] The feature comparison process of the second signal waveform image is the same as that of the first image.

[0103] Among them, this embodiment uses SQLite database as a storage container for entering signal waveform image eigenvalue vectors, and stores: type labels corresponding to signal waveform images, including "sine", "square", "triangle", "noise" and "none", and 512 eigenvalue vectors of signal waveform images. Furthermore, SQLite database can also store other types of signal waveform image features to achieve the expansion of signal waveform image recognition types.

[0104] Furthermore, the eigenvalue vector of the recorded signal waveform image uses the same feature extraction model in the signal waveform image feature extraction stage to ensure the consistency and accuracy of feature extraction when processing and identifying different signal waveform images, thereby ensuring the stability and reliability of the recognition result. Unlike the recognition process, when recording and storing the eigenvalue vector of the signal waveform image, the input is a single-channel signal waveform image, that is, the signal waveform image only displays a single signal waveform. Furthermore, the type of the recorded signal waveform has been determined.

[0105] The specific process of entering the eigenvalue vector of the signal waveform image includes: ① gray-scale the entered signal waveform image and adjust the image size to 320×640; ② input the gray-scaled and resized image into the signal waveform image feature extraction model to extract the 512-value eigenvalue vector of the signal waveform image; ③ enter the extracted 512 eigenvalue vectors and the determined type labels of the signal waveform image, including the "sine", "square", "triangle", "noise", "none", "AM" and "ASK" labels corresponding to the sine wave signal image, square wave signal image, triangle wave signal image, noise signal image, no waveform signal display image, analog amplitude modulation wave signal image and digital amplitude modulation wave signal image respectively, into the SQLite database.

[0106] The experimental results of the three-stage adaptive signal waveform image recognition method of this embodiment are described in detail below:

[0107] According to the final detection results of the model, the performance of the proposed three-stage adaptive signal waveform image recognition method is evaluated from the indicators of accuracy, processing delay, and signal waveform type expansion.

[0108] In order to evaluate the performance of the proposed signal waveform feature image classification model, an ablation experiment was conducted. By comparing with the model after removing the residual structure and SimAM attention mechanism, the impact of each module on the model accuracy and inference delay was analyzed. The comparison results are shown in Table 1:

[0109] Table 1

[0110]

[0111] The results in the above table show that the signal waveform image classification model proposed in the present invention has an accuracy of 98.7% compared to the model without residual structure and SimAM attention mechanism, which is at least 4.6% higher than other models compared in Table 1. The classification accuracy is greatly increased, proving that the signal waveform image feature extraction model obtained after intercepting the model can effectively capture and extract key features in the signal waveform image, ensuring the accuracy and stability of the model in subsequent tasks. In terms of reasoning delay, although the addition of residual structure and attention mechanism delay increases by 11ms, it is less than 100ms, meeting the real-time requirements and satisfying a good user experience. Furthermore, it can be deployed in mobile devices, IoT devices, or edge computing devices.

[0112] Furthermore, in order to evaluate the effectiveness of feature extraction and feature similarity comparison, the accuracy of feature value similarity comparison is compared for five types of signal waveforms, namely, sine wave, square wave, triangle wave, clutter and no waveform, under different numbers of recorded and saved feature value vectors. The comparison results are shown in Table 2:

[0113] Table 2

[0114]

[0115] The results in the above table show that the three-stage adaptive signal waveform image recognition method proposed in the present invention is related to the number of recorded and saved eigenvalue vectors. When 5 eigenvalue vectors of each type of signal waveform image are recorded and saved, the accuracy rate reaches 94.5%; when 20 eigenvalue vectors are recorded and saved, the accuracy rate reaches 97.8%, which proves that a high recognition accuracy rate is achieved when a small number of signal waveform image eigenvalue vectors are recorded. In contrast, although the recognition delay time has increased, it remains below 100ms, meeting the real-time requirements.

[0116] Furthermore, in order to evaluate the signal waveform image recognition expansion capability of the present invention, AM analog amplitude modulation signal waveform image and ASK digital amplitude modulation signal waveform image types are added on the basis of the five signal waveform types of sine wave, square wave, triangle wave, noise wave and no waveform. The comparison results are shown in Table 3:

[0117] Table 3

[0118]

[0119] The results in the above table show that when the signal waveform image type is expanded, the accuracy rate reaches 93.5% when 5 eigenvalue vectors are recorded and saved; when 20 eigenvalue vectors are recorded and saved, the accuracy rate reaches 97.3%. It proves that the method proposed in the present invention can still obtain a higher recognition accuracy rate when a small number of signal waveform image eigenvalue vectors are recorded when the signal waveform image type is increased. In contrast, although the recognition delay time increases slightly, it remains below 100ms, meeting the real-time requirements.

[0120] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A three-stage adaptive signal waveform image recognition method, characterized in that: The following steps are involved: 1) Collect screenshots of the oscilloscope's single-channel signal waveform display as signal waveform images, and preprocess and annotate the images to filter out images of poor quality and images labeled with sine waves, square waves, triangle waves, clutter, and no waveform labels to form the required data set, which is then divided into a training set and a test set; 2) Perform data enhancement on the images in the training set, and then input the enhanced images and corresponding labels into the constructed signal waveform image classification model for training to obtain an optimal model after training; wherein, the model includes 1 initial convolution unit C1, 3 residual blocks R1~R3, 3 SimAM attention mechanism layers, 1 full average pooling layer GAP, 2 ReLU activation function layers and 3 fully connected neural network layers F1~F3, and its connection order is: initial convolution unit C1→first SimAM attention mechanism layer→first residual block R1→second SimAM attention mechanism layer→second residual block R2→third SimAM attention mechanism layer→third residual block R3→full average pooling layer GAP→first fully connected neural network layer F1→first ReLU activation function layer→second fully connected neural network layer F2→second ReLU activation function layer→third fully connected neural network layer F3; 3) The optimal model obtained after training is intercepted, in which the connection from the initial convolution unit C1 to the second fully connected neural network layer F2 is retained as the signal waveform image feature extraction model; 4) After obtaining the signal waveform image feature extraction model, the signal waveform image recognition of the test set is completed in the following three stages. The first stage: the input dual-channel signal waveform image is color filtered CFilter according to the color instruction, so as to separate it into two independent channel signal waveform images, and the two separated signal waveform images are grayed GSale, and the image size is transformed EAmp; The second stage: use the signal waveform image feature extraction model to extract the eigenvalue vector of the image obtained in the first stage; the third stage: perform cosine similarity calculation CosSim on the eigenvalue vector extracted in the second stage and the eigenvalue vector of the recorded signal waveform image, and sort the calculation results in descending order; finally, according to the sorting results, select the type of the recorded signal waveform image corresponding to the first one as the recognition result.

2. A three-stage adaptive signal waveform image recognition method according to claim 1, characterized in that: Step 1) The following steps are involved: 1.1) Data acquisition: Use professional instrument control software, including Virtual Instrument Software Architecture software, or a smart phone to directly take screenshots or photos of different types of signal waveforms displayed by the oscilloscope; connect the oscilloscope with a signal generator, and the generated signal waveforms include sine waves, square waves, and triangle waves, so as to obtain the oscilloscope's display of the above types of signal waveforms; suspend and ground the oscilloscope's waveform acquisition terminal to obtain the oscilloscope's clutter signal display; define the oscilloscope's screenshot or photo without any signal waveform display as no waveform display; when obtaining the signal waveform image, do not specify the signal's amplitude, frequency, and the position of the waveform displayed on the screen; the captured signal waveform images are all single-channel signal waveform images of the oscilloscope, that is, the image only displays a continuous signal waveform; 1.2) Image preprocessing: For each captured image, the signal waveform display area is intercepted and the captured images with poor quality are eliminated, including images with low shooting resolution, severe lighting interference, or blurred and unusable images. After elimination, signal waveform images covering sine waves, square waves, triangle waves, clutter, and no waveform types are obtained; 1.3) Image annotation: According to the captured signal waveform image type, the pre-processed images are divided into five signal waveform image categories: sine wave signal image, square wave signal image, triangle wave signal image, noise signal image and image without waveform signal display, and LabelImg is used to label the images corresponding to the above categories as "sine", "square", "triangle", "noise" and "none" respectively; 1.4) Divide the data set: After the image annotation is completed, all images and the corresponding annotation files are divided into training sets and test sets according to the preset ratio.

3. A three-stage adaptive signal waveform image recognition method according to claim 2, characterized in that, in step 2), the specific situation of the signal waveform image classification model is as follows: The initialization convolution unit C1 includes a 3×3 convolution kernel, edge padding length padding=1, stride=1, a convolution layer with 64 output channels, and a ReLU activation function layer, which realizes the feature conversion and extraction of different dimensions of the input single-channel grayscale signal waveform image; After the initial convolution unit C1 obtains preliminary feature extraction and conversion, the first SimAM attention mechanism layer is connected to distinguish the importance of the influence of different pixels on the features in the feature map, enhance the representation ability of the signal waveform image features, make the model pay more attention to the key areas of the signal waveform image features, and strengthen the expression of the signal waveform image features; for the features extracted by the initial convolution unit C1, the SimAM attention mechanism layer can help the model flexibly allocate the attention weights of the feature map when facing the signal waveform display at different positions, dynamically enhance the feature capture of the signal waveform at different positions, strengthen the expression of key features and effectively suppress interference features, thereby improving the recognition accuracy; After the first SimAM attention mechanism layer, three residual blocks R1 to R3 are connected continuously to ensure the extraction of convolution operation features while alleviating the problem of detail feature loss caused by the reduction of image size in convolution operation. At the same time, the residual connection also alleviates the gradient vanishing problem of deep networks and accelerates the convergence speed of the model. Each residual block includes a main branch and a residual shortcut; the main branch includes 2 convolutional layers and 2 batch normalization layers, and the connection order is convolutional layer → batch normalization layer → convolutional layer → batch normalization layer; at the same time, a residual shortcut is connected between the residual block input and the main branch output of the residual block, which is composed of a convolutional layer → batch normalization layer connection; the input of the first residual block R1 is a 64-channel feature map, and the output is a 128-channel feature map; the input of the second residual block R2 is a 128-channel feature map, and the output is a 256-channel feature map; the input of the second residual block R3 is a 256-channel feature map, and the output is a 512-channel feature map; The outputs of the first residual block R1 and the second residual block R2 are connected to a SimAM attention mechanism layer respectively. By combining the SimAM attention mechanism layer, the attention weight of the feature can be flexibly adjusted, and the features extracted by the residual block can be specifically distinguished and strengthened, further enhancing the robustness of the feature extraction of signal waveforms at different positions. In order to reduce the number of model parameters and speed up the inference of the model, the third residual block R3 is directly input into the full average pooling layer GAP, and the 512 channel feature maps output by the third residual block R3 are abstracted into 512 corresponding average feature values; After obtaining 512 average eigenvalues, the 512 average eigenvalues ​​are input into three fully connected neural network layers F1 to F3 for further extraction and mapping, and converted into specific outputs; wherein, the first fully connected neural network layer F1 has an input of an eigenvalue vector of 512 numerical values, outputs an eigenvalue vector of 1024 numerical values, and then is processed by the ReLU activation function layer; the second fully connected neural network layer F2 has an input of an eigenvalue vector of 1024 numerical values, outputs an eigenvalue vector of 512 numerical values, and then is processed by the ReLU activation function layer; the third fully connected neural network layer F3 has an input of an eigenvalue vector of 512 numerical values, outputs an eigenvalue vector of 5 numerical values, and the sizes of these 5 numerical values ​​respectively represent the possibilities of five types of signal waveform images.

4. A three-stage adaptive signal waveform image recognition method according to claim 3, characterized in that: In step 2), the data enhancement methods include: image grayscale, image translation, image shearing, image stretching and image noise injection, and converting the image size to 320×640.

5. A three-stage adaptive signal waveform image recognition method according to claim 4, characterized in that: In step 3), the signal waveform image feature extraction model uses the 512-value feature value vector output by the second fully connected neural network layer F2 as the signal waveform image feature value expression.

6. A three-stage adaptive signal waveform image recognition method according to claim 5, characterized in that: In step 4), the first stage is called the multi-channel signal waveform image separation and enhancement stage, and its color instruction format is: JSON key-value pair of {"color1":"R(8bit),G(8bit),B(8bit)", "color2": "R(8bit),G(8bit),B(8bit)"}, where color1 and color2 represent the colors displayed by the two channel signal waveforms; its image separation and enhancement includes the following steps: 4.1.1) By parsing the color command in JSON format, extract the "values" corresponding to the "keys" color1 and color2 in the color command, that is, the red, green, and blue colors displayed by the signal waveform of each channel. Each channel is 8 bits wide and the value range is 0 to 255; 4.1.2) Obtain the color values ​​of color1 and color2 according to step 4.1.1) and perform color filtering on the input waveform signal image respectively, separate it into images showing the signal waveform of a single channel and save it; wherein, the color filtering method is as follows: Set the red, green, and blue color values ​​corresponding to color1 of the received signal waveform image color instruction to R, G, and B respectively. The red, green, and blue color values ​​of a pixel at a certain position of the input signal waveform image are r, g, and b respectively. Calculate the total color variation value according to the Euclidean distance : ； right Perform a range relaxation of 200 as a condition for preserving pixel colors; Based on the above settings, the color filtering process is as follows: Traverse all position pixels of the input signal waveform image and perform the following operations on each position pixel: if , retain the color value of the pixel at that position; otherwise, set the color of the pixel at that position to r=0, g=0, b=0, that is, black; The traversed position pixels are formed into a new image and saved; similarly, the color filtering process of color2 is the same as color1, and a new image is also formed and saved; 4.1.3) The two signal waveform images saved by color filtering are grayed out respectively, and then the sizes of the two signal waveform images are adjusted to 320×640, so that the image height size is stretched, so that the details of the signal waveform images with small amplitudes are magnified, and the display of the signal waveform details is expanded.

7. A three-stage adaptive signal waveform image recognition method according to claim 6, characterized in that: In step 4), the second stage is called the signal waveform image feature extraction stage, which includes sequentially inputting the two signal waveform images processed in step 4.1.3) into the signal waveform image feature extraction model to sequentially obtain 512 eigenvalue vectors of the two signal waveform images.

8. A three-stage adaptive signal waveform image recognition method according to claim 7, characterized in that: In step 4), the third stage is called feature comparison stage, which includes: Represents the 512 eigenvalue vectors of the first signal waveform image output by the signal waveform image feature extraction model, represents the eigenvalue of the stored nth signal waveform image, then the cosine similarity calculation process is: ； In the formula, Represents the similarity comparison result with the stored nth signal waveform image feature value, represents the dot product operation of two vectors, and Represents vectors and The Euclidean norm of .

9. A three-stage adaptive signal waveform image recognition method according to claim 8, characterized in that: In step 4), it is necessary to have the eigenvalue vector of the recorded signal waveform image. When recording the eigenvalue vector of the signal waveform image, an SQLite database is used as a storage container for the recorded signal waveform image eigenvalue vector, and the storage includes: type labels corresponding to the signal waveform image, including "sine", "square", "triangle", "noise" and "none", and 512 eigenvalue vectors of the signal waveform image; the SQLite database can also store other types of signal waveform image features to achieve the expansion of the signal waveform image recognition type; the eigenvalue vector of the recorded signal waveform image uses the same feature extraction model in the signal waveform image feature extraction stage. By using the same feature extraction model, the consistency and accuracy of feature extraction when processing and identifying different signal waveform images are ensured, thereby ensuring the stability and reliability of the recognition result; when recording and storing the signal waveform image eigenvalue vector, the input is a single-channel signal waveform image, that is, the signal waveform image only displays a single signal waveform, and the recorded signal waveform type has been determined.

10. A three-stage adaptive signal waveform image recognition method according to claim 9, characterized in that: In step 4), the specific process of pre-entering the signal waveform image eigenvalue vector includes: ① gray-scale the entered signal waveform image and adjust the image size to 320×640; ② input the gray-scaled and resized image into the signal waveform image feature extraction model to extract 512 numerical eigenvalue vectors of the signal waveform image; ③ enter the extracted 512 eigenvalue vectors and the determined type labels of the signal waveform image, including the "sine", "square", "triangle", "noise", "none", "AM" and "ASK" labels corresponding to the sine wave signal image, square wave signal image, triangle wave signal image, noise signal image, no waveform signal display image, analog amplitude modulation wave signal image and digital amplitude modulation wave signal image, into the SQLite database.

Images