An instrument identification method and system fusing semantic and visual information
By integrating semantic and visual information, constructing an adjacency matrix using multiple neural network models, optimizing character image feature representation, and automatically grouping characters, the problem of misjudgment in instrument identification in power enterprises is solved, and efficient and accurate automated identification of instrument data is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CSG EHV POWER TRANSMISSION
- Filing Date
- 2023-06-20
- Publication Date
- 2026-06-26
Smart Images
Figure CN116704512B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image recognition technology, and in particular to an instrument recognition method and system that integrates semantic and visual information. Background Technology
[0002] With the rapid development of the power industry, the scale of power grids and the relationship between power supply and demand have become increasingly complex. Power companies need to collect, analyze, and manage large amounts of instrument data to cope with this complexity. Traditional instrument identification methods involve manual input, which is prone to errors. Furthermore, manual input requires identifying each instrument individually, making the work difficult and demanding a high level of expertise and experience. For power systems requiring real-time monitoring, there is an increasing preference for automated instrument data identification.
[0003] Existing OCR (Optical Character Recognition) technology is a technique that converts text in images into editable text. Its basic principle is to analyze, recognize, and convert the characters in an image into computer-readable text. However, OCR technology also faces some challenges in the instrument recognition tasks of power companies.
[0004] In instrument recognition tasks, it is necessary to identify the readings displayed on instruments. Most existing datasets are based on word-level annotation, so the trained models directly output word-level text recognition results. Because two readings may be close together, the detector may misclassify them as a single word, resulting in incorrect readings. Manually annotating images at the character level is labor-intensive and makes accurate instrument data recognition difficult. Summary of the Invention
[0005] Based on the above analysis, the embodiments of the present invention aim to provide an instrument recognition method and system that integrates semantic and visual information to solve the problem that multiple readings that are close to each other on the instrument screen are misjudged as belonging to the same reading.
[0006] On one hand, embodiments of the present invention provide an instrument recognition method that integrates semantic and visual information, comprising the following steps:
[0007] The screen image of the instrument display is acquired and preprocessed to obtain the image to be recognized;
[0008] Input the image to be recognized into the trained character recognition network, and output the coordinates of each character box and the character label in the image to be recognized;
[0009] Using the character images corresponding to each character box as nodes, a visual graph and a semantic graph are constructed. The similarity between the feature vectors of each node is used as the edge weight of the visual graph, and the similarity between the character attributes of each node is used as the edge weight of the semantic graph. The edge weights of the visual graph and the semantic graph are fused to construct an adjacency matrix. Based on the feature matrix composed of the feature vectors of each node, the adjacency matrix, and the character labels corresponding to each node, a graph neural network is trained to obtain the trained feature matrix.
[0010] Based on the distance between the feature vectors in the trained feature matrix, the character images are grouped, and the character labels of each character image are output for each group to obtain the recognition results.
[0011] Based on a further improvement of the above method, the character recognition network includes multiple neural network models. The step of inputting the image to be recognized into the trained character recognition network and outputting the coordinates of each character frame and the character label in the image to be recognized includes:
[0012] The image to be recognized is fed into the first neural network model, the text region is extracted, the image to be recognized is segmented according to the text region, and the text box image and its coordinates are obtained.
[0013] After scaling the text box image to the same size, it is fed into the second neural network model to identify the character label sequence in the text box image;
[0014] Based on the number of character labels in each text box image and the coordinates of each text box, calculate the center coordinates of each character in each text box; based on the center coordinates of each character, obtain the coordinates of each character box and the character label in each text box.
[0015] Based on the above method, the center coordinates of each character in each text box are calculated using the following formula:
[0016]
[0017] Among them, (x' n ,y' n (x0, y1) represents the center coordinates of the nth character in each text box, and (x1, y0) and (x0, y1) represent the coordinates of the top left and bottom right corners of the text box, respectively.
[0018] Based on the further improvement of the above method, the image to be recognized is fed into the first neural network model to extract the text region, including:
[0019] The feature map of the image to be recognized is extracted by the convolutional neural network; the feature vectors corresponding to each window are obtained by sliding according to the preset window, and the feature matrix is formed and then fed into the bidirectional long short-term memory network to recognize each text region of each image to be recognized.
[0020] Based on further improvements to the above methods, the second neural network model is a CNN-LSTM model based on the attention mechanism.
[0021] Based on the further improvement of the above method, the feature vectors of each node are extracted according to the convolutional neural network, and the similarity between the feature vectors of each node is calculated using cosine similarity.
[0022] Based on the further improvement of the above method, the similarity between the character attributes of each node is obtained through the following steps: according to the instrument display screen and character coordinates of the image to be identified to which each node belongs, the corresponding font, color and font size attribute values are obtained, and attribute vectors are formed by encoding each attribute value. Cosine similarity is used to calculate the similarity between the attribute vectors of each node.
[0023] Based on the above method, further improvements are made by fusing the edge weights of the visual graph and the semantic graph to construct an adjacency matrix, including:
[0024] The edge weights of the visual graph are mapped to the element values of the corresponding nodes in the visual adjacency matrix; the edge weights of the semantic graph are mapped to the element values of the corresponding nodes in the semantic adjacency matrix; the element values of the same nodes in the visual adjacency matrix and the semantic adjacency matrix are superimposed according to a preset ratio, and the final adjacency matrix is obtained based on the comparison result of the superimposed element values and the preset neighbor threshold.
[0025] Based on a further improvement of the above method, the screen image of the instrument display is preprocessed to obtain the image to be recognized, including:
[0026] The screen image is noise-removed by sequentially using a neighborhood averaging smoothing filter and a wavelet denoising method; then, a perspective transformation is performed on the denoised screen image according to the perspective transformation matrix to obtain the corrected screen image; finally, the corrected screen image is scaled to the same size to obtain the image to be recognized.
[0027] On the other hand, embodiments of the present invention provide an instrument recognition system that integrates semantic and visual information, including:
[0028] The image acquisition module is used to acquire and preprocess the screen image of the instrument display to obtain the image to be recognized;
[0029] The character annotation module is used to input the image to be recognized into the trained character recognition network and output the coordinates of each character box and the character label in the image to be recognized.
[0030] The feature extraction module is used to construct a visual graph and a semantic graph using the character images corresponding to each character box as nodes. The similarity between the feature vectors of each node is used as the edge weight of the visual graph, and the similarity between the character attributes of each node is used as the edge weight of the semantic graph. An adjacency matrix is constructed based on the edge weights of the visual graph and the semantic graph. Based on the feature matrix composed of the feature vectors of each node, the adjacency matrix, and the character labels corresponding to each node, a graph neural network is trained to obtain the trained feature matrix.
[0031] The result recognition module is used to group character images according to the distance between each feature vector in the trained feature matrix, and output the character labels of each character image in each group to obtain the recognition result.
[0032] Compared with existing technologies, the present invention can achieve at least one of the following beneficial effects: by recognizing each character in the text through multiple neural network models, it can achieve automatic character labeling and reduce manual costs; by constructing a dual-graph model with individual character images as nodes and visual and attribute features as edges, it can optimize the feature representation of character images, obtain a feature matrix that integrates visual and attribute features, overcome interference factors during image acquisition, improve the accuracy of character grouping, and thus solve the problem that readings that are close to each other will be misjudged as the same number, thereby improving the adaptability and accuracy of automated identification instrument data.
[0033] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description
[0034] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.
[0035] Figure 1 This is a flowchart of an instrument recognition method that integrates semantic and visual information according to Embodiment 1 of the present invention;
[0036] Figure 2 This is a schematic diagram illustrating the calculation of the coordinates of the character center point in Embodiment 1 of the present invention. Detailed Implementation
[0037] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.
[0038] Example 1
[0039] A specific embodiment of the present invention discloses an instrument recognition method that integrates semantic and visual information, such as... Figure 1 As shown, it includes the following steps:
[0040] S11. Acquire and preprocess the screen image of the instrument display to obtain the image to be recognized.
[0041] It should be noted that in this embodiment, the instrument display screen refers to the display of various workstation equipment in the power company. The workstation is used to monitor the operation of the power system, and the workstation display screen will show the current system operation data in real time. Since there are many parameters on the system interface monitored by the workstation, and the numbers are of different values, the position, angle and focal length of the camera are adjusted according to the distribution characteristics of the numbers on the display screen. The camera captures the screen image to obtain a complete and clear original image, and establishes a connection with the corresponding instrument display screen.
[0042] Because the camera has a wide-angle lens, the raw images captured suffer from barrel distortion. Combined with variations in lighting, this interferes with subsequent automatic recognition. Therefore, the raw images undergo the following preprocessing:
[0043] ① The screen image is noise-removed by sequentially using a neighborhood averaging smoothing filter and a wavelet denoising method.
[0044] It should be noted that the original image is first smoothed using a neighborhood averaging smoothing filter to remove some noise. Then, wavelet denoising is used to further suppress noise in the smoothed image while preserving the image's details and edge features.
[0045] ② Based on the perspective transformation matrix, perform perspective transformation on the denoised screen image to obtain the corrected screen image.
[0046] It should be noted that the four vertices of the instrument display screen corresponding to the screen image are used as reference points. The perspective transformation matrix is calculated based on the coordinates of these reference points on both the instrument display screen and the screen image. For example, the `getPerspectiveTransform` function from the Python OpenCV library is used, taking the coordinates of the reference points on both the instrument display screen and the acquired screen image as input, to calculate the perspective transformation matrix. Then, the `warpPerspective` function from the OpenCV library is used to apply the calculated perspective transformation matrix to the entire denoised screen image, thereby correcting barrel distortion.
[0047] ③ Scale the corrected screen image to the same size to complete the preprocessing.
[0048] It should be noted that in practical applications, the original size of the preprocessed screen images may not be consistent, but they are all scaled to the same size, and the same size as the training samples for the subsequent character recognition network. For example, the corrected screen images are uniformly scaled to 224×224.
[0049] S12. Input the image to be recognized into the trained character recognition network, and output the coordinates of each character box and the character label in the image to be recognized.
[0050] It should be noted that the character recognition network includes multiple neural network models, and each neural network model is composed of a combination of multiple neural networks. Step S12 is further subdivided into steps S121-S123, which will be explained in detail below.
[0051] S121. Input the image to be recognized into the first neural network model, extract the text region, segment the image to be recognized according to the text region, and obtain the text box image and its coordinates.
[0052] Specifically, the first neural network model sequentially includes a convolutional neural network, a sliding window module, and a bidirectional long short-term memory network. The convolutional neural network extracts feature maps from the image, the sliding window module selects feature vectors from the feature maps by sliding a window, and passes these vectors to the bidirectional long short-term memory network module. The bidirectional long short-term memory network learns sequence features and determines whether each feature vector belongs to a character. Preferably, the convolutional neural network module uses a VGG16 network, and the window size is in the range of 3×3 to 5×5.
[0053] It should be noted that the first neural network model is trained using publicly available datasets, such as ICDAR2013 and ICDAR2015, which cover the characters and numbers to be identified in this embodiment. In the annotation files of the public datasets, each line represents a text target. For example, in the annotation file, 22249113286“The” indicates that the top-left coordinate of the text “The” is (22, 249) and the bottom-right coordinate is (113, 286). A training set is constructed using the public datasets and their annotations, and the first neural network model is trained to identify the text regions in each image in the training set.
[0054] After the first neural network model is trained, the feature map of the image to be recognized is extracted by the convolutional neural network; the feature vectors corresponding to each window are obtained by sliding according to the preset window, and the feature matrix is formed and then fed into the bidirectional long short-term memory network to identify each text region of each image to be recognized.
[0055] Image segmentation technology is used to crop out each text region in each image to be recognized, and obtain the text box image and its coordinates corresponding to each text region.
[0056] It is worth noting that the text in the text area in this embodiment is at the word level, that is, each text area includes one or more characters, and the size of each text area is not necessarily the same.
[0057] S122. After scaling the text box image to the same size, it is fed into the second neural network model to identify the character label sequence in the text box image.
[0058] It should be noted that the second neural network model is a CNN-LSTM model based on an attention mechanism, which consists of a Convolutional Neural Network (CNN), a sliding window module, and a Long Short-Term Memory (LSTM) network with an attention mechanism. The CNN extracts feature maps from the image, while the sliding window module selects feature vectors from these maps by sliding a window. These feature vectors are then combined into a feature matrix and fed into the LSTM network module for encoding and decoding the feature sequence, outputting a label sequence that identifies the characters in each image.
[0059] When training the second neural network model, the same training samples as the first neural network model are used, and the window size in the sliding window module is the same as the window size in the first neural network model.
[0060] After the second neural network model is trained, it takes a text box image scaled to the same size as input and outputs a sequence of character labels for each text box image.
[0061] S123. Calculate the center coordinates of each character in each text box based on the number of character labels in each text box image and the coordinates of each text box; obtain the coordinates of each character box and the character label in each text box based on the center coordinates of each character.
[0062] It should be noted that, as Figure 2 As shown, assuming all characters in each text box have the same width, the center coordinates of each character in each text box are calculated using the following formula:
[0063]
[0064] Among them, (x' n ,y' n (x0, y1) represents the center coordinates of the nth character in each text box, and (x1, y0) and (x0, y1) represent the coordinates of the top left and bottom right corners of the text box, respectively.
[0065] Based on the center coordinates of each character, and assuming the character frame is a rectangle, obtain the coordinates of the top left and bottom right corners of each character frame, as well as the character label.
[0066] Compared with existing technologies, this step uses multiple neural network models to first perform word-level text recognition, and then uses an automated word segmentation method to identify each character in the text, thereby achieving automatic character labeling and reducing manual costs.
[0067] S13. Using the character images corresponding to each character box as nodes, construct a visual graph and a semantic graph. Use the similarity between the feature vectors of each node as the edge weights of the visual graph and the similarity between the character attributes of each node as the edge weights of the semantic graph. Combine the edge weights of the visual graph and the semantic graph to construct an adjacency matrix. Train the graph neural network based on the feature matrix composed of the feature vectors of each node, the adjacency matrix, and the character labels corresponding to each node to obtain the trained feature matrix.
[0068] It should be noted that in the instrument recognition task, the main object of detection is numbers. A misjudgment will result in an incorrect reading. For example, if the numbers 11 and 0.1 are close together in the acquired image, misidentifying them as a single number 110.1 will lead to a serious error. This embodiment uses each character image as a node and visual features and attribute features as edges to construct a bi-graph model.
[0069] Furthermore, based on the coordinates of each character frame, the corresponding image to be recognized is segmented to obtain the image of each character. The visual graph G in the bi-graph model... V ={N,ε V This illustrates the visual relationships between character images, where N represents the set of nodes, with each node representing a character image; every two nodes form an edge, and the edge weight... This represents the visual similarity between the i-th and j-th character images.
[0070] Specifically, based on a convolutional neural network, feature vectors of each character image are extracted. For example, a ResNet-50 network is used, which is pre-trained using the publicly available ImageNet dataset. Then, cosine similarity is used to calculate the similarity between each feature vector to represent the similarity between characters. That is, the similarity between the feature vectors of each node is used as the edge weights of the visual graph, as shown in the following formula:
[0071]
[0072] Among them, V i and V j Let these represent the feature vectors of the i-th and j-th character images, respectively. Let represent the transpose of the feature vector of the j-th character image.
[0073] Furthermore, the semantic graph G in the bi-graph model M ={N,ε MThe left margin displays the semantic relationships between character attributes and the edge weights. M represents the consistency of character attributes between the i-th and j-th character images. i and M j Let i and j represent the character attribute vectors of the i-th and j-th character images, respectively.
[0074] It should be noted that character attributes include font, color, and font size. Since the captured screen image is scaled, it cannot accurately reflect the font size. Therefore, in this embodiment, the font size of the characters is divided into three types: small, medium, and large.
[0075] All attribute values for font, color, and font size are combined into an attribute vector. By default, the encoding of all attribute values is 0. Based on the actual character attributes obtained, the corresponding encoding is set to 1. For example, the attribute vector includes: bold, black, dark gray, gray, large, medium, and small. If a character has a bold font, a dark gray color, and a medium font size, then the combined attribute vector is: 10010010.
[0076] It should be noted that since the screen interfaces of each instrument display in a power company do not change frequently, i.e., the system interfaces monitored by each workstation do not change frequently, the coordinate range and character attributes of each area are recorded for each system interface. Therefore, based on the instrument display corresponding to the image to be recognized to which each character image belongs, the coordinate range and character attributes of each corresponding area are obtained, and the character attributes are automatically obtained based on the coordinates of the characters in the character image. When the system interface of the instrument display changes, the coordinate range and character attributes of each area are updated in a timely manner.
[0077] The similarity between attribute vectors of each node is also calculated using cosine similarity as the edge weight of the semantic graph. The greater the similarity of the attribute vectors, the higher the degree of attribute consistency between the two images.
[0078] It should be noted that images with high visual similarity or high attribute consistency often come from the same word-level text box. Therefore, by fusing the edge weights of the visual graph and the semantic graph, the similarity between two character images is defined as the sum of visual similarity and attribute consistency, and an adjacency matrix is constructed.
[0079] Specifically, the edge weights of the visual graph are mapped to the visual adjacency matrix A. V The element value of the corresponding node Map the edge weights of the semantic graph to a semantic adjacency matrix A. M The element value of the corresponding node Using formula (3), the element values A' of the same nodes in the visual adjacency matrix and the semantic adjacency matrix are superimposed according to a preset ratio. ijUsing formula (4), the final adjacency matrix A is obtained based on the comparison between the superimposed element values and the preset neighbor threshold r:
[0080]
[0081]
[0082] Where β is a preset ratio value, that is, the ratio of visual to semantic is 1:β, and A in the adjacency matrix A ij =1 indicates that the i-th and j-th character images are neighbors.
[0083] Based on the feature matrix composed of the feature vectors of each node, the adjacency matrix, and the character labels corresponding to each node, a graph neural network is trained to obtain the trained feature matrix.
[0084] It should be noted that the character labels corresponding to each node are the labels automatically identified in step S12, which are used as the true results; the feature matrix composed of the feature vectors of each node and the adjacency matrix calculated according to formula (4) are used as initial values and are fed into the graph neural network along with the character labels corresponding to each node. In each training process, the graph neural network updates the feature vector of the corresponding node according to the feature vector of the neighboring nodes of each node to obtain the updated feature matrix, and then updates the adjacency matrix according to formulas (3) and (4); the character labels output by the classification layer in the graph neural network are used as the prediction results. Based on the difference between the true results and the prediction results, the graph neural network is trained through the cross-entropy loss function, the parameters of the graph neural network are updated, and the prediction accuracy is calculated. Therefore, the feature matrix and the adjacency matrix will change as the number of training iterations increases. When the training reaches the maximum number of iterations or the accuracy meets the requirements, the training stops. At this time, the feature matrix optimizes the feature representation of the character image and can accurately identify the grouping information of the characters.
[0085] S14. Based on the distance between each feature vector in the trained feature matrix, group the character images and output the character labels of each character image in each group to obtain the recognition results.
[0086] It should be noted that, based on the calculated feature matrix after training, the Euclidean distance between any two feature vectors is calculated and compared with a preset threshold. If the Euclidean distance is less than the preset threshold, the character images corresponding to the feature vectors belong to the same group; otherwise, they do not belong to the same group.
[0087] Based on the grouping information and the character labels of each character image obtained in step S12, the characters are output in groups to obtain the recognition results.
[0088] Compared with existing technologies, this embodiment provides an instrument recognition method that integrates semantic and visual information. It identifies each character in the text through multiple neural network models, realizes automatic character labeling, and reduces manual costs. By constructing a dual-graph model with individual character images as nodes and visual and attribute features as edges, it optimizes the feature representation of character images, obtains a feature matrix that integrates visual and attribute features, overcomes interference factors during image acquisition, improves the accuracy of character grouping, and solves the problem that readings that are close to each other may be misjudged as the same number, thereby improving the adaptability and accuracy of automated instrument data recognition.
[0089] Example 2
[0090] Another embodiment of the present invention discloses an instrument recognition system that integrates semantic and visual information, thereby implementing the instrument recognition method that integrates semantic and visual information in Embodiment 1. The specific implementation of each module is described in the corresponding description in Embodiment 1. The system includes:
[0091] The image acquisition module is used to acquire and preprocess the screen image of the instrument display to obtain the image to be recognized;
[0092] The character annotation module is used to input the image to be recognized into the trained character recognition network and output the coordinates of each character box and the character label in the image to be recognized.
[0093] The feature extraction module is used to construct a visual graph and a semantic graph using the character images corresponding to each character box as nodes. The similarity between the feature vectors of each node is used as the edge weight of the visual graph, and the similarity between the character attributes of each node is used as the edge weight of the semantic graph. An adjacency matrix is constructed based on the edge weights of the visual graph and the semantic graph. Based on the feature matrix composed of the feature vectors of each node, the adjacency matrix, and the character labels corresponding to each node, a graph neural network is trained to obtain the trained feature matrix.
[0094] The result recognition module is used to group character images according to the distance between each feature vector in the trained feature matrix, and output the character labels of each character image in each group to obtain the recognition result.
[0095] Since this embodiment and the aforementioned instrument recognition method that integrates semantic and visual information can be mutually referenced, and this is a repetition of the previous description, they will not be repeated here. Because this system embodiment shares the same principle as the above method embodiment, it also possesses the corresponding technical effects of the above method embodiment.
[0096] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.
[0097] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for instrument recognition that integrates semantic and visual information, characterized in that, Includes the following steps: The screen image of the instrument display is acquired and preprocessed to obtain the image to be recognized; Input the image to be recognized into the trained character recognition network, and output the coordinates of each character box and the character label in the image to be recognized; Using the character images corresponding to each character frame as nodes, a visual graph and a semantic graph are constructed. The similarity between the feature vectors of each node is used as the edge weight of the visual graph, and the similarity between the character attributes of each node is used as the edge weight of the semantic graph. Character attributes include font, color, and font size. The edge weights of the visual graph and the semantic graph are fused to construct an adjacency matrix. Based on the feature matrix composed of the feature vectors of each node, the adjacency matrix, and the character labels corresponding to each node, a graph neural network is trained. In each training iteration, the graph neural network updates the feature vector of the corresponding node based on the feature vectors of the neighboring nodes of each node to obtain the updated feature matrix. Then, the adjacency matrix is updated according to the method of fusing the edge weights of the visual graph and the semantic graph until training stops and the trained feature matrix is obtained. The method of fusing the edge weights of the visual graph and the semantic graph to construct an adjacency matrix includes: mapping the edge weights of the visual graph to the element values of the corresponding nodes in the visual adjacency matrix; mapping the edge weights of the semantic graph to the element values of the corresponding nodes in the semantic adjacency matrix; superimposing the element values of the same nodes in the visual adjacency matrix and the semantic adjacency matrix at a preset ratio; and obtaining the final adjacency matrix based on the comparison result of the superimposed element values and a preset neighbor threshold. Based on the distance between the feature vectors in the trained feature matrix, the character images are grouped, and the character labels of each character image are output for each group to obtain the recognition results.
2. The instrument recognition method integrating semantic and visual information according to claim 1, characterized in that, The character recognition network includes multiple neural network models. The process of inputting the image to be recognized into the trained character recognition network and outputting the coordinates of each character frame and the character label in the image includes: The image to be recognized is fed into the first neural network model, the text region is extracted, the image to be recognized is segmented according to the text region, and the text box image and its coordinates are obtained. After scaling the text box image to the same size, it is fed into the second neural network model to identify the character label sequence in the text box image; Based on the number of character labels in each text box image and the coordinates of each text box, calculate the center coordinates of each character in each text box; based on the center coordinates of each character, obtain the coordinates of each character box and the character label in each text box.
3. The instrument recognition method integrating semantic and visual information according to claim 2, characterized in that, The center coordinates of each character in each text box can be calculated using the following formula: in, This indicates the first text box. n The center coordinates of each character and These represent the coordinates of the top left and bottom right corners of the text box, respectively.
4. The instrument recognition method integrating semantic and visual information according to claim 2, characterized in that, The step of inputting the image to be recognized into the first neural network model and extracting the text region includes: The feature map of the image to be recognized is extracted by the convolutional neural network; the feature vectors corresponding to each window are obtained by sliding according to the preset window, and the feature matrix is formed and then fed into the bidirectional long short-term memory network to recognize each text region of each image to be recognized.
5. The instrument recognition method integrating semantic and visual information according to claim 2, characterized in that, The second neural network model is a CNN-LSTM model based on the attention mechanism.
6. The instrument recognition method integrating semantic and visual information according to claim 1 or 2, characterized in that, The feature vectors of each node are extracted using a convolutional neural network, and the similarity between the feature vectors of each node is calculated using cosine similarity.
7. The instrument recognition method integrating semantic and visual information according to claim 1 or 2, characterized in that, The similarity between the character attributes of each node is obtained through the following steps: based on the instrument display screen and character coordinates corresponding to the image to be identified of each node, the corresponding font, color and font size attribute values are obtained, and attribute vectors are formed by encoding each attribute value. Cosine similarity is used to calculate the similarity between the attribute vectors of each node.
8. The instrument recognition method integrating semantic and visual information according to claim 1, characterized in that, The preprocessing of the instrument display screen image yields the image to be recognized, including: The screen image is noise-removed by sequentially using a neighborhood averaging smoothing filter and a wavelet denoising method; then, a perspective transformation is performed on the denoised screen image according to the perspective transformation matrix to obtain the corrected screen image; finally, the corrected screen image is scaled to the same size to obtain the image to be recognized.
9. An instrument recognition system that integrates semantic and visual information, characterized in that, include: The image acquisition module is used to acquire and preprocess the screen image of the instrument display to obtain the image to be recognized; The character annotation module is used to input the image to be recognized into the trained character recognition network and output the coordinates of each character box and the character label in the image to be recognized. The feature extraction module is used to construct a visual graph and a semantic graph using the character images corresponding to each character box as nodes. The similarity between the feature vectors of each node is used as the edge weight of the visual graph, and the similarity between the character attributes of each node is used as the edge weight of the semantic graph. The character attributes include font, color, and font size. An adjacency matrix is constructed based on the edge weights of the visual graph and the semantic graph. The graph neural network is trained based on the feature matrix composed of the feature vectors of each node, the adjacency matrix, and the character labels corresponding to each node. In each training iteration, the graph neural network updates the feature vector of the corresponding node based on the feature vectors of the neighboring nodes of each node to obtain the updated feature matrix. The adjacency matrix is then updated according to the method of fusing the edge weights of the visual graph and the semantic graph until training stops and the trained feature matrix is obtained. The method of fusing the edge weights of the visual graph and the semantic graph to construct an adjacency matrix includes: mapping the edge weights of the visual graph to the element values of the corresponding nodes in the visual adjacency matrix; mapping the edge weights of the semantic graph to the element values of the corresponding nodes in the semantic adjacency matrix; superimposing the element values of the same nodes in the visual adjacency matrix and the semantic adjacency matrix at a preset ratio; and obtaining the final adjacency matrix based on the comparison result of the superimposed element values and a preset neighbor threshold. The result recognition module is used to group character images according to the distance between each feature vector in the trained feature matrix, and output the character labels of each character image in each group to obtain the recognition result.