A highly robust elevator button detection method based on improved YOLOv8 network and graph convolutional network inference

Abstract

The present invention discloses a highly robust elevator button detection method based on an improved YOLOv8 network and graph convolutional network reasoning. By comprehensively using the YOLOv8 target detection model, optical character recognition technology and graph convolutional network, efficient and accurate detection of elevator buttons is achieved. The method first collects images of different elevator button panels to construct a data set, then uses the YOLOv8 model for detection, and combines optical character recognition technology to recognize the text information on the button. Then, the elevator button panel is modeled using a graph convolutional network, and the nodes in the graph are classified, thereby achieving accurate recognition and classification of elevator buttons. The comprehensive application of this method improves the accuracy and robustness of elevator button detection, and provides reliable technical support for the elevator riding and operation tasks of mobile robots.

Description

Technical Field

[0001] The present invention relates to a highly robust elevator button detection method, and in particular to an elevator button detection method based on YOLOv8 and graph convolutional network reasoning. Background Art

[0002] Mobile robots that provide cross-floor services are widely used in office buildings, shopping malls, hotels and other places. Taking the elevator to achieve cross-floor services is the current mainstream technical approach. Although the more mature solution is to solve the problem of robots taking elevators through wireless communication and access to elevator control protocols, this solution has the disadvantages of high cost of elevator modification and fixed elevator riding. Therefore, research on mobile robots that can autonomously take and operate elevators like humans still has important practical value. In order to realize the autonomous riding and operation of robots in elevators, a series of technical problems need to be solved: such as free movement in narrow elevator cars, accurate detection and recognition of elevator buttons, behavior planning for entering and exiting elevators, understanding of the elevator car environment, button control, accurate judgment of floors and elevator operating status, etc., and accurate detection and recognition of elevator buttons is one of the core problems that must be solved. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides an elevator button detection method based on the YOLOv8 target detection model and optical character recognition, as well as graph convolutional network reasoning. The present invention comprehensively utilizes the improved YOLOv8 target detection model and optical character recognition, as well as graph convolutional network reasoning, and can accurately detect and identify elevator buttons.

[0004] The technical solution of the present invention to solve the above problem comprises the following steps:

[0005] (1) Collect a large amount of image information of different elevator button panels to build a dataset;

[0006] (2) building a YOLOv8 network structure and using the data set to train the YOLOv8 network structure;

[0007] (3) building a graph convolutional network structure, and using the data set to train the graph convolutional network structure;

[0008] (4) Use the trained YOLOv8 model to detect the actual elevator image and use the model to detect the location of the elevator buttons in the image;

[0009] (5) cropping the elevator button detected in step (4), that is, cutting out the area where the button is located from the original image to obtain a button image block;

[0010] (6) For each button image block obtained by cutting in step (5), the characters on the image block are recognized using an optical character recognition (OCR) model to obtain the button label and the corresponding confidence level.

[0011] (7) Use the trained graph convolution model and the detection results of YOLOv8 in step (4) to model and predict the actual elevator image. If the confidence of the optical character recognition in step (6) is less than 0.7, replace the category of the optical character recognition with the key category predicted by the graph convolution model;

[0012] Furthermore, in step (1), the following steps are specifically included:

[0013] Step (1.1), collect a large number of different elevator button panel images. These images should cover as many types, styles and layouts of elevator button panels as possible to ensure the diversity and representativeness of the dataset. In the process of collecting images, different lighting conditions, angles and background environments need to be considered so that the trained model has better robustness and generalization ability. Use cameras, online image searches, etc. to obtain these elevator button panel images;

[0014] Step (1.2), preprocess the image, including but not limited to: rotation, translation, scaling, perspective transformation, upside down flipping, left-right flipping and brightness adjustment. These preprocessing operations help increase the diversity of the data set and also reduce the sensitivity of the model to image changes. Through these operations, the elevator button panel images under different angles, scales and lighting conditions can be simulated, thereby improving the robustness of the model;

[0015] Step (1.3), use the dataset annotation software Roboflow to annotate each image. The annotation content includes the category information and coordinate information of the elevator buttons in each image. The category information indicates the category to which each button belongs, such as numbers, letters, function buttons, etc.; while the coordinate information marks the position and size of each button in the image.

[0016] Step (1.4), set the dataset to training set and test set in a ratio of 7:3. The training set is used to train the model, while the test set is used to evaluate the performance and generalization ability of the model.

[0017] Furthermore, in step (2), the following steps are specifically included:

[0018] Step (2.1), the YOLOv8 network structure is mainly composed of three parts: Backbone, Neck, and Head. In the Backbone part, the components are mainly composed of CSP, C2f, SPP / SPPF, etc. Among them, the CSP (Cross Stage Partial) structure improves the efficiency of feature extraction through cross-stage partial connections and local connections, reduces the number of parameters and calculations, and enhances the nonlinear ability of the network. C2f (CSP to FPN) is a conversion module from CSP feature map to FPN, and C2f is improved by SPD-Conv, converting the spatial dimension of the feature map into a depth dimension, and saving more information by increasing the number of channels. This process avoids information loss and captures more detailed features, thereby improving the performance of the network when processing complex tasks. The Neck part is mainly composed of FPN and PAN structures. FPN is a top-down feature pyramid structure that enhances the entire pyramid by passing strong semantic features at high levels downward. The PAN structure adds a bottom-up pyramid behind FPN to supplement the positioning information transmission of FPN, so that the pyramid formed by FPN and PAN has both semantic information and positioning information. In the Head part, three detectors of different scales are generated according to the different target sizes.

[0019] Step (2.2), adopt the Multi-scale training (0.5~1.5x) strategy, that is, during the training process, assuming that the size of the input image is 640×640, the size of the random value is between 0.5×640 and 1.5×640, but the value is an integer multiple of 32. Such a strategy helps to improve the generalization ability of the network, so that it can better adapt to targets of different sizes. At the same time, the Warmup and Cosine LR scheduler strategy is adopted, that is, warm up before training, gradually increase the learning rate, and then adopt the Cosine learning rate reduction strategy. This strategy can make the training more stable, help the network converge to the optimal solution faster, and improve the efficiency and accuracy of training.

[0020] Furthermore, in step (3), the following steps are specifically included:

[0021] Step (3.1), elevator button image feature extraction, use the feature extraction network to extract image features. According to the rectangular box marked by the data set in step (1), the elevator button image block is cropped and input into the feature extraction network to extract the feature map. The extracted image features are convolved with a 1×1 convolution kernel with 1280 channels and flattened into a one-dimensional vector with a length of 1280. After feature extraction and linear vector transformation, the image features of the button are obtained.

[0022] Step (3.2), elevator button distance feature extraction, the distance feature of the button is composed of its relative distance to its nearest neighbor in the four main directions (left, right, up, and down). All values ​​are normalized by the possible maximum value so that the absolute value of these four values ​​is always less than 1. By splicing these four normalized distances, we get the position feature of the button. The image feature and position feature of the button are spliced ​​to get its overall feature.

[0023] Step (3.3) models the elevator button panel, which is represented as a graph. Edges are drawn between the elevator button and its four nearest neighbor elevator buttons. The button is used as a node, and the edge represents the nearest neighbor of the button in the four main directions. The representation of the elevator button panel graph is obtained.

[0024] Step (3.4) classifies the nodes in the graph. The present invention uses a graph convolutional neural network (GCN). The GCN model used in the present invention combines a message passing network (MPNN) and an attention network (GAT). The network is a combination of three layers of graph attention network convolution (GAtConv) and linear layers, and uses a linear rectifier function (ReLU) as the activation function. In the last layer, we perform a discard process to prevent the signal from entering the next layer with a certain probability to prevent overfitting.

[0025] The beneficial effects of the present invention are:

[0026] 1. The present invention uses a combination of an improved YOLOv8 target detection model, optical character recognition technology, and graph convolutional networks (GCNs) to address the challenges of elevator button detection. The YOLOv8 model accurately locates the button position in a complex elevator button panel through its efficient target detection capabilities. Optical character recognition technology can recognize text information on the button, further improving the usability of the detection results. The graph convolutional network (GCN) can extract features from the spatial relationship of the elevator buttons, enhancing the understanding of the overall structure of the elevator panel, making the detection more robust and accurate.

[0027] 2. The present invention describes in detail the method of extracting elevator button image features and elevator button distance features, as well as representing the elevator button panel in the form of a graph, and using a graph convolutional neural network to classify the nodes in the graph. Through elevator button image feature extraction, rich visual information can be extracted from the image, including features such as the shape and texture of the button; while button distance feature extraction can capture the spatial relationship between the buttons, which helps to build a connection graph between the buttons. Using a graph convolutional neural network to classify the nodes in the graph can better understand the structure and layout of the elevator button panel, thereby achieving accurate detection and classification of the buttons.

[0028] 3. The design of the present invention combines target detection, character recognition and graph convolutional network to form a complete set of elevator button detection methods. This combination can make full use of the positioning ability of target detection, the information extraction ability of character recognition and the spatial feature extraction ability of graph convolutional network, so as to solve the problem of elevator button detection more comprehensively. Compared with a single technical means, this combination can improve the accuracy and robustness of detection, so that the mobile robot can perform elevator riding and operation tasks more reliably. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for use in the present invention.

[0030] Figure 1 It is a flowchart of a highly robust elevator button detection method based on an improved YOLOv8 network and graph convolutional network reasoning in the present invention.

[0031] Figure 2 This is the improved YOLOv8 network structure diagram.

[0032] Figure 3 This is the nearest neighbor diagram of the elevator buttons.

[0033] Figure 4 This is a schematic diagram of elevator button modeling.

[0034] Figure 5 It is a graph convolutional network structure diagram. DETAILED DESCRIPTION

[0035] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0036] like Figure 1 As shown, the elevator button detection method based on improved YOLOv8 and optical character recognition and graph convolutional network reasoning of the present invention has the following specific process:

[0037] (1) Collect a large amount of image information of different elevator button panels to build a dataset;

[0038] (2) building a YOLOv8 network structure and using the data set to train the YOLOv8 network structure;

[0039] (3) building a graph convolutional network structure, and using the data set to train the graph convolutional network structure;

[0040] (4) Use the trained YOLOv8 model to detect the actual elevator image and use the model to detect the location of the elevator buttons in the image;

[0041] (5) cropping the elevator button detected in step (4), that is, cutting out the area where the button is located from the original image to obtain a button image block;

[0042] (6) Next, for each button image block obtained by cutting in step (5), the optical character recognition (OCR) model is used to recognize the characters on the image block to obtain the button label and the corresponding confidence level.

[0043] (7) Use the trained graph convolution model and the detection results of YOLOv8 in step (4) to model and predict the actual elevator image. If the confidence of the optical character recognition in step (6) is less than 0.7, the key category predicted by the graph convolution model replaces the category of the optical character recognition;

[0044] Furthermore, in step (1), the following steps are specifically included:

[0045] Step (1.1), collect a large number of different elevator button panel images. These images should cover as many types, styles and layouts of elevator button panels as possible to ensure the diversity and representativeness of the dataset. In the process of collecting images, different lighting conditions, angles and background environments need to be considered so that the trained model has better robustness and generalization ability. Use cameras, online image searches, etc. to obtain these elevator button panel images;

[0046] Step (1.2), preprocess the image, including but not limited to: rotation, translation, scaling, perspective transformation, upside down flipping, left-right flipping and brightness adjustment. These preprocessing operations help increase the diversity of the data set and also reduce the sensitivity of the model to image changes. Through these operations, the elevator button panel images under different angles, scales and lighting conditions can be simulated, thereby improving the robustness of the model;

[0047] Step (1.3), use the dataset annotation software Roboflow to annotate each image. The annotation content includes the category information and coordinate information of the elevator buttons in each image. The category information indicates the category to which each button belongs, such as numbers, letters, function buttons, etc.; while the coordinate information marks the position and size of each button in the image.

[0048] Step (1.4), set the dataset to training set and test set in a ratio of 7:3. The training set is used to train the model, while the test set is used to evaluate the performance and generalization ability of the model.

[0049] Furthermore, in step (2), the following steps are specifically included:

[0050] Step (2.1), the YOLOv8 network structure mainly consists of three parts: Backbone, Neck, and Head. Figure 2 As shown in the figure, in the Backbone part, the components are mainly composed of CSP, C2f, SPP / SPPF, etc. Among them, the CSP (Cross Stage Partial) structure improves the efficiency of feature extraction through cross-stage partial connections and local connections, reduces the number of parameters and calculations, and enhances the nonlinear ability of the network. C2f (CSP to FPN) is a conversion module from CSP feature map to FPN, and improves C2f through SPD-Conv, converts the spatial dimension of the feature map into the depth dimension, and saves more information by increasing the number of channels. This process avoids information loss and captures more detailed features, thereby improving the performance of the network when processing complex tasks. The Neck part is mainly composed of FPN and PAN structures. FPN is a top-down feature pyramid structure that enhances the entire pyramid by passing high-level strong semantic features downward. The PAN structure adds a bottom-up pyramid after FPN to supplement the positioning information transmission of FPN, so that the pyramid formed by FPN and PAN has both semantic information and positioning information. In the Head part, three detectors of different scales are generated according to the different target sizes.

[0051] Step (2.2), adopt the Multi-scale training (0.5~1.5x) strategy, that is, during the training process, assuming that the size of the input image is 640×640, the size of the random value is between 0.5×640 and 1.5×640, but the value is an integer multiple of 32. Such a strategy helps to improve the generalization ability of the network, so that it can better adapt to targets of different sizes. At the same time, the Warmup and Cosine LR scheduler strategy is adopted, that is, warm up before training, gradually increase the learning rate, and then adopt the Cosine learning rate reduction strategy. This strategy can make the training more stable, help the network converge to the optimal solution faster, and improve the efficiency and accuracy of training.

[0052] Furthermore, in step (3), the following steps are specifically included:

[0053] Step (3.1), elevator button image feature extraction, use the feature extraction network to extract image features. According to the rectangular box marked by the data set in step (1), the elevator button image block is cropped and input into the feature extraction network to extract the feature map. The extracted image features are convolved with a 1×1 convolution kernel with 1280 channels and flattened into a one-dimensional vector with a length of 1280. After feature extraction and linear vector transformation, the image features of the button are obtained.

[0054] Step (3.2), elevator button distance feature extraction, the button distance feature is composed of its distance in four main directions (left, right, up, down, such as Figure 3 All values ​​are normalized by the possible maximum value so that the absolute values ​​of these four values ​​are always less than 1. By splicing these four normalized distances, we get the position feature of the key. The image feature and position feature of the key are spliced ​​to get its overall feature.

[0055] Step (3.3) models the elevator button panel, which is represented in the form of a graph, such as Figure 4 As shown, edges are drawn between the elevator button and its four nearest neighbor elevator buttons, with the button as a node and the edges representing the nearest neighbors of the button in the four main directions. The representation of the elevator button panel graph is obtained.

[0056] Step (3.4) classifies the nodes in the graph. The present invention uses a graph convolutional neural network (GCN). The GCN model used in the present invention combines a message passing network (MPNN) and an attention network (GAT). The network is a combination of three layers of graph attention network convolution (GAtConv) and a linear layer, and uses a linear rectifier function (ReLU) as the activation function. In the last layer, we perform a discard process to prevent the signal from entering the next layer with a certain probability to prevent overfitting. The network structure is as follows: Figure 5 shown.

[0057] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A highly robust elevator button detection method based on an improved YOLOv8 network and graph convolutional network inference, comprising the following steps: Step (1), collecting a large amount of image information of different elevator button panels to construct a data set; Step (2), building a YOLOv8 network structure, and using the data set to train the YOLOv8 network structure; Step (3), building a graph convolutional network structure, and using the data set to train the graph convolutional network structure; Step (4), use the trained YOLOv8 model to detect the actual elevator image, and use the model to detect the position of the elevator buttons in the image; Step (5), cutting out the elevator button detected in step (4), that is, cutting out the area where the button is located from the original image to obtain a button image block; Step (6), for each button image block obtained by cutting in step (5), using an optical character recognition (OCR) model to recognize the characters on the image block, and obtain a button label and a corresponding confidence level; Step (7), use the trained graph convolution model and the detection result of YOLOv8 in step (4) to model and predict the actual elevator panel. If the confidence of the optical character recognition in step (6) is less than 0.7, replace the category of the optical character recognition with the button category predicted by the graph convolution model; The specific steps of step (3) are: (a) Elevator button image feature extraction. The feature extraction network is used to extract image features. According to the rectangular frame marked in the data set in step (1), the elevator button image block is cropped and input into the feature extraction network to extract the feature map. The extracted image features are convolved with a 1×1 convolution kernel of 1280 channels and flattened into a one-dimensional vector with a length of 1280. After feature extraction and linear vector transformation, the image features of the button are obtained. (b) Elevator button distance feature extraction. The distance feature of a button is composed of the relative distance values ​​from its nearest neighbor in the four directions of left, right, up, and down. All relative distance values ​​are normalized by the maximum value so that the absolute values ​​of these four values ​​are always less than 1. By splicing these four normalized distances, we get the position feature of the button. The image feature and position feature of the button are spliced ​​to get its overall feature. (c) Model the elevator button panel. The elevator panel is represented as a graph. Edges are drawn between the elevator button and its four nearest neighbor elevator buttons. The button is used as a node, and the edge represents the nearest neighbor of the button in four directions. The elevator button panel graph is obtained. (d) The graph convolutional neural network GCN is used to classify the nodes in the graph. The model combines the message passing network MPNN and the attention network GAT. The network is a three-layer graph attention network convolution GATConv and linear layer combination, and uses the linear rectification function ReLU as the activation function. The last layer is discarded to prevent the signal from entering the next layer with a certain probability, thereby preventing overfitting.

Images