A pommel horse seating analysis method and system based on deep learning

Abstract

The present invention relates to the field of intelligent technology for steelmaking machinery, and in particular to a pommel horse seating analysis method and system based on deep learning, which is used to solve the existing manual analysis and judgment of the pommel horse seating status, so that the health and life safety of on-site personnel are always threatened in such a production environment, and human judgment is highly subjective, and there is a certain risk of human misjudgment. The present invention installs the latest low-latency camera in the melting hoisting area of ​​the steel plant, so that the complete picture of the ladle sitting on the pommel horse can be clearly displayed in real time in the video image, and adopts the pommel horse seating status intelligent analysis method based on deep learning to analyze the pommel horse seating status in real time, guide the hoisting staff to perform the hoisting operation, make the hoisting operation safer and more reliable, and enable the hoisting personnel to move from the "dangerous, toxic, and heavy" dangerous areas on site to the comfortable production environment of the dispatching room, effectively protecting the health and safety of employees.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent steelmaking machinery, and in particular to a pommel horse seating analysis method and system based on deep learning. Background Art

[0002] In the melting area of ​​a steel plant, a hoisting operation occurs. At the hoisting point, an overhead crane hoists a ladle and seats it on a pommel. Once the ladle is properly positioned, the pommel carries the ladle to the next process step. A ladle is a high-temperature, heavyweight container weighing tens of tons, used to hold molten steel. Lifting and lowering the ladle requires a specialized overhead crane within the plant. The ladle's placement on the pommel is accomplished by the crane operator, who, under the direction of the crane operator, gradually positions the ladle on the pommel. Only after this position is secured can the pommel be moved. Real-time analysis of the pommel's seating status during the process determines when the pommel is properly positioned. Failure to accurately assess the pommel's seating status can lead to the ladle moving before it is properly positioned, potentially causing the ladle to fall.

[0003] Currently, when the ladle is seated on the pommel in the melting hoisting area of ​​the steel plant, the seating status of the pommel is analyzed and judged manually by the crane operator on site. On the one hand, the on-site environment is very harsh, with multiple dangerous factors such as high temperature, dust, and splashing. The health and life safety of on-site personnel are always threatened in such a production environment; on the other hand, it is only judged manually, and the human perspective is very limited. As well as the human judgment is highly subjective, there is a certain risk of human misjudgment, which may lead to safety accidents once a misjudgment occurs.

[0004] In summary, the existing manual analysis and judgment of the pommel horse seat seating status puts the health and life safety of on-site personnel at risk in such a production environment. In addition, human judgment is highly subjective and there is a certain risk of human misjudgment, which may lead to safety accidents if a misjudgment occurs. Summary of the Invention

[0005] In response to the problems existing in the background technology, the present invention provides a pommel horse seating analysis method and system based on deep learning.

[0006] A pommel horse seating analysis method based on deep learning, the method comprising:

[0007] Real-time acquisition of video images of the contact points between the ladle and the pommel horse;

[0008] Analyze the crane's position and weight information to determine which location has a hoisting operation, and start the analysis of that location;

[0009] Analyze the collected video images to obtain the seating status information of the pommel horse seat;

[0010] The status information is marked on the video image to generate a result video image, and the result video image is pushed to the overhead crane cab and the dispatching room so that the overhead crane cab and the dispatching room can control the seating of the pommel horse seat according to the real-time result video image.

[0011] Furthermore, the collected video images are analyzed to obtain the seated state result of the pommel horse seat, specifically including:

[0012] Design a deep learning model structure based on the Cascade RCNN target detection algorithm, set the size of the model input image to the target size, and implement the model structure based on the pytorch deep learning framework.

[0013] An intelligent analysis program is written based on the optimal model obtained through training to identify the status of the left, middle, and right contact points of the pommel horse in real time, and to provide the status information of each contact point and the overall placement information, i.e., the pommel horse seating status analysis results, in real time.

[0014] Furthermore, the optimal model obtained by the training specifically includes:

[0015] Images of the ladle sitting on the pommel horse were collected as the original dataset for deep learning model training and verification. The collected original dataset was annotated, and the three contact points of the pommel horse, namely the left, middle, and right contact points, were annotated separately. Each contact point had two labels: contact and non-contact. After annotation, the model training dataset and verification dataset were obtained.

[0016] The training set is augmented by adding noise, brightness transformation, and color transformation. The augmented data and the original training set are combined into the final training set.

[0017] Use the training dataset and the designed deep learning model structure to iteratively train multiple deep learning models, use the validation dataset to verify the trained deep learning models, and select the deep learning model with the highest recognition accuracy as the final model.

[0018] Furthermore, after the annotation, a model training dataset and a validation dataset are obtained, which specifically include:

[0019] Image samples are collected at different time periods of the day for all points where the ladle sits on the pommel horse, and new image samples are continuously added to obtain the original data set. The original data set is then annotated, and three rectangular boxes are used to frame the left, middle, and right parts where the ladle contacts the pommel horse. Each part may be in contact or non-contact state, so each contact point rectangular box may have two labels. The annotated data contains a total of six labels. The annotated data set is divided into a training set and a validation set in proportion.

[0020] Furthermore, the deep learning model structure designed based on the Cascade RCNN target detection algorithm specifically includes:

[0021] A deep learning model structure is designed based on the Cascade RCNN target detection algorithm. Different IOU thresholds are used at different levels to divide positive and negative samples. The IOU threshold of the candidate target box is gradually increased. The candidate target box resampled at the previous level can adapt to the higher IOU threshold of the next level, thereby gradually improving the detection quality of the candidate target box.

[0022] A pommel horse seating analysis system based on deep learning, comprising: an image acquisition module, an overhead crane positioning and weighing module, an information processing and display module, and a pommel horse seating state analysis module;

[0023] Image acquisition module, used to collect real-time video images of the contact points between the ladle and the pommel horse seat;

[0024] Overhead crane positioning and weighing module, used to obtain overhead crane position information and weight information;

[0025] The information processing and display module is used to analyze the crane's position and weight information to determine which point has a hoisting operation and start the analysis of that point;

[0026] The pommel horse seat seating state analysis module is used to analyze the collected video images to obtain the pommel horse seat seating state information;

[0027] The information processing and display module is also used to mark the status information on the video image to generate a result video image, and push the result video image to the overhead crane cab and the dispatching room so that the overhead crane cab and the dispatching room can control the seating of the pommel horse seat according to the real-time result video image.

[0028] Furthermore, the pommel horse seating state analysis module is specifically used to design a deep learning model structure based on the Cascade RCNN target detection algorithm, and to design the size of the model input image to the target size, and to implement the model structure based on the pytorch deep learning framework programming.

[0029] An intelligent analysis program is written based on the optimal model obtained through training to identify the status of the left, middle, and right contact points of the pommel horse in real time, and to provide the status information of each contact point and the overall placement information, i.e., the pommel horse seating status analysis results, in real time.

[0030] Furthermore, the optimal model obtained by the training specifically includes:

[0031] Images of the ladle sitting on the pommel horse were collected as the original dataset for deep learning model training and verification. The collected original dataset was annotated, and the three contact points of the pommel horse, namely the left, middle, and right contact points, were annotated separately. Each contact point had two labels: contact and non-contact. After annotation, the model training dataset and verification dataset were obtained.

[0032] The training set is augmented by adding noise, brightness transformation, and color transformation. The augmented data and the original training set are combined into the final training set.

[0033] Use the training dataset and the designed deep learning model structure to iteratively train multiple deep learning models, use the validation dataset to verify the trained deep learning models, and select the deep learning model with the highest recognition accuracy as the final model.

[0034] Furthermore, the image acquisition module selects a low-latency camera to ensure that the video image for intelligent analysis and judgment is synchronized with the real scene.

[0035] Furthermore, a fill light device is installed at the position where the camera is installed in the image acquisition module.

[0036] Compared with the prior art, the deep learning-based intelligent analysis system for pommel horse seating status of the present invention has the following beneficial effects:

[0037] The latest low-latency cameras are installed in the steelworks' molten metal hoisting area, enabling a clear, real-time display of the entire ladle seated on the pommel. A deep learning-based intelligent pommel seat status analysis method is then used to analyze the pommel seat's seating status in real time, guiding hoisting personnel during lifting operations. The key steps include: First, the low-latency cameras capture video images of corresponding locations 24 hours a day in real time. Then, the intelligent analysis module, combined with the position and weight information of the overhead crane, begins analyzing the pommel seat's seating status, providing real-time analysis results within the system. Finally, the analysis results are visualized in the resulting video and pushed to the overhead crane cab and dispatch room. From the dispatch room, the crane operator can remotely direct the operator in the cab to perform lifting operations using an intercom based on the intelligent analysis results in the video. The operator in the cab can also receive real-time information on the pommel seat's seating status, further confirming proper operation and ensuring the pommel seat is properly positioned. The collaborative work of intelligent analysis, crane operators, and cab operators makes lifting operations safer and more reliable, and enables crane operators to move from the "dangerous, toxic, and heavy" dangerous areas on site to the comfortable production environment of the dispatching room, effectively protecting the health and safety of employees.

[0038] The present invention installs a fill light device when installing the camera. The camera continuously collects images of the ladle sitting on the pommel seat at the current point 24 hours a day, ensuring that the contact part between the ladle and the pommel seat appears completely and clearly in the picture, thereby ensuring that the system operates effectively 24 hours a day.

[0039] The low-latency camera used in the present invention can ensure that the video images used for intelligent analysis are synchronized with the pictures of the actual scene, thereby ensuring that the operation instructions issued by the crane operator based on the result video images meet the timeliness, so that the operation of the operator in the cab is accurate and error-free, and achieving the same or even better effect as the crane operator directing the operation by watching the video picture and the crane operator directing the operation on site.

[0040] When the crane reaches a preset point and its weight meets the ladle weight condition, the present invention sends an intelligent analysis trigger signal, and the intelligent analysis module starts working under the trigger condition, which greatly saves the computing resources consumed by the intelligent analysis module.

[0041] The present invention adopts a cutting-edge deep learning-based intelligent analysis method for the pommel horse seat seating state with high recognition accuracy, so that the final analyzed pommel horse seat seating state result has high accuracy and meets practical application requirements.

[0042] The deep learning-based intelligent analysis method for the pommel horse seat seating state used in the present invention meets the real-time recognition requirements and ensures that the display of the pommel horse seat seating state analysis results is synchronized with the actual scene situation.

[0043] The present invention can successfully complete the hoisting operation of the ladle to be seated on the pommel horse seat directly based on the video image, so that the on-site hoisting personnel can complete the hoisting work directly in the dispatching room, away from the harm of the harsh environment on site.

[0044] The present invention further ensures the safety and reliability of the ladle being seated on the pommel horse seat through the coordinated work of intelligent analysis, cab operators and crane operators.

[0045] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 This is a flow chart of an embodiment of the present invention;

[0048] Figure 2 This is a flowchart for intelligent analysis of the pommel horse seating state based on deep learning;

[0049] Figure 3 This is a block diagram of the pommel horse seating status intelligent analysis system based on deep learning. DETAILED DESCRIPTION

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0051] Currently, when the ladle is seated on the pommel in the melting hoisting area of ​​a steel plant, the seating status of the pommel is analyzed and judged manually by the hoisting personnel on site. On the one hand, the on-site environment is very harsh, with multiple dangerous factors such as high temperature, dust, and splashing. The health and life safety of on-site personnel are always threatened in such a production environment. On the other hand, it is only judged manually, and the human perspective is very limited. Human judgment is highly subjective, and there is a certain risk of human misjudgment. Once a misjudgment occurs, it may lead to a safety accident.

[0052] To this end, the present invention proposes a pommel horse seat seating analysis method and system based on deep learning, including a pommel horse seat seating analysis method based on deep learning and a pommel horse seat seating analysis system based on deep learning.

[0053] In a first aspect, the present invention provides a pommel horse seating analysis method based on deep learning, the method comprising:

[0054] Real-time acquisition of video images of the contact points between the ladle and the pommel horse;

[0055] Analyze the crane's position and weight information to determine which location has a hoisting operation, and start the analysis of that location;

[0056] Analyze the collected video images to obtain the seating status information of the pommel horse seat;

[0057] The status information is marked on the video image to generate a result video image, and the result video image is pushed to the overhead crane cab and the dispatching room so that the overhead crane cab and the dispatching room can control the seating of the pommel horse seat according to the real-time result video image.

[0058] When implementing it specifically, Figure 1 As shown, first, the image acquisition module, which consists of a low-latency camera and a fill light device, collects video images of corresponding points in real time to ensure that the contact area between the ladle and the pommel horse seat appears completely and clearly in the picture;

[0059] The information processing and display module analyzes and determines the crane position and weight information from the crane positioning and weighing module to determine which point has a hoisting operation and triggers the intelligent analysis module corresponding to that point to start working;

[0060] After receiving the trigger signal, the pommel horse seat state analysis module performs an intelligent analysis of the pommel horse seat state on the image collected by the video image acquisition module, and sends the final analyzed pommel horse seat state result to the information processing and display module;

[0061] After receiving the analysis results of the pommel horse seat seating status, the information processing and display module marks the analysis results on the video image to generate a result video image, and pushes the result video image to the overhead crane cab and dispatch room.

[0062] On the basis of this method, it is possible to combine the three aspects of intelligent analysis, cab operators, and crane operators to work together to further ensure that the ladle can be safely and smoothly seated on the pommel seat.

[0063] In this embodiment, the collected video image is analyzed to obtain the sitting state information of the pommel horse seat, specifically including:

[0064] Design a deep learning model structure based on the Cascade RCNN target detection algorithm, set the size of the model input image to the target size, and implement the model structure based on the pytorch deep learning framework.

[0065] An intelligent analysis program is written based on the optimal model obtained through training to identify the status of the left, middle, and right contact points of the pommel horse in real time, and to provide the status information of each contact point and the overall placement information, i.e., the pommel horse seating status analysis results, in real time.

[0066] During specific implementation, the size of the model input image is designed to be 640*360 pixels.

[0067] Based on the dataset, models were trained with input sizes of 1333*800, 960*640, and 640*360, and the optimal performance achieved by each model was compared. The accuracy of the three models was found to be relatively small, ranging from 98.1% to 98.5%. However, significant differences in inference speed were observed. Inference computing tests were conducted on the same server using a Tesla T4 graphics card. The 1333*800 model took an average of approximately 180ms to complete inference per image, while the 960*640 model took approximately 95ms, and the 640*360 model took approximately 35ms. When the model input size was 640*360, the computational speed was essentially real-time, and the model achieved high accuracy, meeting application requirements. Therefore, the smaller 640*360 input size was ultimately chosen.

[0068] The analysis results of the pommel horse seat state are that the overall placement state is positive only when all three contact parts are in contact state, and the overall placement state is negative in all other cases.

[0069] In this embodiment, the optimal model obtained by training is as follows: Figure 2 As shown, specifically including:

[0070] Images of the ladle sitting on the pommel horse were collected as the original dataset for deep learning model training and verification. The collected original dataset was annotated, and the three contact points of the pommel horse, namely the left, middle, and right contact points, were annotated separately. Each contact point had two labels: contact and non-contact. After annotation, the model training dataset and verification dataset were obtained.

[0071] The training set is augmented by adding noise, brightness transformation, and color transformation. The augmented data and the original training set are combined into the final training set.

[0072] Use the training dataset and the designed deep learning model structure to iteratively train multiple deep learning models, use the validation dataset to verify the trained deep learning models, and select the deep learning model with the highest recognition accuracy as the final model.

[0073] In this embodiment, the model training dataset and validation dataset are obtained after the annotation, specifically including:

[0074] Image samples are collected at different time periods of the day for all points where the ladle sits on the pommel horse, and new image samples are continuously added to obtain the original data set. The original data set is then annotated, and three rectangular boxes are used to frame the left, middle, and right parts where the ladle contacts the pommel horse. Each part may be in contact or non-contact state, so each contact point rectangular box may have two labels. The annotated data contains a total of six labels. The annotated data set is divided into a training set and a validation set in a ratio of 8:2.

[0075] In this embodiment, the deep learning model structure is designed based on the Cascade RCNN target detection algorithm, specifically including:

[0076] A deep learning model structure is designed based on the Cascade RCNN target detection algorithm. Different IOU thresholds are used at different levels to divide positive and negative samples. The IOU threshold of the candidate target box is gradually increased. The candidate target box resampled at the previous level can adapt to the higher IOU threshold of the next level, thereby gradually improving the detection quality of the candidate target box.

[0077] In a second aspect, the present invention provides a pommel horse seat seating analysis system based on deep learning, comprising: an image acquisition module, an overhead crane positioning and weighing module, an information processing and display module, and a pommel horse seat seating state analysis module;

[0078] Image acquisition module, used to collect real-time video images of the contact points between the ladle and the pommel horse seat;

[0079] Overhead crane positioning and weighing module, used to obtain overhead crane position information and weight information;

[0080] The information processing and display module is used to analyze the crane's position and weight information to determine which point has a hoisting operation and start the analysis of that point;

[0081] The pommel horse seat seating state analysis module is used to analyze the collected video images to obtain the pommel horse seat seating state information;

[0082] The information processing and display module is also used to mark the status information on the video image to generate a result video image, and push the result video image to the overhead crane cab and the dispatching room so that the overhead crane cab and the dispatching room can control the seating of the pommel horse seat according to the real-time result video image.

[0083] When implementing it specifically, Figure 3 As shown, the system includes an image acquisition module, an overhead crane positioning and weighing module, an information processing and display module, and a pommel horse seat seating state analysis module.

[0084] Video image acquisition module: Select a suitable location to install the latest low-latency camera and fill light device at the ladle hoisting operation point. The camera continuously collects video images at the current point 24 hours a day. It must be ensured that the pommel horse and the ladle are clearly visible in the video image during the process of the ladle sitting on the pommel horse. The collected images are used for intelligent analysis.

[0085] Overhead Crane Positioning and Weighing Module: The overhead crane used in ladle hoisting operations has been modified to incorporate overhead crane positioning and weighing technology, enabling the system to obtain real-time crane position and weight information. Once the crane reaches the preset hoisting position and its weight meets the ladle weight requirements, it issues a trigger signal to initiate intelligent analysis. The intelligent analysis module only begins processing upon receipt of the trigger signal, significantly conserving computing resources for intelligent analysis.

[0086] Pommel horse seat sitting state analysis module: responsible for extracting frames from the collected real-time video images, performing pommel horse seat sitting state analysis on each extracted frame image, and obtaining the pommel horse seat sitting state analysis results.

[0087] Information processing and display module: Receives the position and weight information of the overhead crane, analyzes and judges the information, and sends an identification start signal to the corresponding point; receives the analysis results of the pommel horse seat seating status, and marks the analysis results on the real-time video image to generate a result video image; pushes the result video image to the overhead crane cab and dispatch room.

[0088] The position and weight information of the overhead crane are analyzed and judged. When the overhead crane reaches the preset position and its weight meets the ladle weight conditions, an intelligent analysis trigger signal is issued. The intelligent analysis module starts working only under the trigger conditions, which greatly saves computing resources for intelligent analysis.

[0089] In this embodiment, the pommel horse seat state analysis module is specifically used to design a deep learning model structure based on the CascadeRCNN target detection algorithm, and the size of the model input image is designed to be the target size, and the model structure is implemented based on the pytorch deep learning framework programming.

[0090] An intelligent analysis program is written based on the optimal model obtained through training to identify the status of the left, middle, and right contact points of the pommel horse in real time, and to provide the status information of each contact point and the overall placement information, i.e., the pommel horse seating status analysis results, in real time.

[0091] In this embodiment, the optimal model obtained by training specifically includes:

[0092] Images of the ladle sitting on the pommel horse were collected as the original dataset for deep learning model training and verification. The collected original dataset was annotated, and the three contact points of the pommel horse, namely the left, middle, and right contact points, were annotated separately. Each contact point had two labels: contact and non-contact. After annotation, the model training dataset and verification dataset were obtained.

[0093] The training set is augmented by adding noise, brightness transformation, and color transformation. The augmented data and the original training set are combined into the final training set.

[0094] Use the training dataset and the designed deep learning model structure to iteratively train multiple deep learning models, use the validation dataset to verify the trained deep learning models, and select the deep learning model with the highest recognition accuracy as the final model.

[0095] In this embodiment, the image acquisition module selects a low-latency camera to ensure that the video image for intelligent analysis and judgment is synchronized with the real scene.

[0096] During specific implementation, the latest low-latency cameras are selected when selecting cameras to ensure that the video images used for intelligent analysis and judgment are synchronized with the real scene.

[0097] In this embodiment, a fill light device is installed at the position where the camera is installed in the image acquisition module.

[0098] During specific implementation, in order to ensure that the camera can capture clear images at night, a fill light device is installed at the location where the camera is installed.

[0099] In order to enable those skilled in the art to better understand the present invention, the principles of the present invention are described as follows with reference to the accompanying drawings:

[0100] The present invention provides a pommel seat seating state intelligent analysis system based on deep learning. The latest low-latency camera is installed at the hoisting point where the ladle is seated on the pommel seat, so that the complete picture of the ladle sitting on the pommel seat can be clearly displayed in real time in the video image. In addition, the position information and weight information of the overhead crane are combined to assist in triggering the intelligent analysis module to start the analysis work. The pommel seat seating state intelligent analysis method based on deep learning is used to analyze the pommel seat seating state in real time. Finally, the pommel seat seating state analysis result is given in real time in the system, the analysis result is visualized in the result video picture, and the result picture is pushed to the overhead crane cab and dispatching room. The crane operator can directly use the walkie-talkie to remotely command the cab operator to perform the hoisting operation based on the intelligent analysis result in the video picture in the dispatching room. At the same time, the cab operator can also learn the seating state of the pommel seat in real time, and further confirm whether the operation is proper and whether the seating state of the pommel seat is properly placed. The collaborative work of intelligent analysis, crane operators, and cab operators makes lifting operations safer and more reliable, and enables crane operators to move from the "dangerous, toxic, and heavy" dangerous areas on site to the comfortable production environment of the dispatching room, effectively protecting the health and safety of employees.

[0101] During the implementation process, an important prerequisite must be met, which is to use a low-latency camera to capture video images in real time to ensure that the video images processed by the intelligent analysis module are synchronized with the actual scene. In addition, the key is to design a reasonable deep learning model structure that can accurately identify the status of the three contact points of the pommel horse seat, and the real-time recognition meets the application requirements. Figure 1 The figure shows a specific implementation flow chart of a pommel horse seat seat state intelligent analysis system based on deep learning in the present invention. The specific implementation process includes the following steps:

[0102] Step S1: The image acquisition module, which mainly consists of a low-latency camera and a fill light device, collects video images of corresponding points in real time to ensure that the contact area between the ladle and the pommel horse seat appears completely and clearly in the picture;

[0103] Step S2: The information processing and display module analyzes and determines the crane position information and weight information from the crane positioning and weighing module to determine which point has a hoisting operation, and triggers the intelligent analysis module corresponding to the point to start working;

[0104] Step S3: After receiving the trigger signal, the pommel horse seat seating state analysis module performs an intelligent analysis of the pommel horse seat seating state on the image collected by the video image acquisition module, and sends the final analyzed pommel horse seat seating state result to the information processing and display module;

[0105] Step S4: After receiving the analysis result of the pommel horse seat seating state, the information processing and display module marks the analysis result on the video image to generate a result video image, and pushes the result video image to the overhead crane cab and the dispatching room.

[0106] The key points involved in the implementation of the present invention are described in detail below:

[0107] When selecting a camera, choose the latest low-latency camera. It is necessary to ensure that the video image used for intelligent analysis and judgment is synchronized with the real scene. If the video image used for intelligent analysis and judgment cannot be synchronized with the real scene, then the analysis result video image transmitted to the overhead crane operation room and dispatch room will also be out of sync with the real scene, resulting in incorrect operation instructions issued by the crane operator based on the result video image.

[0108] The latest low-latency camera and fill light device are installed at a suitable location where the ladle sits on the pommel. The camera collects images of the ladle sitting on the pommel at the current location 24 hours a day to ensure that the contact area between the pommel and the ladle appears completely and clearly in the picture. The collected images will be used for intelligent analysis.

[0109] All the overhead cranes that carry the ladle and sit on the pommel seat are modified to have precise positioning and weighing functions, so that the intelligent analysis system can obtain the position information and weight information of the overhead crane in real time. Once the overhead crane reaches the corresponding point and the weight meets the weight conditions of the ladle, a trigger signal will be issued, and the intelligent analysis module will start working.

[0110] The deep learning model structure is designed based on the Cascade RCNN target detection algorithm, and the model input image size is designed to be a small size of 640*360. The deep learning model structure is implemented based on the pytorch deep learning framework. This model structure ensures recognition accuracy while meeting real-time requirements. The flowchart of the intelligent analysis of the sitting state of the pommel horse based on deep learning is as follows Figure 2 As shown, it mainly includes: data collection and annotation, training data augmentation, deep learning model training and verification, and deep learning model reasoning and application:

[0111] (4-1) Image samples are collected at different time periods of the day for all points where the ladle sits on the pommel, and image samples are continuously added to obtain the original data set. The original data set is manually annotated, and three rectangular boxes are used to frame the left, middle, and right parts where the ladle contacts the pommel. Each part may exist in two states: contact or non-contact, so each contact point rectangular box may have two labels. The annotated data contains a total of six labels. The annotated data set is divided into a training set and a validation set in a ratio of 8:2.

[0112] (4-2) The training set is augmented by adding noise, brightness transformation, color transformation, and other data augmentation processes. The resulting augmented data is combined with the original training set to form the final training set. The model trained with such a data set is more adaptable to light changes within the factory.

[0113] (4-3) Design a deep learning model structure based on the Cascade RCNN target detection algorithm. Cascade RCNN mainly addresses the limitations of the RCNN part of the classic two-stage target detection algorithm Faster RCNN in selecting the IOU threshold when training the model. A cascade RCNN structure is proposed. Different levels use different IOU thresholds to divide positive and negative samples, and the IOU threshold of the candidate target frame is gradually increased. The candidate target frame resampled in the previous level can adapt to the higher IOU threshold of the next level, and the detection quality of the candidate target frame is gradually improved, so that the quality of the final target frame is higher.

[0114] The cascaded architecture of the Cascade RCNN deep learning model significantly improves object detection and recognition accuracy, but this structure also slows the algorithm's inference speed. The video of the ladle sitting on the pommel horse is captured at a close distance, capturing a large portion of the image and making the object particularly clear. Based on this dataset, models with input sizes of 1333*800, 960*640, and 640*360 were trained and compared. The best performance achieved by each model was found to be between 98.1% and 98.5%, with minimal differences. However, there was a significant difference in inference speed. Inference tests on the same server using a Tesla T4 graphics card showed that the 1333*800 model took an average of approximately 180ms to infer a single image, while the 960*640 model took approximately 95ms, and the 640*360 model took approximately 35ms. When the model input size is 640*360, the calculation speed basically meets the real-time requirements, and the model accuracy is high, which can meet the application requirements. Therefore, the model input size is finally designed to be a smaller size of 640*360.

[0115] The model structure is implemented based on the pytorch deep learning framework. Pytorch is a deep learning framework with great flexibility. Based on it, a large number of practical tools and functions can be used to speed up the construction of models. In addition, to modify the input size of the model, only the parameters of the input end need to be modified, without modifying the related code of other parts of the model, making it easy to build model structures with different input sizes.

[0116] (4-4) Based on the designed deep learning model structure, the training dataset was used to train the detection and recognition model of the three contact point states of the pommel horse. The number of iterations of model training was set to 15 rounds. The same validation dataset was used to verify each round of model, and the model with the highest accuracy was selected as the final application model.

[0117] (4-5) Based on the obtained optimal model, an inference deployment program is written. The input of the inference code is the current frame video image, and the output result is the status information of the pommel horse seat. Each contact point of the pommel horse seat may exist in two states: contact or non-contact. Only when the three contact points are in the contact state is it determined that the ladle is placed on the pommel horse seat, and the status of the three contact points and whether the pommel horse seat as a whole is in the placed state are fed back to the system in real time.

[0118] (5) The pommel horse seat status information obtained by the deep learning model reasoning analysis is superimposed on the real-time video image, that is, the three rectangular frames of the three contact points of the pommel horse seat, the status of each contact point, and the status of whether the pommel horse seat is placed properly as a whole are marked on the real-time video image, and an intelligent analysis result video image is generated. The result video image is pushed to the overhead crane cab and the dispatching room. The crane operator issues a crane command based on the video image, and the cab operator further confirms whether the operation is proper and whether the ladle is completely placed on the pommel horse seat based on the video image.

[0119] The pommel seat seating status intelligent analysis system based on deep learning provided by the present invention accurately and in real time analyzes the status of the ladle seating on the pommel seat based on the deep learning model, and marks the analysis results on the video image in real time and pushes them to the overhead crane cab and the dispatching room. The accurate and objective analysis results of the intelligent analysis algorithm have a great auxiliary support role for the overhead crane cab operator and the finger crane personnel. The coordinated work of the three makes the operation process of the ladle seating on the pommel seat safer and more reliable, and enables the finger crane personnel to complete the finger crane work in the dispatching room, away from the dangerous and harsh environment on site, which is conducive to protecting the health and life safety of the finger crane personnel.

[0120] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pommel horse seating analysis method based on deep learning, characterized in that: The method comprises: Real-time acquisition of video images of the contact points between the ladle and the pommel horse; Analyze the crane's position and weight information to determine which location has a hoisting operation, and start the analysis of that location; Analyze the collected video images to obtain the seating status information of the pommel horse seat; Marking the status information on the video image to generate a result video image, and pushing the result video image to the overhead crane cab and the dispatching room so that the overhead crane cab and the dispatching room can control the seating of the pommel horse seat according to the real-time result video image; The collected video images are analyzed to obtain the seating status information of the pommel horse seat, specifically including: Design a deep learning model structure based on the Cascade RCNN target detection algorithm, set the model input image size to the target size, implement the model structure based on the pytorch deep learning framework, and train the model structure to obtain the optimal model. An intelligent analysis program is written based on the optimal model obtained through training to identify the status of the left, middle, and right contact points of the pommel horse in real time, and to provide the status information of each contact point and the overall placement information, i.e., the pommel horse seating status analysis results, in real time.

2. The method for analyzing the sitting position of a pommel horse based on deep learning according to claim 1, characterized in that: The optimal model obtained by the training specifically includes: Images of the ladle sitting on the pommel horse were collected as the original dataset for deep learning model training and verification. The collected original dataset was annotated, and the three contact points of the pommel horse, namely the left, middle, and right contact points, were annotated separately. Each contact point had two labels: contact and non-contact. After annotation, the model training dataset and verification dataset were obtained. The training set is augmented by adding noise, brightness transformation, and color transformation. The augmented data and the original training set are combined into the final training set. Use the training dataset and the designed deep learning model structure to iteratively train multiple deep learning models, use the validation dataset to verify the trained deep learning models, and select the deep learning model with the highest recognition accuracy as the final model.

3. The method for analyzing the sitting position of a pommel horse based on deep learning according to claim 2, wherein: After the annotation, the model training dataset and validation dataset are obtained, which specifically include: Image samples are collected at different time periods of the day for all points where the ladle sits on the pommel horse, and new image samples are continuously added to obtain the original data set. The original data set is then annotated, and three rectangular boxes are used to frame the left, middle, and right parts where the ladle contacts the pommel horse. Each part may be in contact or non-contact state, so each contact point rectangular box may have two labels. The annotated data contains a total of six labels. The annotated data set is divided into a training set and a validation set in proportion.

4. A pommel horse seating analysis method based on deep learning according to any one of claims 1 to 3, characterized in that: The deep learning model structure designed based on the Cascade RCNN target detection algorithm specifically includes: A deep learning model structure is designed based on the Cascade RCNN target detection algorithm. Different IOU thresholds are used at different levels to divide positive and negative samples. The IOU threshold of the candidate target box is gradually increased. The candidate target box resampled at the previous level can adapt to the higher IOU threshold of the next level, thereby gradually improving the detection quality of the candidate target box.

5. A pommel horse seating analysis system based on deep learning, characterized in that: include: Image acquisition module, overhead crane positioning and weighing module, information processing and display module, and pommel horse seating status analysis module; Image acquisition module, used to collect real-time video images of the contact points between the ladle and the pommel horse seat; Overhead crane positioning and weighing module, used to obtain overhead crane position information and weight information; The information processing and display module is used to analyze the crane's position and weight information to determine which point has a hoisting operation and start the analysis of that point; The pommel horse seat seating state analysis module is used to analyze the collected video images to obtain the pommel horse seat seating state information; The information processing and display module is further used to mark the status information on the video image to generate a result video image, and push the result video image to the overhead crane cab and the dispatching room so that the overhead crane cab and the dispatching room can control the seating of the pommel horse seat according to the real-time result video image; The collected video images are analyzed to obtain the seating status information of the pommel horse seat, specifically including: Design a deep learning model structure based on the Cascade RCNN target detection algorithm, set the model input image size to the target size, implement the model structure based on the pytorch deep learning framework, and train the model structure to obtain the optimal model. An intelligent analysis program is written based on the optimal model obtained through training to identify the status of the left, middle, and right contact points of the pommel horse in real time, and to provide the status information of each contact point and the overall placement information, i.e., the pommel horse seating status analysis results, in real time.

6. The pommel horse seat seating analysis system based on deep learning according to claim 5, characterized in that: The optimal model obtained by the training specifically includes: Images of the ladle sitting on the pommel horse were collected as the original dataset for deep learning model training and verification. The collected original dataset was annotated, and the three contact points of the pommel horse, namely the left, middle, and right contact points, were annotated separately. Each contact point had two labels: contact and non-contact. After annotation, the model training dataset and verification dataset were obtained. The training set is augmented by adding noise, brightness transformation, and color transformation. The augmented data and the original training set are combined into the final training set. Use the training dataset and the designed deep learning model structure to iteratively train multiple deep learning models, use the validation dataset to verify the trained deep learning models, and select the deep learning model with the highest recognition accuracy as the final model.

7. The pommel horse seating analysis system based on deep learning according to claim 4, characterized in that: The image acquisition module selects a low-latency camera to ensure that the video image for intelligent analysis and judgment is synchronized with the real scene.

8. A pommel horse seat seating analysis system based on deep learning according to any one of claims 4 to 7, characterized in that: A fill light device is installed at the position where the camera is installed in the image acquisition module.

Images