Bucket wheel machine intelligent remote control system and method based on video monitoring

Abstract

The invention discloses a bucket wheel machine intelligent remote control system and method based on video monitoring. The method comprises the following steps: firstly, acquiring a bucket-wheel state monitoring video acquired by a camera, then, extracting a first frame and a last frame from the bucket-wheel state monitoring video to obtain a first bucket-wheel state monitoring image and a second bucket-wheel state monitoring image, and then, acquiring a second bucket-wheel state monitoring image and a third bucket-wheel state monitoring image; extracting wheel bucket state features of the first wheel bucket state monitoring image and the second wheel bucket state monitoring image to obtain a sequence of first wheel bucket state local feature vectors and a sequence of second wheel bucket state local feature vectors; and determining whether the wheel bucket is damaged or not based on a comparison semantic association relationship between the sequence of the first wheel bucket state local feature vectors and the sequence of the second wheel bucket state local feature vectors. In this way, related technicians can be prompted to take measures in time to control the bucket wheel machine.

Description

Technical Field

[0001] The present application relates to the field of remote control, and more specifically, to an intelligent remote control system and method for a bucket wheel excavator based on video monitoring. Background Art

[0002] With the continuous improvement of industrial automation, bucket wheel machines, as important loading and unloading equipment, play a key role in mining production and other fields. However, failure and damage of bucket wheel machines can lead to production interruptions and even safety hazards, resulting in huge economic losses. For example, due to the complex and harsh working environment of bucket wheel machines, their bucket parts often fail or are damaged, resulting in increased downtime and maintenance time for bucket wheel machines, and even accidents.

[0003] At present, manual inspection is mainly used for monitoring and diagnosing the bucket status of bucket wheel excavators. Manual inspection is not only time-consuming and labor-intensive, but also has subjective judgment errors and safety hazards. Therefore, an optimized remote control system and method for bucket wheel excavators is expected. Summary of the invention

[0004] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides an intelligent remote control system and method for bucket wheel machine based on video monitoring, which can use a camera to collect the monitoring video of the bucket wheel state of the bucket wheel machine, and analyze the state of the bucket wheel through a deep learning intelligent algorithm to realize the identification of whether the bucket wheel is damaged, so as to prompt relevant technical personnel to take measures to control the bucket wheel machine in time.

[0005] According to one aspect of the present application, there is provided a method for intelligent remote control of a bucket wheel machine based on video monitoring, which comprises:

[0006] Obtain bucket wheel status monitoring video collected by the camera;

[0007] Extracting the first frame and the last frame from the bucket wheel state monitoring video to obtain a first bucket wheel state monitoring image and a second bucket wheel state monitoring image;

[0008] Extracting bucket state features of the first bucket state monitoring image and the second bucket state monitoring image to obtain a sequence of first bucket state local feature vectors and a sequence of second bucket state local feature vectors; and

[0009] Based on the comparative semantic association relationship between the sequence of the first wheel bucket state local feature vectors and the sequence of the second wheel bucket state local feature vectors, it is determined whether the wheel bucket is damaged.

[0010] According to another aspect of the present application, there is provided an intelligent remote control system for a bucket wheel machine based on video monitoring, which comprises:

[0011] A video acquisition module is used to acquire bucket wheel status monitoring video collected by a camera;

[0012] A key frame extraction module, used for extracting the first frame and the last frame from the bucket wheel state monitoring video to obtain a first bucket wheel state monitoring image and a second bucket wheel state monitoring image;

[0013] a state feature extraction module, used for extracting the wheel bucket state features of the first wheel bucket state monitoring image and the second wheel bucket state monitoring image to obtain a sequence of first wheel bucket state local feature vectors and a sequence of second wheel bucket state local feature vectors; and

[0014] The damage analysis module is used to determine whether the wheel bucket is damaged based on the comparative semantic association relationship between the sequence of the first wheel bucket state local feature vectors and the sequence of the second wheel bucket state local feature vectors.

[0015] Compared with the prior art, the intelligent remote control system and method for bucket wheel machine based on video monitoring provided by the present application first obtains the bucket wheel state monitoring video collected by the camera, then extracts the first frame and the last frame from the bucket wheel state monitoring video to obtain the first bucket wheel state monitoring image and the second bucket wheel state monitoring image, then extracts the bucket wheel state features of the first bucket wheel state monitoring image and the second bucket wheel state monitoring image to obtain the sequence of the first bucket wheel state local feature vector and the sequence of the second bucket wheel state local feature vector, and finally, based on the comparative semantic association relationship between the sequence of the first bucket wheel state local feature vector and the sequence of the second bucket wheel state local feature vector, determines whether the bucket wheel is damaged. In this way, relevant technicians can be prompted to take timely measures to control the bucket wheel machine. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. The following drawings are not intentionally scaled to the actual size, and the focus is on illustrating the main purpose of the present application.

[0017] Figure 1 It is a flow chart of an intelligent remote control method of a bucket wheel excavator based on video monitoring according to an embodiment of the present application.

[0018] Figure 2 Schematic diagram of the architecture of an intelligent remote control method for a bucket wheel excavator based on video monitoring according to an embodiment of the present application.

[0019] Figure 3 It is a flowchart of sub-step S140 of the intelligent remote control method of bucket wheel excavator based on video monitoring according to an embodiment of the present application.

[0020] Figure 4 It is a block diagram of an intelligent remote control system of a bucket wheel excavator based on video monitoring according to an embodiment of the present application.

[0021] Figure 5 This is an application scenario diagram of the bucket wheel excavator intelligent remote control method based on video monitoring according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work also fall within the scope of protection of the present application.

[0023] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0024] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0025] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0026] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0027] In response to the above technical problems, the technical concept of the present application is to use a camera to collect monitoring videos of the bucket wheel status of the bucket wheel machine, and to analyze the status of the bucket wheel through a deep learning intelligent algorithm to identify whether the bucket wheel is damaged, thereby prompting relevant technical personnel to take timely measures to control the bucket wheel machine.

[0028] Based on this, Figure 1It is a flow chart of an intelligent remote control method of a bucket wheel excavator based on video monitoring according to an embodiment of the present application. Figure 2 FIG. 1 is a schematic diagram of the architecture of an intelligent remote control method for a bucket wheel machine based on video monitoring according to an embodiment of the present application. Figure 1 and Figure 2 As shown, according to the embodiment of the present application, the intelligent remote control method of the bucket wheel machine based on video monitoring includes the following steps: S110, acquiring a bucket wheel state monitoring video collected by a camera; S120, extracting the first frame and the last frame from the bucket wheel state monitoring video to obtain a first bucket wheel state monitoring image and a second bucket wheel state monitoring image; S130, extracting the bucket wheel state features of the first bucket wheel state monitoring image and the second bucket wheel state monitoring image to obtain a sequence of first bucket wheel state local feature vectors and a sequence of second bucket wheel state local feature vectors; and, S140, determining whether the bucket wheel is damaged based on the comparative semantic association relationship between the sequence of the first bucket wheel state local feature vectors and the sequence of the second bucket wheel state local feature vectors.

[0029] Specifically, in the technical solution of the present application, firstly, a bucket wheel status monitoring video captured by a camera is obtained; and the first frame and the last frame are extracted from the bucket wheel status monitoring video to obtain a first bucket wheel status monitoring image and a second bucket wheel status monitoring image. Here, the images of the first frame and the last frame can reflect the state difference of the bucket wheel machine during the monitoring time period. It should be understood that during the operation of the bucket wheel machine, its operating state may change slowly over time. In the technical solution of the present application, it is expected to extract the first frame and the last frame to highlight the initial bucket wheel state and the bucket wheel state at the end of the bucket wheel machine, so as to guide the model to focus on the differences between the two frames of images, which is of great significance for judging whether the bucket wheel state is damaged.

[0030] At the same time, the processing method of extracting image frames can reduce the amount of calculation of the model and help optimize the calculation efficiency. Specifically, the bucket wheel status monitoring video contains a large number of frames, and processing the complete video may require a lot of computing resources and time. In the technical solution of this application, selecting to extract key image frames can reduce the calculation load without losing key information.

[0031] Then, the first bucket state monitoring image and the second bucket state monitoring image are respectively passed through a bucket state feature extractor based on the ViT model to obtain a sequence of first bucket state local feature vectors and a sequence of second bucket state local feature vectors. The bucket state feature extractor based on the ViT model visualizes the detailed information of different local spatial regions in the first bucket state monitoring image and the second bucket state monitoring image through block processing, while taking into account the global semantic correlation relationship in each local spatial region to capture the overall bucket state information.

[0032] Correspondingly, in step S130, the bucket state features of the first bucket state monitoring image and the second bucket state monitoring image are extracted to obtain a sequence of first bucket state local feature vectors and a sequence of second bucket state local feature vectors, including: using a deep learning network model to perform feature extraction on the first bucket state monitoring image and the second bucket state monitoring image to obtain a sequence of first bucket state local feature vectors and a sequence of second bucket state local feature vectors.

[0033] Specifically, the deep learning network model is a wheel bucket state feature extractor based on the ViT model; wherein, the deep learning network model is used to perform feature extraction on the first wheel bucket state monitoring image and the second wheel bucket state monitoring image to obtain a sequence of the first wheel bucket state local feature vector and a sequence of the second wheel bucket state local feature vector, including: passing the first wheel bucket state monitoring image and the second wheel bucket state monitoring image through the wheel bucket state feature extractor based on the ViT model respectively to obtain a sequence of the first wheel bucket state local feature vector and a sequence of the second wheel bucket state local feature vector.

[0034] More specifically, the first bucket state monitoring image and the second bucket state monitoring image are respectively passed through the bucket state feature extractor based on the ViT model to obtain a sequence of the first bucket state local feature vector and a sequence of the second bucket state local feature vector, including: respectively performing image segmentation on the first bucket state monitoring image and the second bucket state monitoring image to obtain a sequence of first bucket state monitoring image blocks and a sequence of second bucket state monitoring image blocks; using the image block embedding layer of the bucket state feature extractor based on the ViT model to respectively embed each first bucket state monitoring image block in the sequence of the first bucket state monitoring image blocks and each second bucket state monitoring image block in the sequence of the second bucket state monitoring image blocks to obtain a sequence of first bucket state monitoring image block embedding vectors and a sequence of second bucket state monitoring image block embedding vectors; and, passing the sequence of the first bucket state monitoring image block embedding vectors and the sequence of the second bucket state monitoring image block embedding vectors through the bucket state feature extractor based on the ViT model to obtain a sequence of the first bucket state local feature vector and a sequence of the second bucket state local feature vector.

[0035] It is worth mentioning that the image block embedding layer refers to a layer that divides the image into small blocks and converts these small blocks into vector representations when processing the image. The image block embedding layer is based on a part of the ViT (Vision Transformer) model, which is used to convert small blocks of the wheel bucket state monitoring image into vector representations. The role of this process is to convert local information in the image into vector representations for subsequent processing and analysis. By dividing the image into small blocks and converting each small block into a vector representation, local features in the image, such as texture, shape, etc., can be better captured. This representation method usually helps the model better understand the image and can be more effectively applied to subsequent tasks, such as feature extraction, classification, object detection, etc. In general, the role of the image block embedding layer is to convert local information in the image into vector representations so that the deep learning model can better understand and process image data.

[0036] Next, the correlation between each group of corresponding first bucket state local feature vectors and second bucket state local feature vectors in the sequence of the first bucket state local feature vectors and the sequence of the second bucket state local feature vectors is calculated to obtain a bucket state global comparison semantic feature vector composed of multiple correlations. That is, the similarity between the bucket states of the two stages is quantified by calculating the correlation between each group of corresponding first bucket state local feature vectors and second bucket state local feature vectors. Among them, the degree of correlation can characterize the degree of proximity between the two in the feature space, thereby helping to understand the degree of similarity between the two bucket states. To a certain extent, the correlation can measure and reveal the difference between the bucket states of the two stages. When the correlation is low, it indicates that there may be a large difference between the bucket states of the two stages, and this difference information is of great significance for judging whether the bucket is damaged.

[0037] Furthermore, the bucket wheel state global comparison semantic feature vector is passed through a classifier to obtain a classification result, and the classification result is used to indicate whether the bucket wheel is damaged. In this way, the fault diagnosis and maintenance efficiency of the bucket wheel machine is improved, and the labor cost and safety risk are reduced.

[0038] Accordingly, in step S140, Figure 3As shown, based on the contrast semantic association relationship between the sequence of the first wheel bucket state local feature vectors and the sequence of the second wheel bucket state local feature vectors, determining whether the wheel bucket is damaged includes: S141, performing feature distribution correction on the sequence of the second wheel bucket state local feature vectors to obtain a sequence of corrected second wheel bucket state local feature vectors; S142, constructing a contrast semantic association relationship between each group of corresponding first wheel bucket state local feature vectors and corrected second wheel bucket state local feature vectors in the sequence of the first wheel bucket state local feature vectors and the sequence of the corrected second wheel bucket state local feature vectors to obtain a wheel bucket state global contrast semantic feature vector; and, S143, passing the wheel bucket state global contrast semantic feature vector through a classifier to obtain a classification result, and the classification result is used to indicate whether the wheel bucket is damaged.

[0039] It should be understood that the purpose of step S141 is to correct the local feature vector of the second bucket state to eliminate the possible feature distribution deviation. The corrected feature vector can improve the accuracy of subsequent comparison and similarity measurement. The purpose of step S142 is to construct a global comparison semantic feature vector of the bucket state by comparing the comparison semantic association relationship between the local feature vector of the first bucket state and the corrected local feature vector of the second bucket state. By analyzing the similarity and correlation between the features, a global feature vector representing the bucket state can be obtained. The purpose of step S143 is to input the global comparison semantic feature vector of the bucket state into the classifier for classification judgment. The classifier will judge whether the bucket is damaged based on the input feature vector and output the corresponding classification result. In summary, these three steps compare and analyze the relationship between the local feature vectors of the bucket state, construct the global feature vector of the bucket state, and judge through the classifier to determine whether the bucket is damaged. This process uses feature comparison and similarity measurement to extract and characterize key information of the bucket state to achieve the purpose of damage judgment.

[0040] Wherein, in step S142, constructing a comparative semantic association relationship between each group of corresponding first wheel bucket state local feature vectors and corrected second wheel bucket state local feature vectors in the sequence of the first wheel bucket state local feature vectors and the sequence of the corrected second wheel bucket state local feature vectors to obtain a wheel bucket state global comparative semantic feature vector, including: respectively calculating the correlation between each group of corresponding first wheel bucket state local feature vectors and corrected second wheel bucket state local feature vectors in the sequence of the first wheel bucket state local feature vectors and the sequence of the corrected second wheel bucket state local feature vectors to obtain the wheel bucket state global comparative semantic feature vector composed of multiple correlations.

[0041] It should be understood that the correlation is a statistic used to measure the degree of association between two variables. Specifically, the correlation is used to measure the degree of association between the local feature vector of the first wheel bucket state and the local feature vector of the second wheel bucket state after correction, so as to construct a global contrast semantic feature vector of the wheel bucket state. Commonly used correlation statistics include Pearson correlation coefficient, Spearman's rank correlation coefficient and Kendall's rank correlation coefficient, etc. The value range of these statistics is usually between -1 and 1, indicating the strength and direction of the linear relationship between the variables. In step S142, by calculating the correlation between the local feature vector of the first wheel bucket state and the local feature vector of the second wheel bucket state after correction, a global contrast semantic feature vector of the wheel bucket state composed of multiple correlations can be obtained, and these correlations can provide information about the similarity or difference between the two feature vectors, thereby constructing a global feature vector of the wheel bucket state for subsequent classification judgment.

[0042] Specifically, respectively calculating the correlation between each group of corresponding first bucket state local feature vectors and corrected second bucket state local feature vectors in the sequence of the first bucket state local feature vectors and the sequence of the corrected second bucket state local feature vectors to obtain the bucket state global comparison semantic feature vector composed of multiple correlations, including: calculating the correlation between each group of corresponding first bucket state local feature vectors and corrected second bucket state local feature vectors in the sequence of the first bucket state local feature vectors and the sequence of the corrected second bucket state local feature vectors using the following correlation formula; wherein the correlation formula is:

[0043] S i =h i T r j

[0044] Among them, h i is the i-th first wheel bucket state local feature vector in the sequence of the first wheel bucket state local feature vectors, (·) T represents the transpose operation, r j is the jth corrected second wheel bucket state local feature vector in the sequence of the corrected second wheel bucket state local feature vectors, S i It is the feature value of the ith position in the global comparison semantic feature vector of the wheel bucket state, that is, the ith correlation.

[0045] Among them, in step S143, the global contrast semantic feature vector of the wheel bucket state is passed through a classifier to obtain a classification result, and the classification result is used to indicate whether the wheel bucket is damaged, including: using the fully connected layer of the classifier to fully connect encode the global contrast semantic feature vector of the wheel bucket state to obtain an encoded classification feature vector; and inputting the encoded classification feature vector into the Softmax classification function of the classifier to obtain the classification result.

[0046] It should be understood that the role of the classifier is to use the given categories and known training data to learn classification rules and classifiers, and then classify (or predict) unknown data. Logistic regression and SVM are often used to solve binary classification problems. For multi-class classification problems, logistic regression or SVM can also be used, but multiple binary classifications are required to form a multi-classification. However, this is prone to errors and is not efficient. Commonly used multi-classification methods include the Softmax classification function.

[0047] It is worth mentioning that fully connected encoding refers to the process of encoding input data through a fully connected layer. A fully connected layer is a common layer type in a neural network, in which each neuron is connected to all neurons in the previous layer. In a fully connected layer, each input feature is connected to each neuron to form a fully connected network structure. The function of fully connected encoding is to map the input wheel bucket state global contrast semantic feature vector to a higher dimensional feature space. Through the weights and bias parameters of the fully connected layer, the input features can be linearly combined and nonlinearly transformed to extract richer and more abstract feature representations. Fully connected encoding can help the network learn more discriminative and representational features to better distinguish whether the wheel bucket state is damaged. In the specific steps, the fully connected layer of the classifier fully connects the wheel bucket state global contrast semantic feature vector and maps the feature vector to an encoded classification feature vector space. This encoded classification feature vector contains a higher dimensional feature representation after being encoded by the fully connected layer, and has better representation capabilities. Then, the encoded classification feature vector is input into the Softmax classification function of the classifier, and normalized by the Softmax function to obtain the final classification result, which is used to indicate whether the wheel bucket is damaged. In this process, the fully connected encoding plays the role of extracting and converting features to support the classifier to accurately classify the wheel bucket status.

[0048] Here, when the first bucket state monitoring image and the second bucket state monitoring image are respectively passed through the bucket state feature extractor based on the ViT model, the sequence of the first bucket state local feature vector and the sequence of the second bucket state local feature vector respectively represent the sequence distribution of the context image semantic association features of the bucket wheel local image of the first bucket state monitoring image and the sequence distribution of the context image semantic association features of the bucket wheel local image of the second bucket state monitoring image. When calculating the correlation between each group of corresponding first bucket state local feature vectors and second bucket state local feature vectors in the sequence of the first bucket state local feature vector and the sequence of the second bucket state local feature vector to obtain a bucket state global contrast semantic feature vector composed of multiple correlations, and performing classification and regression on the bucket state global contrast semantic feature vector through a classifier, the sequence of the first bucket state local feature vector substantially needs to conform to the interpolation-type mixture of the sequence of the second bucket state local feature vector based on the context semantic association features between local images for the classification and regression targets.

[0049] Therefore, in order to improve the feature enhancement expression effect of the sequence of the first wheel bucket state local feature vectors based on the expression consistency of the sequence of the second wheel bucket state local feature vectors based on the contextual semantic association features between local images, for each group of the first wheel bucket state local feature vectors and the second wheel bucket state local feature vectors, the second wheel bucket state local feature vectors are optimized based on the first wheel bucket state local feature vectors.

[0050] Accordingly, in step S141, in one example, performing feature distribution correction on the sequence of the second wheel bucket state local feature vectors to obtain a corrected sequence of the second wheel bucket state local feature vectors includes: performing feature distribution correction on the sequence of the second wheel bucket state local feature vectors using the following optimization formula to obtain the corrected sequence of the second wheel bucket state local feature vectors; wherein the optimization formula is:

[0051]

[0052] Among them, V 1 is each first bucket state local feature vector in the sequence of the first bucket state local feature vectors, V 2 is each second bucket state local feature vector in the sequence of the second bucket state local feature vectors. Specifically, each first bucket state local feature vector corresponds to each second bucket state local feature vector. 1max -1 and v 2max -1The first bucket state local feature vectors V in the sequence of the first bucket state local feature vectors are represented respectively. 1 and each second bucket state local feature vector V in the sequence of the second bucket state local feature vectors 2 The inverse of the global maximum value of , I is a unit vector, and V 2 ⊙-1 represents each second bucket state local feature vector V in the sequence of the second bucket state local feature vectors 2 The reciprocal of the position-by-position eigenvalues, V b is the inductive bias vector, ⊙ represents the point multiplication by position, represents vector subtraction, ⊕ represents vector addition, V ′ 2 is the sequence of local feature vectors of the second bucket state after correction.

[0053] Specifically, for the interpolation mixing of classification and regression targets in the feature extraction process, based on the idea of ​​interpolation regularization, by unmixing the feature mapping of outlier features, the high-dimensional feature manifold is restored to the manifold geometry based on weak enhancement based on the inductive bias, and a consistent feature enhancement mapping of interpolation samples based on feature extraction and interpolation prediction is achieved, so as to obtain the feature enhancement effect while maintaining feature consistency, thereby improving the accuracy of the correlation between each group of corresponding first wheel bucket state local feature vectors and second wheel bucket state local feature vectors, so as to improve the accuracy of the classification result obtained by the classifier of the wheel bucket state global contrast semantic feature vector composed of multiple correlations.

[0054] In summary, the intelligent remote control method of the bucket wheel machine based on video monitoring according to the embodiment of the present application is explained, which can prompt relevant technical personnel to take timely measures to control the bucket wheel machine.

[0055] Figure 4 FIG. 1 is a block diagram of an intelligent remote control system 100 for a bucket wheel machine based on video monitoring according to an embodiment of the present application. Figure 4As shown, according to an embodiment of the present application, an intelligent remote control system 100 for a bucket wheel machine based on video monitoring includes: a video acquisition module 110, for acquiring a bucket wheel state monitoring video collected by a camera; a key frame extraction module 120, for extracting a first frame and a last frame from the bucket wheel state monitoring video to obtain a first bucket wheel state monitoring image and a second bucket wheel state monitoring image; a state feature extraction module 130, for extracting bucket wheel state features of the first bucket wheel state monitoring image and the second bucket wheel state monitoring image to obtain a sequence of first bucket wheel state local feature vectors and a sequence of second bucket wheel state local feature vectors; and a damage analysis module 140, for determining whether the bucket wheel is damaged based on a comparative semantic association relationship between the sequence of the first bucket wheel state local feature vectors and the sequence of the second bucket wheel state local feature vectors.

[0056] In one example, in the above-mentioned bucket wheel machine intelligent remote control system 100 based on video monitoring, the state feature extraction module 130 is used to: use a deep learning network model to perform feature extraction on the first wheel bucket state monitoring image and the second wheel bucket state monitoring image to obtain a sequence of the first wheel bucket state local feature vector and a sequence of the second wheel bucket state local feature vector.

[0057] Here, those skilled in the art can understand that the specific functions and operations of each module in the above-mentioned bucket wheel machine intelligent remote control system 100 based on video monitoring have been referred to above. Figures 1 to 3 The description of the intelligent remote control method of bucket wheel machine based on video monitoring has been introduced in detail, and therefore, its repeated description will be omitted.

[0058] As described above, the bucket wheel machine intelligent remote control system 100 based on video surveillance according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with a bucket wheel machine intelligent remote control algorithm based on video surveillance. In one example, the bucket wheel machine intelligent remote control system 100 based on video surveillance according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the bucket wheel machine intelligent remote control system 100 based on video surveillance can be a software module in the operating system of the wireless terminal, or can be an application developed for the wireless terminal; of course, the bucket wheel machine intelligent remote control system 100 based on video surveillance can also be one of the many hardware modules of the wireless terminal.

[0059] Alternatively, in another example, the bucket wheel machine intelligent remote control system 100 based on video surveillance and the wireless terminal may also be separate devices, and the bucket wheel machine intelligent remote control system 100 based on video surveillance may be connected to the wireless terminal via a wired and / or wireless network, and transmit interactive information in accordance with an agreed data format.

[0060] Figure 5 FIG. 1 is an application scenario diagram of the bucket wheel machine intelligent remote control method based on video monitoring according to an embodiment of the present application. Figure 5 As shown, in this application scenario, first, the bucket wheel status monitoring video collected by the camera is obtained (for example, Figure 5 D), then, the bucket wheel status monitoring video is input to a server (for example, Figure 5 In S) shown in , the server is able to use the bucket wheel machine intelligent remote control algorithm based on video monitoring to process the bucket wheel status monitoring video to obtain a classification result indicating whether the bucket wheel is damaged.

[0061] According to another aspect of the present application, a non-volatile computer-readable storage medium is provided, on which computer-readable instructions are stored. When the instructions are executed by a computer, the above-mentioned method can be executed.

[0062] The program part in the technology can be considered as a "product" or "manufactured product" in the form of executable code and / or related data, which is participated in or realized by computer-readable media. Tangible and permanent storage media can include any memory or storage used by any computer, processor, or similar device or related module. For example, various semiconductor memories, tape drives, disk drives or any similar devices that can provide storage functions for software.

[0063] All or part of the software may sometimes communicate over a network, such as the Internet or other communications network. Such communications can load software from one computer device or processor to another. For example: from a server or host computer of a video target detection device to a hardware platform of a computer environment, or other computer environment that implements the system, or a system with similar functions related to providing information required for target detection. Therefore, another medium capable of transmitting software elements can also be used as a physical connection between local devices, such as light waves, radio waves, electromagnetic waves, etc., transmitted through cables, optical cables or air. Physical media used to carry carriers, such as cables, wireless connections or optical cables and the like, can also be considered as media that carry software. As used herein, unless limited to tangible "storage" media, other terms referring to computer or machine "readable media" refer to media involved in the process of the processor executing any instructions.

[0064] In addition, it will be appreciated by those skilled in the art that various aspects of the present application may be illustrated and described by a number of patentable categories or situations, including any new and useful process, machine, product or combination of substances, or any new and useful improvements thereto. Accordingly, various aspects of the present application may be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may all be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". In addition, various aspects of the present application may be represented as a computer product located in one or more computer-readable media, which includes computer-readable program code.

[0065] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an idealized or extremely formal sense, unless explicitly defined as such herein.

[0066] The above is an explanation of the present application and should not be considered as a limitation thereof. Although several exemplary embodiments of the present application are described, it will be readily understood by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present application. Therefore, all of these modifications are intended to be included within the scope of the present application as defined by the claims. It should be understood that the above is an explanation of the present application and should not be considered to be limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present application is defined by the claims and their equivalents.

Claims

1. An intelligent remote control method for bucket wheel excavator based on video monitoring, characterized in that: include: Obtain bucket wheel status monitoring video collected by the camera; Extracting the first frame and the last frame from the bucket wheel state monitoring video to obtain a first bucket wheel state monitoring image and a second bucket wheel state monitoring image; Extracting bucket state features of the first bucket state monitoring image and the second bucket state monitoring image to obtain a sequence of first bucket state local feature vectors and a sequence of second bucket state local feature vectors; and Based on the comparative semantic association relationship between the sequence of the first wheel bucket state local feature vectors and the sequence of the second wheel bucket state local feature vectors, it is determined whether the wheel bucket is damaged.

2. The method for intelligent remote control of bucket wheel excavator based on video monitoring according to claim 1 is characterized in that: Extracting the wheel bucket state features of the first wheel bucket state monitoring image and the second wheel bucket state monitoring image to obtain a sequence of first wheel bucket state local feature vectors and a sequence of second wheel bucket state local feature vectors, including: A deep learning network model is used to perform feature extraction on the first wheel bucket state monitoring image and the second wheel bucket state monitoring image to obtain a sequence of the first wheel bucket state local feature vectors and a sequence of the second wheel bucket state local feature vectors.

3. The intelligent remote control method for bucket wheel excavator based on video monitoring according to claim 2 is characterized in that: The deep learning network model is a wheel bucket state feature extractor based on the ViT model; The method of using a deep learning network model to extract features from the first wheel bucket state monitoring image and the second wheel bucket state monitoring image to obtain a sequence of local feature vectors of the first wheel bucket state and a sequence of local feature vectors of the second wheel bucket state includes: The first wheel bucket state monitoring image and the second wheel bucket state monitoring image are respectively passed through the wheel bucket state feature extractor based on the ViT model to obtain a sequence of the first wheel bucket state local feature vectors and a sequence of the second wheel bucket state local feature vectors.

4. The method for intelligent remote control of bucket wheel excavator based on video monitoring according to claim 3 is characterized in that: The first wheel bucket state monitoring image and the second wheel bucket state monitoring image are respectively passed through the wheel bucket state feature extractor based on the ViT model to obtain a sequence of the first wheel bucket state local feature vector and a sequence of the second wheel bucket state local feature vector, including: Separately dividing the first wheel bucket state monitoring image and the second wheel bucket state monitoring image into image blocks to obtain a sequence of first wheel bucket state monitoring image blocks and a sequence of second wheel bucket state monitoring image blocks; Using the image block embedding layer of the wheel bucket state feature extractor based on the ViT model, each first wheel bucket state monitoring image block in the sequence of the first wheel bucket state monitoring image blocks and each second wheel bucket state monitoring image block in the sequence of the second wheel bucket state monitoring image blocks are respectively embedded to obtain a sequence of first wheel bucket state monitoring image block embedding vectors and a sequence of second wheel bucket state monitoring image block embedding vectors; and The sequence of the first wheel bucket state monitoring image block embedding vectors and the sequence of the second wheel bucket state monitoring image block embedding vectors are passed through the wheel bucket state feature extractor based on the ViT model to obtain a sequence of the first wheel bucket state local feature vectors and a sequence of the second wheel bucket state local feature vectors.

5. The method for intelligent remote control of bucket wheel excavator based on video monitoring according to claim 4 is characterized in that: Determining whether the wheel bucket is damaged based on a comparison semantic association relationship between a sequence of the first wheel bucket state local feature vectors and a sequence of the second wheel bucket state local feature vectors includes: Performing feature distribution correction on the sequence of the second wheel bucket state local feature vectors to obtain a corrected sequence of the second wheel bucket state local feature vectors; Constructing a contrast semantic association relationship between each corresponding group of the first wheel bucket state local feature vector and the corrected second wheel bucket state local feature vector in the sequence of the first wheel bucket state local feature vector and the sequence of the corrected second wheel bucket state local feature vector to obtain a wheel bucket state global contrast semantic feature vector; and The global comparison semantic feature vector of the wheel bucket state is passed through a classifier to obtain a classification result, and the classification result is used to indicate whether the wheel bucket is damaged.

6. The method for intelligent remote control of bucket wheel excavator based on video monitoring according to claim 5 is characterized in that: Constructing a comparative semantic association relationship between each group of corresponding first wheel bucket state local feature vectors and corrected second wheel bucket state local feature vectors in the sequence of the first wheel bucket state local feature vectors and the sequence of the corrected second wheel bucket state local feature vectors to obtain a wheel bucket state global comparative semantic feature vector, including: The correlation between each group of corresponding first wheel bucket state local feature vectors and corrected second wheel bucket state local feature vectors in the sequence of the first wheel bucket state local feature vectors and the sequence of the corrected second wheel bucket state local feature vectors is calculated respectively to obtain the wheel bucket state global comparison semantic feature vector composed of multiple correlations.

7. The method for intelligent remote control of bucket wheel excavator based on video monitoring according to claim 6 is characterized in that: Respectively calculating the correlation between each group of corresponding first wheel bucket state local feature vectors and corrected second wheel bucket state local feature vectors in the sequence of the first wheel bucket state local feature vectors and the sequence of the corrected second wheel bucket state local feature vectors to obtain the wheel bucket state global comparison semantic feature vector composed of multiple correlations, including: The correlation between each corresponding group of the first wheel bucket state local feature vector and the corrected second wheel bucket state local feature vector in the sequence of the first wheel bucket state local feature vector and the sequence of the corrected second wheel bucket state local feature vector is calculated using the following correlation formula; Wherein, the correlation formula is: S i =h i T r j Among them, h i is the i-th first wheel bucket state local feature vector in the sequence of the first wheel bucket state local feature vectors, (·) T represents the transpose operation, r j is the jth corrected second wheel bucket state local feature vector in the sequence of the corrected second wheel bucket state local feature vectors, S i is the feature value of the ith position in the global comparison semantic feature vector of the wheel bucket state.

8. The method for intelligent remote control of bucket wheel excavator based on video monitoring according to claim 7 is characterized in that: The global comparison semantic feature vector of the wheel bucket state is passed through a classifier to obtain a classification result, and the classification result is used to indicate whether the wheel bucket is damaged, including: Using the fully connected layer of the classifier to perform fully connected encoding on the bucket state global contrast semantic feature vector to obtain an encoded classification feature vector; and The encoded classification feature vector is input into the Softmax classification function of the classifier to obtain the classification result.

9. An intelligent remote control system for bucket wheel excavator based on video monitoring, characterized in that: include: A video acquisition module is used to acquire bucket wheel status monitoring video collected by a camera; A key frame extraction module, used for extracting the first frame and the last frame from the bucket wheel state monitoring video to obtain a first bucket wheel state monitoring image and a second bucket wheel state monitoring image; a state feature extraction module, used for extracting the wheel bucket state features of the first wheel bucket state monitoring image and the second wheel bucket state monitoring image to obtain a sequence of first wheel bucket state local feature vectors and a sequence of second wheel bucket state local feature vectors; and The damage analysis module is used to determine whether the wheel bucket is damaged based on the comparative semantic association relationship between the sequence of the first wheel bucket state local feature vectors and the sequence of the second wheel bucket state local feature vectors.

10. The intelligent remote control system for bucket wheel excavator based on video monitoring according to claim 9 is characterized in that: The state feature extraction module is used to: A deep learning network model is used to perform feature extraction on the first wheel bucket state monitoring image and the second wheel bucket state monitoring image to obtain a sequence of the first wheel bucket state local feature vectors and a sequence of the second wheel bucket state local feature vectors.

Images