Robot monitoring method, device, robot monitoring equipment and storage medium

Abstract

The present invention discloses a robot monitoring method, apparatus, robot monitoring device, and storage medium, comprising: receiving an operation image sent by a robot, and receiving the robot's operation position, determining whether the robot is operating abnormally based on the operation image; if so, determining the abnormal position corresponding to the abnormal operation; determining a first position corresponding to the operation position on a pre-set circuit diagram, and determining a second position corresponding to the abnormal position; displaying a robot identifier at the first position on the circuit diagram, and displaying an abnormal operation identifier at the second position. This method automatically identifies abnormal operations and displays corresponding identifiers on the circuit diagram to alert monitoring personnel of robot operation abnormalities. Manual observation of images is unnecessary to determine robot abnormalities, reducing the workload of personnel monitoring the robot. Furthermore, the determination of robot abnormalities is not affected by subjective factors of personnel, and the accuracy of determining robot operation abnormalities is high.

Description

Technical Field

[0001] The present invention relates to the field of power installation technology, and in particular to a robot monitoring method, device, robot monitoring equipment and storage medium. Background Art

[0002] At present, with the development of industrial robots, robots are used in distribution networks to perform grid-related operations, such as installation operations and obstacle removal operations.

[0003] At present, cameras are usually installed on robots to collect images during the operation process, and the images are uploaded to the background. In the background, the images are manually observed to determine whether the robot is working abnormally. Personnel are required to watch each image in real time, which has high requirements for personnel and is prone to fatigue. In addition, manual determination of abnormalities is easily affected by subjective factors. Summary of the Invention

[0004] The present invention provides a robot monitoring method, apparatus, robot monitoring equipment and storage medium to solve the problems that the existing robot monitoring by manually observing images collected by the robot has high requirements for personnel, is prone to fatigue, and is easily affected by subjective factors.

[0005] In a first aspect, the present invention provides a robot monitoring method, comprising:

[0006] receiving an operation image sent by the robot and receiving the operation position of the robot, wherein the operation image is an image captured by a camera when the robot is operating;

[0007] determining whether the robot is performing abnormal operation according to the operation image;

[0008] If yes, determine the abnormal location corresponding to the abnormal operation;

[0009] Determining a first position corresponding to the operating position on a preset route map, and determining a second position corresponding to the abnormal position;

[0010] A robot identifier is displayed at a first position on the circuit diagram, and an abnormal operation identifier is displayed at the second position.

[0011] In a second aspect, the present invention provides a robot monitoring device, comprising:

[0012] A data receiving module is used to receive an operation image sent by the robot and receive the operation position of the robot, wherein the operation image is an image captured by the camera when the robot is operating;

[0013] an abnormal operation judgment module, configured to determine whether the robot is performing abnormal operation according to the operation image;

[0014] An abnormal location determination module, used to determine the abnormal location corresponding to the abnormal operation;

[0015] a circuit diagram display position determination module, configured to determine a first position corresponding to the operation position and a second position corresponding to the abnormal position on a preset circuit diagram;

[0016] The identification display module is used to display the robot identification at a first position on the circuit diagram and to display the abnormal operation identification at a second position.

[0017] In a third aspect, the present invention provides a robot monitoring device, the robot monitoring device comprising:

[0018] at least one processor, and

[0019] a memory connected to the at least one processor, wherein:

[0020] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the robot monitoring method described in any one of the first aspects of the present invention.

[0021] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement any robot monitoring method described in the first aspect of the present invention when executed.

[0022] After receiving the operation image sent by the robot and the operation position of the robot, the embodiment of the present invention determines whether the robot is operating abnormally based on the operation image. If so, the abnormal position corresponding to the abnormal operation is determined, and a first position corresponding to the operation position and a second position corresponding to the abnormal position are determined on a pre-set circuit map. The robot identifier is displayed at the first position on the circuit map, and the abnormal operation identifier is displayed at the second position. This realizes the identification of whether the robot is abnormal based on the operation image collected during the robot operation, and displays the robot identifier at the first position of the robot on the circuit map when abnormal operation exists, and displays the abnormal operation identifier at the second position of the abnormal operation. This realizes the automatic identification of abnormal operation and the display of the corresponding identifier on the circuit map to prompt the monitoring personnel that the robot operation is abnormal. There is no need to manually observe the image to determine the robot abnormality, which reduces the workload of the personnel monitoring the robot. The determination of the robot abnormality is not affected by the subjective factors of the personnel, and the accuracy of determining the robot operation abnormality is high.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 This is a flow chart of a robot monitoring method provided by Example 1 of the present invention;

[0026] Figure 2 This is a flow chart of a robot monitoring method provided by Embodiment 2 of the present invention;

[0027] Figure 3 This is a schematic structural diagram of a robot monitoring device provided by Embodiment 3 of the present invention;

[0028] Figure 4 It is a structural diagram of the robot monitoring device provided in Example 4 of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0030] Example 1

[0031] Figure 1 This is a flowchart of a robot monitoring method provided in the first embodiment of the present invention. This embodiment is applicable to monitoring robots performing operations on power transmission lines. The method can be performed by a robot monitoring device, which can be implemented in the form of hardware and / or software and can be configured in a robot monitoring device. Figure 1 As shown, the robot monitoring method includes:

[0032] S101: Receive an operation image sent by a robot and receive the operation position of the robot. The operation image is an image collected by a camera when the robot is operating.

[0033] The robot of this embodiment can be a robot that performs various operations on the transmission lines in the distribution network, such as a tow-clearing robot, an insulating sheath installation robot, a binding belt robot or other robots, and the corresponding operations can be tow-clearing operations, insulating sheath installation operations, binding belt operations, etc., wherein the robot can communicate with the robot monitoring device, and the robot monitoring device can be a remote monitoring device, such as a server, or an on-site monitoring device, such as a host computer that communicates with the robot.

[0034] The robot is provided with a camera to capture images of the working area when the robot is working on the transmission line, and the captured images are sent to the robot monitoring device through the communication network. The robot monitoring device can capture the working images and store the working images in the memory. In addition, a positioning module is also provided on the robot. When the positioning module locates the robot and obtains the working position of the robot, the working position together with the working image is sent to the robot monitoring device, and the robot monitoring device can receive the working position of the robot.

[0035] S102: Determine whether the robot has any abnormal operation based on the operation image.

[0036] In one embodiment, a work image can be input into a pre-trained work detection model to obtain a robot work detection result. If the work detection result is abnormal, it is determined that the robot is performing an abnormal work. The work detection model can be pre-trained using images of abnormal work, so that the work detection model can learn to identify abnormal work from work images.

[0037] In another embodiment, a template image for normal operation can be set, and the similarity between the collected operation image and the template image can be calculated. When the similarity is less than a threshold, it means that the operation image and the template image are very different, and it is determined that the robot is operating abnormally.

[0038] Of course, those skilled in the art may also determine whether the robot is operating abnormally by other means, and this embodiment does not limit this.

[0039] S103: Determine the abnormal location corresponding to the abnormal operation.

[0040] The abnormal position may be the position of the corresponding work object during abnormal operation. In one example, taking the installation of an insulating sheath as an example, the abnormal position may refer to the position when the installed insulating sheath becomes abnormal. The abnormal position may be the position in the world coordinate system or the position relative to the robot's coordinate system.

[0041] This embodiment can determine the abnormal position of the abnormal operation in the world coordinate system based on the working position of the robot and the image position of the abnormal operation in the working image. For example, the image position of the abnormal operation in the image can be converted into the position in the camera coordinate system based on the working position of the working robot and the calibration parameters of the camera, and then the position in the camera coordinate system can be converted into the position in the world coordinate system, so that the abnormal position of the abnormal operation in the world coordinate system can be obtained.

[0042] S104: Determine a first position corresponding to the operation position on a preset route map, and determine a second position corresponding to the abnormal position.

[0043] The circuit diagram can be an electronic circuit diagram, on which elements such as transmission lines and towers are displayed. The electronic circuit diagram has map coordinates. Then, based on the conversion relationship from world coordinates to map coordinates, the robot's operating position can be converted to a position in map coordinates, that is, the corresponding first position of the operating position on the circuit diagram is obtained. Similarly, the abnormal position of an abnormal operation can be converted to a position in map coordinates, that is, the corresponding second position of the abnormal position on the circuit diagram is obtained. Of course, the map coordinates and world coordinates can also be unified without the need for coordinate conversion. In this case, the operating position is the first position on the circuit diagram, and the abnormal position is the second position on the circuit diagram.

[0044] S105: Display a robot identifier at a first position on the circuit diagram, and display an abnormal operation identifier at a second position.

[0045] The robot identifier represents the robot and can be a thumbnail of the robot or a commonly used identifier for robots in the industrial field. The abnormal operation identifier can be an identifier that serves as a warning. By displaying the robot identifier at a first position on the circuit map, monitoring personnel can determine the robot's location based on the circuit map. By displaying the abnormal operation identifier at a second position, monitoring personnel can determine the presence of abnormal operation at that location on the circuit map, prompting the monitoring personnel to pay attention to the operation at the second location. For example, the monitoring personnel can touch or click the abnormal operation identifier, and in response to this operation, an operation image can be displayed, allowing the operator to view the actual abnormal operation through the displayed operation image.

[0046] After receiving the operation image sent by the robot and the operation position of the robot, the embodiment of the present invention determines whether the robot has an abnormal operation based on the operation image. If so, the abnormal position corresponding to the abnormal operation is determined, and a first position corresponding to the operation position and a second position corresponding to the abnormal position are determined on a pre-set circuit map. The robot identifier is displayed at the first position on the circuit map, and the abnormal operation identifier is displayed at the second position. This realizes the identification of whether the robot is abnormal through the operation image collected during the robot operation, and displays the robot identifier at the first position of the robot on the circuit map when an abnormal operation exists, and displays the abnormal operation identifier at the second position of the abnormal operation. This realizes the automatic identification of abnormal operation and the display of the corresponding identifier on the circuit map to prompt the monitoring personnel that the robot operation is abnormal. There is no need for manual real-time observation of the image to determine the robot abnormality, which reduces the workload of the personnel monitoring the robot. The determination of the robot abnormality is not affected by the subjective factors of the personnel, and the accuracy of determining the robot operation abnormality is high.

[0047] Example 2

[0048] Figure 2 This is a flowchart of a robot monitoring method provided in the second embodiment of the present invention. The embodiment of the present invention is optimized based on the above-mentioned first embodiment. Figure 2 As shown, the robot monitoring method includes:

[0049] S201: Receive an operation image sent by the robot and receive the operation position of the robot. The operation image is an image collected by the camera when the robot is operating.

[0050] The robot of this embodiment can be a robot used in obstacle clearance operations, insulation sheath installation operations, binding operations, insulator cleaning operations, etc. Taking the insulation sheath installation operation as an example, when the robot performs the insulation sheath installation operation, the robot uses a camera to capture an image of the insulation sheath installed on the line as an operation image, and obtains the robot's operating position through the positioning module on the robot. The operation position can be the position of the robot when the image is captured. The robot sends the operation image and operation position to the robot monitoring device through the communication network, and the robot monitoring device can receive the operation image and operation position.

[0051] S202: Input the operation image into the operation detection model to obtain the robot operation detection result.

[0052] In one embodiment, when training job images, training images can be obtained. The training images can be images of the robot performing the job after annotating the images with normal and abnormal labels. Then, N training images are randomly extracted and input into the job detection model to obtain the job detection results of each training image. The job results include normal and abnormal installation results. Then, the loss rate is calculated based on the job results and the annotated labels of the N training images to determine whether the loss rate is less than a preset loss rate threshold. If so, stop training the job detection model to obtain a trained job detection model. If not, perform gradient descent on the parameters of the job detection model according to the loss rate, and return to the step of randomly extracting N training images and inputting them into the job detection model.

[0053] Among them, the job detection model can be various neural networks. When labeling, different labels can be labeled according to different job types. For example, in addition to labeling whether there is an abnormality in the image, the score or probability of the existence of the abnormality, the score or probability that the abnormality belongs to a certain type, the image position of the abnormal job in the image, etc. can also be labeled. The loss function can be a mean square error loss function, a cross entropy loss function, etc. This embodiment does not limit the calculation method of the loss rate.

[0054] S203: When the operation detection result is abnormal, determine that the robot has abnormal operation.

[0055] In this embodiment, the operation detection result may include a score indicating the presence of abnormal operation in the operation image. When the score is greater than a score threshold, it is determined that abnormal operation exists in the operation detection result, that is, it is determined that the robot has performed abnormal operation.

[0056] In another embodiment, the robot of this embodiment may be an insulating sheath installation robot that can capture an image of the wiring after the insulating sheath is installed as a template image. After capturing the work image, the robot can identify the wire area in the work image and compare the color of the wire area with the color of the wire in the template image. When the color difference is greater than a threshold, it is determined that an insulating sheath installation anomaly has occurred. Of course, those skilled in the art can also use other methods to determine whether the robot has performed an abnormal operation based on the image, and this embodiment does not limit this.

[0057] In another embodiment, when it is determined that the robot is operating normally, the operated route is determined according to the operating position, and the operated route is set to a preset color in the route map. In one example, the operating position of the robot can be used as a dividing point, and the route behind the robot's walking direction can be determined as the operated route. In another example, the route between the operating starting point and the robot's operating position can be determined as the operated route, and the color of the operated route is set to a preset color, illustratively, to green, so that the monitoring personnel can quickly determine the operated route based on the color of the line.

[0058] S204: Determine the abnormal location corresponding to the abnormal operation.

[0059] In this embodiment, the operation detection result also includes the image position of the abnormal operation in the image. The image position can be converted into a third position in the camera coordinate system, and the third position can be converted into a fourth position in the world coordinate system using the pre-set camera calibration parameters as the abnormal position corresponding to the abnormal operation. The coordinate system conversion can refer to the existing coordinate conversion and will not be described in detail here.

[0060] S205: Determine a first position corresponding to the operation position on a preset route map, and determine a second position corresponding to the abnormal position.

[0061] The electronic circuit diagram of this embodiment displays elements such as transmission lines and towers. The electronic circuit diagram has map coordinates. Therefore, the robot's working position can be converted into a position under the map coordinates based on the conversion relationship from world coordinates to map coordinates, that is, the corresponding first position of the working position on the circuit map is obtained. Similarly, the abnormal position of the abnormal operation is converted into a position under the map coordinates, that is, the corresponding second position of the abnormal position on the circuit map is obtained.

[0062] S206: Display a robot identifier at a first position on the circuit diagram, and display an abnormal operation identifier at a second position.

[0063] The robot identifier represents the robot and can be a thumbnail of the robot or a commonly used identifier for robots in the industrial field. The abnormal operation identifier can be an identifier that serves as a warning. By displaying the robot identifier at a first position on the circuit map, monitoring personnel can determine the robot's location based on the circuit map. By displaying the abnormal operation identifier at a second position, monitoring personnel can determine the presence of abnormal operation at that location on the circuit map, prompting the monitoring personnel to pay attention to the operation at the second location. For example, the monitoring personnel can touch or click the abnormal operation identifier, and in response to this operation, an operation image can be displayed, allowing the operator to view the actual abnormal operation through the displayed operation image.

[0064] S207 : In response to the operation on the abnormal operation mark, display the operation image at the second position and display the heavy operation mark.

[0065] In this embodiment, the abnormal operation indicator can be configured as a control. After the abnormal operation indicator is displayed, the monitoring personnel can click or touch it to trigger the display of the operation image and the re-operation indicator in the second position. The operation image is an image captured during the abnormal operation, and the re-operation indicator is used to trigger the robot to stop the operation. For example, the operation image and the re-operation indicator can be displayed in the form of a pop-up window in the second position, allowing the monitoring personnel to view the actual abnormal operation and determine whether to trigger the robot to re-operate using the re-operation indicator.

[0066] S208. In response to the operation acting on the heavy operation indicator, send a heavy operation instruction to the robot.

[0067] When the monitoring personnel determines that the abnormal operation needs to be re-operated, the monitoring personnel can operate the re-operation mark, such as clicking, touching, etc., to trigger the sending of the re-operation instruction to the robot. After receiving the re-operation instruction, the robot performs the re-operation on the abnormal operation.

[0068] In another embodiment, the robot monitoring device may further display an operation control upon receiving a stop operation request from the robot. The stop operation request includes a target image of the abnormal operation. The operation control is used to control the robot to stop the operation, display the target image, and send a stop operation instruction to the robot in response to a user's operation on the operation control. In actual application, when the robot detects an abnormal operation during operation, it may send a stop operation request to the robot monitoring device. The operation request may include a target image of the abnormal operation. The robot monitoring device may display the operation control and the target image. The monitoring personnel may determine the abnormal operation by viewing the target image. When it is determined that the robot needs to be stopped, the monitoring personnel may operate the operation control to determine whether to stop the operation. Exemplarily, the operation control includes options for allowing and prohibiting the operation. When the monitoring personnel selects to allow the operation to stop, the robot sends a stop operation instruction to the monitoring device, and the robot stops the operation after receiving the instruction. When the monitoring personnel selects to prohibit the operation to stop, the robot sends a prohibition stop operation instruction to the monitoring device, and the robot continues to perform the operation after receiving the instruction. In this way, when the robot detects a special event and requests to stop the operation, the monitoring personnel can determine whether to control the robot to stop the operation.

[0069] In another embodiment, the number of abnormal operations and the distance worked by the robot can be counted, and the ratio of the distance worked to the number of abnormal operations can be calculated. When the ratio is less than or equal to a preset ratio, a prompt message is displayed and a stop operation instruction is sent to the robot. By calculating the ratio of the distance worked to the number of abnormal operations, the operation distance-abnormal number ratio can be obtained. The smaller the operation distance-abnormal number ratio, the more frequently abnormal operations occur. When the ratio is less than a preset threshold, such as the preset threshold is 1000 meters / 5 times, when the operation distance-abnormal number ratio is 100 meters / 5 times, it means that the robot is abnormal. If the robot continues to work, it will cause too many abnormal operations and the working efficiency will be reduced. A prompt message can be generated to remind the monitoring personnel, and a stop operation instruction can be generated and sent to the robot to control the robot to stop working. This realizes the automatic determination of whether to control the robot to stop working based on the ratio of the distance worked to the number of abnormal operations, without the need for the monitoring personnel to count the number of abnormal operations and the distance worked to control the robot to stop working.

[0070] After receiving the robot's work image and work position, this embodiment inputs the work image into a work detection model to obtain a robot work detection result. If the work detection result is abnormal, the robot is determined to have performed an abnormal operation. The abnormal position corresponding to the abnormal operation is further determined. A first position corresponding to the work position and a second position corresponding to the abnormal position are determined on a pre-set circuit map. Then, a robot identifier is displayed at the first position on the circuit map, along with an abnormal work identifier at the second position. In response to an operation on the abnormal work identifier, the work image and a rework identifier are displayed at the second position. In response to an operation on the rework identifier, a rework instruction is sent to the robot. This embodiment automatically identifies abnormal operations and displays corresponding identifiers on the circuit map to alert monitoring personnel of robot operation anomalies. This eliminates the need for manual real-time image observation to determine robot anomalies, reducing the workload for monitoring personnel. Furthermore, the determination of robot anomalies is unaffected by subjective factors, resulting in high accuracy in determining robot operation anomalies. The rework identifier is displayed to allow monitoring personnel to determine whether to control the robot to perform a rework operation. This provides a convenient interactive monitoring interface, facilitating timely control of robot rework by monitoring personnel, and improving the completion rate of robot operations.

[0071] Example 3

[0072] Figure 3 This is a schematic diagram of the structure of a robot monitoring device provided by the third embodiment of the present invention. Figure 3 As shown, the robot monitoring device includes:

[0073] The data receiving module 301 is used to receive the operation image sent by the robot and the operation position of the robot. The operation image is an image captured by the camera of the robot during operation.

[0074] An abnormal operation judgment module 302 is used to determine whether the robot is performing abnormal operation according to the operation image;

[0075] An abnormal location determination module 303 is used to determine the abnormal location corresponding to the abnormal operation;

[0076] A circuit diagram display position determination module 304 is configured to determine a first position corresponding to the operation position and a second position corresponding to the abnormal position on a preset circuit diagram;

[0077] The identification display module 305 is used to display the robot identification at the first position on the circuit diagram and to display the abnormal operation identification at the second position.

[0078] Optionally, the abnormal operation judgment module 302 includes:

[0079] An operation image input unit, configured to input the operation image into an abnormal operation detection model to obtain the robot operation detection result;

[0080] The operation abnormality judgment unit is used to determine that the robot has an abnormal operation when the operation detection result is abnormal.

[0081] Optionally, the job detection result includes an image position corresponding to the abnormal job in the job image, and the abnormal position determination module 303 includes:

[0082] An image position conversion unit, configured to convert the image position into a third position in a camera coordinate system;

[0083] The abnormal position conversion unit is used to convert the third position into a fourth position in the world coordinate system by using a pre-set camera calibration parameter, so as to serve as the abnormal position corresponding to the abnormal operation.

[0084] Optionally, it also includes:

[0085] an operation image and heavy operation mark display module, configured to display the operation image and the heavy operation mark at the second position in response to an operation on the abnormal operation mark;

[0086] The heavy operation instruction sending module is used to respond to the operation acting on the heavy operation identifier and send a heavy operation instruction to the robot.

[0087] Optionally, it also includes:

[0088] An operated route determining module, configured to determine an operated route according to the operating position when determining that the robot is operating normally;

[0089] The line coloring module is used to set the operated lines in the line diagram to a preset color.

[0090] Optionally, it also includes:

[0091] a control display module, configured to display an operation control upon receiving a stop operation request from the robot, wherein the stop operation request includes a target image of an abnormal operation, and the operation control is used to control the robot to stop the operation;

[0092] A target image display module, configured to display the target image;

[0093] The stop operation instruction sending module is used to send a stop operation instruction to the robot in response to the user's operation on the operation control.

[0094] Optionally, it also includes:

[0095] A statistics module, used to count the number of abnormal operations and the distance operated by the robot;

[0096] a ratio calculation module, configured to calculate a ratio of the operated distance to the number of abnormal operations;

[0097] The stop operation instruction sending module is used to display a prompt message and send a stop operation instruction to the robot when the ratio is less than or equal to a preset ratio.

[0098] The robot monitoring device provided in the embodiment of the present invention can execute the robot monitoring method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0099] Example 4

[0100] Figure 4 A schematic diagram of a robotic monitoring device 40 that can be used to implement embodiments of the present invention is shown. The robotic monitoring device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The robotic monitoring device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0101] like Figure 4As shown, the robot monitoring device 40 includes at least one processor 41 and memory, such as a read-only memory (ROM) 42 and a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the ROM 42 or loaded from the storage unit 48 into the RAM 43. The RAM 43 can also store various programs and data required for the operation of the robot monitoring device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0102] Several components in the robot monitoring device 40 are connected to an I / O interface 45, including an input unit 46, such as a keyboard and mouse; an output unit 47, such as various types of displays and speakers; a storage unit 48, such as a magnetic disk and optical disk; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the robot monitoring device 40 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0103] Processor 41 can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. Processor 41 executes the various methods and processes described above, such as the robot monitoring method.

[0104] In some embodiments, the robot monitoring method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program can be loaded and / or installed on the robot monitoring device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the robot monitoring method described above can be performed. Alternatively, in other embodiments, processor 41 can be configured to perform the robot monitoring method in any other suitable manner (e.g., via firmware).

[0105] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0106] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0107] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0108] To provide interaction with a user, the systems and techniques described herein can be implemented on a robot monitoring device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the robot monitoring device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0109] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0110] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0111] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0112] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A robot monitoring method, characterized in that: Used to monitor robots working on power lines, including: receiving an operation image sent by a robot and receiving the operation position of the robot, wherein the operation image is an image captured by a camera of the robot during operation; determining, based on the operation image, whether the robot is performing an abnormal operation, wherein the abnormal operation comprises one of an obstacle removal operation, an insulation sheath installation operation, and a binding tape operation performed by the robot; If yes, determine the abnormal location corresponding to the abnormal operation; Determining a first position corresponding to the operating position on a preset route map, and determining a second position corresponding to the abnormal position; Displaying a robot identifier at a first position on the circuit diagram, and displaying an abnormal operation identifier at the second position; Counting the number of abnormal operations and the distance operated by the robot; Calculating a ratio of the operated distance to the number of abnormal operations; When the ratio is less than or equal to a preset ratio, a prompt message is displayed and a stop operation instruction is sent to the robot; The determining, based on the operation image, whether the robot is operating abnormally includes: Inputting the operation image into the operation detection model to obtain the robot operation detection result; When the operation detection result is abnormal, determining that the robot has abnormal operation; The job detection model is trained through the following steps: Acquire training images; the training images are images collected by the robot performing tasks, and are labeled with labels such as normal, abnormal, and task type, as well as scores or probabilities of abnormalities, scores or probabilities of abnormalities belonging to various task types, and image positions of abnormal tasks in the images; Then, N training images are randomly extracted and input into the job detection model to obtain the job detection results of each training image; Calculate the loss rate based on the job detection results and annotated labels of N training images; Determine whether the loss rate is less than a preset loss rate threshold; If so, stop training the job detection model to obtain the trained job detection model; If not, perform gradient descent on the parameters of the job detection model according to the loss rate, and return to the step of randomly extracting N training images and inputting them into the job detection model.

2. The robot monitoring method according to claim 1, wherein: The job detection result includes an image position corresponding to the abnormal job in the job image, and determining the abnormal position corresponding to the abnormal job includes: Converting the image position to a third position in a camera coordinate system; The third position is converted into a fourth position in a world coordinate system using pre-set camera calibration parameters to serve as the abnormal position corresponding to the abnormal operation.

3. The robot monitoring method according to any one of claims 1 to 2, characterized in that: After displaying a robot identifier at a first position on the circuit diagram and displaying an abnormal operation identifier at a second position, the method further includes: In response to an operation on the abnormal operation indicator, displaying the operation image at the second position and displaying a heavy operation indicator; In response to the operation acting on the heavy operation indicator, a heavy operation instruction is sent to the robot.

4. The robot monitoring method according to any one of claims 1 to 2, characterized in that: After determining whether the robot is performing abnormal operation according to the operation image, the method further includes: When it is determined that the robot is operating normally, determining an operated route according to the operating position; After displaying a robot identifier at a first position on the circuit diagram and displaying an abnormal operation identifier at a second position, the method further includes: In the circuit diagram, the operated circuit is set to a preset color.

5. The robot monitoring method according to any one of claims 1 to 2, characterized in that: After displaying a robot identifier at a first position on the circuit diagram and displaying an abnormal operation identifier at a second position, the method further includes: Upon receiving a stop operation request from the robot, displaying an operation control, wherein the stop operation request includes a target image of the abnormal operation, and the operation control is used to control the robot to stop the operation; displaying the target image; In response to a user's operation on the operation control, a stop operation instruction is sent to the robot.

6. A robot monitoring device, characterized in that: Used to monitor robots working on power lines, including: A data receiving module is used to receive an operation image sent by the robot and receive the operation position of the robot, wherein the operation image is an image captured by the camera when the robot is operating; an abnormal operation judgment module, configured to determine, based on the operation image, whether the robot is performing an abnormal operation, wherein the abnormal operation includes one of an obstacle removal operation, an insulation sheath installation operation, and a binding tape operation performed by the robot; An abnormal location determination module, used to determine the abnormal location corresponding to the abnormal operation; a circuit diagram display position determination module, configured to determine a first position corresponding to the operation position and a second position corresponding to the abnormal position on a preset circuit diagram; an identification display module, configured to display a robot identification at a first position on the circuit diagram and an abnormal operation identification at a second position; Also includes: A statistics module, used to count the number of abnormal operations and the distance operated by the robot; a ratio calculation module, configured to calculate a ratio of the operated distance to the number of abnormal operations; a stop operation instruction sending module, configured to display a prompt message and send a stop operation instruction to the robot when the ratio is less than or equal to a preset ratio; The abnormal operation judgment module includes: An operation image input unit, configured to input the operation image into an abnormal operation detection model to obtain the robot operation detection result; an operation abnormality judgment unit, configured to determine that the robot has an abnormal operation when the operation detection result is abnormal; The job detection model is trained through the following steps: Acquire training images; the training images are images collected by the robot performing tasks, and are labeled with labels such as normal, abnormal, and task type, as well as scores or probabilities of abnormalities, scores or probabilities of abnormalities belonging to various task types, and image positions of abnormal tasks in the images; Then, N training images are randomly extracted and input into the job detection model to obtain the job detection results of each training image; Calculate the loss rate based on the job detection results and annotated labels of N training images; Determine whether the loss rate is less than a preset loss rate threshold; If so, stop training the job detection model to obtain the trained job detection model; If not, perform gradient descent on the parameters of the job detection model according to the loss rate, and return to the step of randomly extracting N training images and inputting them into the job detection model.

7. A robot monitoring device, characterized in that: The robot monitoring device comprises: at least one processor, and a memory connected to the at least one processor, wherein: The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the robot monitoring method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the robot monitoring method according to any one of claims 1 to 5 when executed.

Images