Inspection monitoring method and device, electronic equipment and storage medium

Abstract

The invention discloses an inspection monitoring method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining at least one inspection data message corresponding to a target robot in an inspection process by responding to a trigger operation of inspection processing, the inspection data message comprising a first message and a second message; determining a message time difference value according to the first receiving timestamp of the first message and the second receiving timestamp of the second message; according to the message time difference, an inspection video frame obtained through analysis of the first message and inspection record information and robot state information obtained through analysis of the second message, video reconstruction processing is carried out on the inspection process of the robot, and a target inspection video is obtained; and performing playback processing on the inspection process of the target robot according to the target inspection video, and determining an inspection monitoring result. According to the method, direct communication between the robot and the client is realized, the problem of polling monitoring lag caused by the aid of a transit server is avoided, and the real-time performance of polling monitoring is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a patrol monitoring method, device, electronic equipment and storage medium. Background Art

[0002] In order to ensure the safety of the track, inspection robots are usually arranged to inspect the track. During the inspection process, the inspection robots may encounter various emergencies, such as obstacles on the track, track damage, signal failure and other problems.

[0003] In order to record the inspection situation, the inspection data during the inspection process will be uploaded to the server for storage, so that the client can obtain and view the inspection data from the server. However, there is an intermediate link in the data storage in the above method. That is, after the inspection data is uploaded and stored on the server, the client obtains the inspection data locally, which is easily limited by bandwidth, affecting the real-time transmission and access speed of the inspection data, causing lags in the inspection process monitoring, and failing to handle abnormal inspection situations in a timely manner. In addition, due to the limited storage space of the server, it is also necessary to clean and maintain the inspection data regularly. Summary of the invention

[0004] The present invention provides a patrol monitoring method, device, electronic device and storage medium, which realize a method of direct communication between a robot and a client, avoids the problem of patrol monitoring lag caused by relying on a transit server, and improves the real-time performance of patrol monitoring.

[0005] According to one aspect of the present invention, a patrol monitoring method is provided, the method comprising:

[0006] In response to a trigger operation of the inspection process, at least one inspection data message corresponding to the target robot during the inspection process is obtained, wherein the inspection data message includes a first message and a second message, the first message is a message obtained after the target robot encodes the obtained inspection video frame, and the second message is a message obtained after the target robot performs data preprocessing on the obtained original inspection position record information and original robot state information;

[0007] Determine a message time difference according to a first receiving timestamp of the first message and a second receiving timestamp of the second message;

[0008] According to the message time difference, the inspection video frame obtained by parsing the first message, the inspection record information and the robot status information obtained by parsing the second message, the inspection process of the robot is reconstructed to obtain the target inspection video;

[0009] The inspection process of the target robot is replayed based on the target inspection video to determine the inspection monitoring results.

[0010] According to another aspect of the present invention, there is provided a patrol monitoring device, the device comprising:

[0011] A message acquisition module, used to respond to the trigger operation of the inspection process and acquire at least one inspection data message corresponding to the target robot during the inspection process, wherein the inspection data message includes a first message and a second message, the first message is a message obtained after the target robot encodes the acquired inspection video frame, and the second message is a message obtained after the target robot performs data preprocessing on the acquired original inspection position record information and original robot state information;

[0012] A time difference determination module, used to determine a message time difference according to a first reception timestamp of the first message and a second reception timestamp of the second message;

[0013] The inspection video determination module is used to perform video reconstruction processing on the robot's inspection process according to the message time difference, the inspection video frame obtained by parsing the first message, and the inspection record information and robot status information obtained by parsing the second message to obtain the target inspection video;

[0014] The monitoring result determination module is used to replay the inspection process of the target robot based on the target inspection video to determine the inspection monitoring result.

[0015] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by at least one processor so that the at least one processor can execute the inspection and monitoring method of any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the inspection and monitoring method of any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, a computer program product is provided, including a computer program, characterized in that when the computer program is executed by a processor, the inspection and monitoring method according to any embodiment of the present invention is implemented.

[0021] The technical solution of the embodiment of the present invention, in response to the trigger operation of the inspection process, obtains at least one inspection data message corresponding to the target robot in the inspection process, and the inspection data message includes a first message and a second message. Based on this, the communication between the client and the target robot is realized, and the robot directly processes the collected inspection video frame, inspection position record information and robot status information into a message and sends it to the client, avoiding the problem of inspection monitoring lag caused by the transit server, and reducing the hardware cost required for inspection monitoring. According to the first receiving timestamp of the first message and the second receiving timestamp of the second message, the message time difference is determined; according to the message time difference, the inspection video frame obtained by parsing the first message, and the inspection record information and robot status information obtained by parsing the second message, the robot's inspection process is reconstructed to obtain the target inspection video. The data in the first message and the data in the second message are time-aligned by the message time difference, ensuring the time matching of the message data and the accuracy of the target inspection video. The inspection process of the target robot is replayed according to the target inspection video to determine the inspection monitoring result. Through the inspection and monitoring results, it can be determined whether the target robot needs to be adjusted, ensuring the timely handling of emergencies during the inspection process. The present invention solves the problem of delayed inspection and monitoring caused by uploading inspection data to the server and then obtaining it by the client in the prior art. A method for direct communication between the robot and the client is realized, which improves the real-time performance of inspection and monitoring.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flow chart of a patrol inspection and monitoring method provided by an embodiment of the present invention;

[0025] Figure 2 is an example diagram of a system architecture for implementing a patrol inspection and monitoring method provided by an embodiment of the present invention;

[0026] Figure 3 is a flow chart of a patrol inspection and monitoring method provided by an embodiment of the present invention;

[0027] Figure 4It is a structural schematic diagram of a patrol monitoring device provided by an embodiment of the present invention;

[0028] Figure 5 It is a structural schematic diagram of an electronic device for implementing the inspection and monitoring method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 This is a flow chart of a patrol monitoring method provided by the first embodiment of the present invention. This embodiment is applicable to the case where the client directly establishes communication with the patrol robot and determines the patrol monitoring result according to the patrol data sent by the patrol robot. The method can be executed by a patrol monitoring device, which can be implemented in the form of hardware and / or software. The patrol monitoring device can be configured in electronic devices such as mobile phones, computers or servers. Figure 1 As shown, the method includes:

[0033] S110. In response to a trigger operation of the inspection process, obtaining at least one inspection data message corresponding to the target robot during the inspection process, wherein the inspection data message includes a first message and a second message.

[0034] The first message is a message obtained by the target robot after encoding the acquired inspection video frame, and the second message is a message obtained by the target robot after data preprocessing the acquired original inspection position record information and original robot state information. Optionally, the target robot is a robot that communicates based on the WebRTC communication protocol.

[0035] The trigger operation may be an operation in which the user clicks on a control related to the inspection process on the client. The target robot may be a robot used to inspect preset scenes. For example, the preset scene may be a floor, a street, or other scenes set according to actual needs. Optionally, the target robot may be a track inspection robot, and the target robot may move on a track on which a radio frequency identification (RFID) card is laid. During the movement, the RFID card information is read by a card reader module on the target robot to obtain the current inspection position information of the target robot. The inspection video frames of the target robot during the inspection process are obtained by an image acquisition device deployed on the target robot.

[0036] The inspection position information of the target robot during the inspection process is used as the original inspection position record information. The robot state information of the target robot during the inspection process is used as the original robot state information. The second message is obtained by performing data preprocessing operations such as data cleaning, outlier processing, missing value processing, and data standardization processing on the original inspection position record information and the original robot state information. The WebRTC communication protocol is a browser-based real-time communication protocol that allows two or more devices to communicate video, audio, and data over the network. In an embodiment of the present invention, communication between the target robot and the client is carried out through the WebRTC communication protocol.

[0037] Specifically, when the user clicks the corresponding inspection processing control, the client responds to the trigger operation of the inspection processing and obtains at least one inspection data message of the target robot during the inspection process. Among them, the inspection data message includes a first message and a second message. The specific acquisition method of the first message is: the target robot collects the inspection video frame of the inspection process through the deployed image acquisition device. The inspection video frame is encoded into a data frame of a preset format, and then the first message is obtained. The specific acquisition method of the second message is: the target robot will record the inspection position information in real time during the inspection to obtain the original inspection position record information. And record the status information of the robot during the inspection in real time to obtain the original robot status information. The second message is obtained by performing data preprocessing operations such as data cleaning, outlier processing, missing value processing, and data standardization processing on the original inspection position record information and the original robot status information. Based on this, the target robot can send the first message and the second message to the client when receiving the inspection data acquisition request.

[0038] For example, see Figure 2 , Figure 2 The following is an example diagram of the system architecture for implementing the inspection and monitoring method. The target robot is a track inspection robot, and the image acquisition device of the target robot is a camera. Figure 2 When the target robot moves for inspection on the track paved with RFID cards, the target robot obtains video data during the inspection process through the camera in the peripheral hardware device deployed by it, that is, the inspection video frame mentioned in the above embodiment. The target robot records the position data during the inspection process through the card reader module deployed by it, that is, the original inspection position record information mentioned in the above embodiment. Among them, the original inspection position record information at least includes information such as the card number of the RFID card, the signal strength of the RFID card, and the card status of the RFID card. The moving distance and specific position of the target robot can be determined by the card number and the card number of the RFID card at the starting position of the inspection. The inspection robot obtains its own status information, that is, the original robot status information, through other devices deployed by it. Among them, the original robot status information includes status information such as the battery power of the target robot, the signal strength of the robot antenna, the CPU workload, and the operating speed.

[0039] The embedded system of the target robot may include a video encoding unit, a perception data processing unit, and a status data processing unit. The video encoding unit is used to isolate and encapsulate interfaces compatible with different camera devices, encode the inspection video frames in the YUV format collected by the camera into video message frames in the h264 format, and pass the video message frames to the data fusion module in the embedded system of the target robot. The perception data processing unit is used to isolate and encapsulate the data transmission protocol of the card reader module, receive the original inspection location record information uploaded by the card reader module, perform data preprocessing, obtain the location message message, and pass it to the data fusion module. The status data processing unit is used to isolate and encapsulate different data collection source channels, receive the original robot status information collected by other devices deployed on the target robot, and process the original robot status information into a status message message, and pass it to the data fusion module.

[0040] The data fusion module in the embedded system of the target robot is used to receive and manage the message data transmitted in the video encoding unit, the perception data processing unit and the state data processing unit. The data fusion module can run and maintain three timing queues, the first timing queue is used to store the video message message frames in time sequence. The second timing queue is used to store the position message messages in time sequence. The third timing queue is used to store the state message messages in time sequence. Each data item in each timing queue contains the timestamp of receiving the corresponding message and the corresponding message data. The timestamp of receiving the corresponding message is used to determine the order of the corresponding message in the timing queue. For example, the first timing queue contains the timestamp of receiving the video message message frame and the corresponding video message message frame. It should be noted that in the timing queue, when executing the enqueue operation of the data item, the timestamp of receiving the corresponding message is compared with the timestamp of the existing data item in the timing queue, and the new data item is inserted into the corresponding position according to the order of the timestamps. When executing the dequeue operation of the data item, the earliest data item, that is, the data item with the smallest timestamp, is removed from the front end of the timing queue. The size of the time series queue is checked regularly to avoid insufficient memory caused by unlimited addition of data items. When the number of data items in the time series queue reaches the preset value, the data item with the smallest timestamp is automatically deleted.

[0041] The data fusion module of the target robot also provides a data extraction interface, which is used to extract data from the three time queues when receiving the inspection data acquisition request issued by the client, that is, to perform the data item dequeue operation. Due to the requirement of real-time inspection monitoring, the data is extracted using the data frequency control algorithm. For example, when extracting the Nth video message frame, the position message and status message within the time difference between the N-1th and Nth video message frames are extracted together to obtain three relatively frequency-matched data, thereby ensuring the time matching of the inspection data.

[0042] In an embodiment of the present invention, the method for obtaining the inspection data message may be: in response to the trigger operation of the inspection processing, calling the preset trajectory service interface to obtain the communication address of the target robot based on the preset trajectory service interface; sending an inspection data acquisition request to the target robot through the communication address, so that the target robot feeds back at least one inspection data message based on the inspection data acquisition request.

[0043] Among them, the preset trajectory service interface can be an interface for obtaining the communication address of the target robot. Since the client can establish communication with multiple robots, it can establish communication with the target robot through the communication address of the target robot. The communication address can be the IP address of the target robot. The inspection data acquisition request can be a request for informing the target robot that its inspection data needs to be obtained. The inspection data message is a message determined by the target robot based on the inspection data during the inspection process. The inspection data message includes a first message and a second message.

[0044] Specifically, in response to the user's trigger operation on the corresponding inspection processing control, the preset trajectory service interface is called, and the communication address of the target robot is obtained through the preset trajectory service interface. A patrol data acquisition request is sent to the target robot through the communication address, so that the target robot feeds back at least one patrol data message corresponding to its patrol process according to the received patrol data acquisition request.

[0045] For example, in conjunction with the above examples, see Figure 2 , the user clicks the "Start" control in the front-end application, and the client responds to the user's trigger operation, calls the preset trajectory service interface, and notifies the trajectory service to start obtaining inspection data messages. The WebRTC playback request address of the target robot is obtained through the preset trajectory service interface, that is, the communication address of the target robot mentioned in the embodiment of the present invention. The inspection data acquisition request is sent to the target robot through the WebRTC playback request address to obtain at least one inspection data message fed back by the target robot.

[0046] Optionally, the specific method of obtaining the inspection data message of the target robot through the communication address may be: sending an inspection data acquisition request to the target robot through the communication address to obtain the robot communication profile fed back by the target robot; sending a local communication profile corresponding to the robot communication profile to the target robot, so that the target robot feeds back at least one inspection data message according to the communication interaction method corresponding to the local communication profile.

[0047] The robot communication profile can be understood as the coding information and port information used by the target robot for communication. If the target robot is a robot that communicates based on the WebRTC communication protocol, the robot communication profile can include the Session Description Protocol (SDP) information of the target robot. The local communication profile is the coding information and port information used by the client for communication. The communication interaction mode can be the way the client receives data.

[0048] Specifically, an inspection data acquisition request is sent to the target robot through a communication address, so that the target robot feeds back a robot communication configuration file according to the received inspection data acquisition request. After receiving the robot communication configuration file of the target robot, a local communication configuration file is sent to the target robot, so that the target robot feeds back at least one inspection data message to the client according to the communication interaction mode in the local communication configuration file.

[0049] For example, Figure 2 The user side in the embodiment of the present invention is the client mentioned in the embodiment of the present invention. The communication between the target robot and the client through the WebRTC communication protocol is used as an example for explanation. Correspondingly, the robot communication configuration file of the target robot is a file containing the SDP information of the target robot. The local communication configuration file is a file containing the SDP information of the client.

[0050] The WebRTC integrated service corresponding to the WebRTC communication protocol includes a signaling server and a media server. The signaling server of the WebRTC integrated service in the target robot is used to process requests from the user side, send the SDP information of the target robot to the client, and receive the SDP information of the client. Among them, the SDP information includes the port information, encoding information, data channel (DataChannel) information of the target robot, information responsible for exchanging Internet Communications Engine (InternetCommunications Engine, ICE) candidates, responsible for transmitting control information, responsible for managing the life cycle of the session, etc. The media server of the target robot is used to package and send the inspection data message according to the communication interaction mode in the client SDP information, and send the corresponding data message according to the DataChannel information. It should be noted that in an embodiment of the present invention, the data transmission mode is one-way transmission, and only the target robot sends the inspection data message to the client.

[0051] Correspondingly, the client's WebRTC integration service is similar to the target robot. The client constructs the client SDP information to make it one party of the point-to-point communication in the WebRTC communication protocol, and the target robot is the other party.

[0052] The client sends a patrol data acquisition request to the target robot through the communication address, so that the target robot that receives the patrol data acquisition request obtains the SDP information of the target robot through its signaling server and sends the SDP information of the target robot to the client. After receiving the SDP information of the target robot, the client sends its local SDP information to the target robot, so that the target robot receives the client SDP information sent by the client. After receiving the client SDP information, the target robot informs the media server of the client SDP information, so that the media server calls the data extraction interface of the data fusion module to obtain the data items in the timing queue. And the data items corresponding to the acquired patrol process are sent to the client according to the communication interaction mode corresponding to the client SDP information. That is, the video message frame is encapsulated and processed according to the communication encapsulation mode provided by the real-time transport protocol (RTP) to obtain the first message. The first message is sent to the client according to the transmission specification of the WebRTC communication protocol. The location message message and the status message message in the timing queue are stored in the DataChannel in the form of text as the second message, and the second message is sent to the client.

[0053] S120: Determine a message time difference according to a first receiving timestamp of the first message and a second receiving timestamp of the second message.

[0054] The first receiving timestamp can be used to indicate the time when the client receives the first first message. The second receiving timestamp is used to indicate the time when the client receives the first second message. The message time difference is the time difference between the first receiving timestamp and the second receiving timestamp.

[0055] Specifically, during the communication between the client and the target robot, there is a difference between the time when the client receives the first message and the time when the client receives the second message. Then, the message time difference is determined by the first receiving timestamp corresponding to the first message received by the client and the second receiving timestamp of the second message, so as to perform time alignment processing on the message data in the received first message and the second message according to the message time difference.

[0056] Exemplarily, in combination with the above example, if the first receiving timestamp corresponding to when the client receives the first message is TimeStart, and the second receiving timestamp of the second message is TimeStart+T, then the message time difference is T.

[0057] S130. Perform video reconstruction on the robot's inspection process according to the message time difference, the inspection video frame obtained by parsing the first message, and the inspection record information and robot status information obtained by parsing the second message to obtain a target inspection video.

[0058] Among them, the inspection video frame can be understood as the inspection video frame corresponding to the inspection process of the target robot carried in the first message. The inspection record information can be the record information carried by the second message, which is obtained after data preprocessing of the original inspection position record information. The robot status information can be the status information carried by the second message, which is obtained after data preprocessing of the original robot status information. The target inspection video can be the inspection video corresponding to the inspection process of the target robot obtained by video reconstruction through the inspection video frame, the inspection record information and the robot status information.

[0059] Specifically, the received first message is parsed and processed to obtain at least one inspection video frame. The received second message is parsed and processed to obtain at least one inspection record information and at least one robot status information. Through the message time difference, the first message receiving timestamp corresponding to the inspection video frame is used as a reference, and the second receiving timestamp corresponding to the inspection record information and the robot status information in the second message is adjusted to achieve time alignment of the data. The inspection record information and the robot status information corresponding to the adjusted second receiving timestamp are rendered into the inspection video frame of the corresponding timestamp, and the target inspection video is obtained through at least one rendered inspection video frame.

[0060] S140: replay the inspection process of the target robot according to the target inspection video to determine the inspection monitoring result.

[0061] Among them, the inspection monitoring result can be used to indicate whether the inspection process of the target robot is smooth. If the target robot encounters unexpected problems such as obstacles, track damage or signal failure during the inspection, the inspection monitoring result is abnormal inspection. If the inspection process of the target robot is smooth, the inspection monitoring result is normal inspection.

[0062] Specifically, the target inspection video is replayed to determine whether there is any abnormality in the inspection process of the target robot, that is, the inspection monitoring result is obtained. Based on this, it can be determined whether the target robot needs to be adjusted through the inspection monitoring result.

[0063] The technical solution of this embodiment, in response to the trigger operation of the inspection process, obtains at least one inspection data message corresponding to the target robot during the inspection process, and the inspection data message includes a first message and a second message. Based on this, the communication between the client and the target robot is realized, and the robot directly processes the collected inspection video frame, inspection position record information and robot status information into a message and sends it to the client, avoiding the problem of delayed inspection monitoring caused by the transit server, and reducing the hardware cost required for inspection monitoring. According to the first receiving timestamp of the first message and the second receiving timestamp of the second message, the message time difference is determined; according to the message time difference, the inspection video frame obtained by parsing the first message, and the inspection record information and robot status information obtained by parsing the second message, the robot's inspection process is reconstructed to obtain the target inspection video. The data in the first message and the data in the second message are time-aligned by the message time difference, ensuring the time matching of the message data and the accuracy of the target inspection video. The inspection process of the target robot is replayed according to the target inspection video to determine the inspection monitoring result. Through the inspection and monitoring results, it can be determined whether the target robot needs to be adjusted, ensuring the timely handling of emergencies during the inspection process. The present invention solves the problem of delayed inspection and monitoring caused by uploading inspection data to the server and then obtaining it by the client in the prior art. A method for direct communication between the robot and the client is realized, which improves the real-time performance of inspection and monitoring.

[0064] Embodiment 2

[0065] Figure 3 1 is a flow chart of a patrol monitoring method provided by Embodiment 2 of the present invention. This embodiment is a preferred embodiment of the above embodiment. The specific implementation method can refer to the technical solution of this embodiment. Among them, the technical terms that are the same or corresponding to the above embodiment are not repeated here. Figure 3 As shown, the method includes:

[0066] S210. In response to a trigger operation of the inspection process, obtain at least one inspection data message corresponding to the target robot during the inspection process, wherein the inspection data message includes a first message and a second message.

[0067] The first message is a message obtained after the target robot encodes the acquired inspection video frame, and the second message is a message obtained after the target robot performs data preprocessing on the acquired original inspection position record information and original robot state information.

[0068] S220: Determine a message time difference according to a first receiving timestamp of the first message and a second receiving timestamp of the second message.

[0069] S230: parse and process at least one first message to obtain at least one inspection video frame.

[0070] Specifically, for at least one first message received, each first message is parsed and processed to obtain an inspection video frame corresponding to each first message.

[0071] S240: parse and process at least one second message to determine at least one inspection location record information and at least one robot status information.

[0072] Specifically, for at least one second message received, each second message is parsed and processed to obtain the inspection record information and robot status information corresponding to each second message.

[0073] S250. Mark each inspection position record information and each robot status information in a corresponding inspection video frame according to the message time difference, to obtain a marked inspection video frame.

[0074] The marked inspection video frame may be a video frame obtained by marking the inspection position record information and the robot state information on the inspection video frame corresponding to the timestamps of the above two.

[0075] Specifically, according to the message time difference, the timestamp corresponding to each inspection record information and each robot status information is adjusted. And the inspection video frame whose timestamp is consistent with the adjusted timestamp of the inspection record information and the robot status information is determined. The inspection record information and the robot status information are marked and rendered in the corresponding inspection video frame to obtain a marked video frame.

[0076] S260: Obtain a target inspection video according to at least one marked inspection video frame.

[0077] Specifically, a target inspection video corresponding to the inspection process of the target robot is obtained through at least one marked inspection video frame, so as to perform inspection and monitoring processing on the target robot according to the target inspection video.

[0078] Optionally, after obtaining the target inspection video, the method also includes: encrypting the target inspection video to obtain an encrypted inspection video, and uploading the encrypted inspection video to a public server, so that the target terminal can obtain the target inspection video from the public server according to the video address and video key of the encrypted inspection video.

[0079] Among them, the encrypted inspection video can be a video file obtained by packaging and encrypting the target inspection video. The public server, that is, the public file server, is used to share the encrypted inspection video. The video address can be the address for obtaining the encrypted inspection video. For example, the video address can be the Uniform Resource Locator (URL) of the encrypted inspection video. The video key can be used to decrypt the encrypted inspection video. The target terminal can be understood as a client used to obtain the target inspection video. There can be one or more target terminals.

[0080] Specifically, after obtaining the target inspection video, in order to enable multiple clients to share the target inspection video, the target inspection video can be uploaded to a public server. In order to ensure the security of the target inspection video, the target inspection video can be encrypted first. The encrypted inspection video is uploaded to the public server, so that the target terminal obtains the encrypted inspection video from the public server according to the video address of the encrypted inspection video obtained, and decrypts the encrypted inspection video through the video key to obtain the target inspection video.

[0081] For example, in combination with the above example, the video address is used as URL and the video key is used as key. Figure 2 , the user clicks the "end" control in the corresponding interface of the front-end application, and the track service responds to the trigger operation and ends the acquisition of the inspection data message. The obtained target inspection video is encrypted and compressed to obtain an encrypted inspection video. The encrypted inspection video is named in the format of "first receiving timestamp + target inspection video acquisition address". The named encrypted inspection video is uploaded to the public server by calling the upload interface of the file service.

[0082] When a user application software on a client needs to perform inspection and monitoring of a target robot, it can first obtain the corresponding video file list from the file service, and obtain the video address URL of the encrypted inspection video with "first receiving timestamp + target inspection video acquisition address". At the same time, apply to the file service for the video key key of the encrypted inspection video. Send the video address URL and the video key key as video playback parameters to the track service. The track service sends the video key key as an authentication identifier to the file service. After verifying that the video key key is correct, the file service sends the target inspection video to the track service, so that the target inspection video can be replayed through the track service to determine the inspection and monitoring results.

[0083] Optionally, after obtaining the target inspection video, the inspection video frame, inspection position record information, robot status information and message time difference can be encrypted and compressed to obtain an inspection video file. The inspection video file is uploaded to a public server, so that the target terminal obtains the inspection video file from the public server according to the video address and video key of the inspection video file and processes the inspection video file to obtain the target inspection video.

[0084] Exemplarily, the uploading method and acquisition method of the inspection video file are similar to those of the target inspection video, which will not be repeated here. Among them, the inspection video file can be named with a first receiving timestamp and a second receiving timestamp. After the trajectory service receives the inspection video file, it decompresses the inspection video frame, the inspection position record information, the robot status information and the message time difference. The inspection video frame is decoded and played by creating a window. At this time, the inspection video frame, the inspection position record information and the robot status information can be time-aligned according to the first receiving timestamp and the second receiving timestamp. And based on the time-aligned inspection video frame, the inspection position record information and the robot status information, the target inspection video is reconstructed.

[0085] S270: replay the inspection process of the target robot according to the target inspection video to determine the inspection monitoring result.

[0086] Optionally, if the inspection monitoring result is an inspection abnormality, the abnormal video frame in the target inspection video and the abnormal video timestamp corresponding to the abnormal video frame are determined; based on the abnormal video frame and the abnormal video timestamp, an early warning prompt information is generated to perform maintenance on the target robot based on the early warning prompt information.

[0087] The inspection video frame may include the inspection time corresponding to the current inspection. The abnormal video frame may be a video frame in which an abnormality occurs during the inspection process. The abnormal video timestamp may be the inspection time included in the abnormal video frame. The early warning prompt information may be information used to prompt the corresponding user that an abnormality occurs during the current inspection process.

[0088] Specifically, if the inspection monitoring result is an inspection abnormality, the abnormal video frame of the inspection abnormality in the target inspection video and the abnormal video timestamp contained in the abnormal video frame are determined. Early warning prompt information is generated through the abnormal video frame and the abnormal video timestamp, and early warning processing is performed based on the early warning prompt information, so that the corresponding user can perform maintenance processing on the target robot according to the early warning prompt information. It should be noted that if the target robot is a track inspection robot, the target robot and the track can be inspected and processed through the early warning prompt information.

[0089] The technical solution of this embodiment, in response to the trigger operation of the inspection process, obtains at least one inspection data message corresponding to the target robot during the inspection process, and the inspection data message includes a first message and a second message. Based on this, the communication between the client and the target robot is realized, and the robot directly processes the collected inspection video frames, inspection position record information and robot status information into messages and sends them to the client, avoiding the problem of inspection monitoring lag caused by the transit server, and reducing the hardware cost required for inspection monitoring. According to the first receiving timestamp of the first message and the second receiving timestamp of the second message, the message time difference is determined. At least one first message is parsed and processed to obtain at least one inspection video frame, and at least one second message is parsed and processed to determine at least one inspection position record information and at least one robot status information. According to the message time difference, each inspection position record information and each robot status information are marked in the corresponding inspection video frame to obtain a marked inspection video frame. According to at least one marked inspection video frame, the target inspection video is obtained. The data in the first message and the data in the second message are time-aligned by the message time difference, ensuring the time matching of the message data and the accuracy of the target inspection video. The inspection process of the target robot is replayed based on the target inspection video to determine the inspection monitoring results. According to the inspection monitoring results, it can be determined whether the target robot needs to be adjusted, ensuring the timely handling of emergencies in the inspection process. The present invention solves the problem of inspection monitoring lag caused by uploading the inspection data to the server and then obtaining it by the client in the prior art. A method for direct communication between the robot and the client is implemented, which improves the real-time performance of inspection monitoring.

[0090] Embodiment 3

[0091] Figure 4 Schematic diagram of the structure of a patrol monitoring device provided by Embodiment 3 of the present invention. Figure 4 As shown, the device includes: a message acquisition module 310, a time difference determination module 320, an inspection video determination module 330 and a monitoring result determination module 340.

[0092] The message acquisition module 310 is used to respond to the trigger operation of the inspection processing and obtain at least one inspection data message corresponding to the target robot during the inspection process, wherein the inspection data message includes a first message and a second message, the first message is a message obtained by the target robot after encoding the obtained inspection video frame, and the second message is a message obtained by the target robot after data preprocessing the obtained original inspection position record information and original robot status information; the time difference determination module 320 is used to determine the message time difference according to the first receiving timestamp of the first message and the second receiving timestamp of the second message; the inspection video determination module 330 is used to perform video reconstruction processing on the robot's inspection process according to the message time difference, the inspection video frame obtained by parsing the first message, and the inspection record information and robot status information obtained by parsing the second message to obtain the target inspection video; the monitoring result determination module 340 is used to replay the inspection process of the target robot according to the target inspection video to determine the inspection monitoring result.

[0093] The technical solution of this embodiment, in response to the trigger operation of the inspection process, obtains at least one inspection data message corresponding to the target robot during the inspection process, and the inspection data message includes a first message and a second message. Based on this, the communication between the client and the target robot is realized, and the robot directly processes the collected inspection video frame, inspection position record information and robot status information into a message and sends it to the client, avoiding the problem of delayed inspection monitoring caused by the transit server, and reducing the hardware cost required for inspection monitoring. According to the first receiving timestamp of the first message and the second receiving timestamp of the second message, the message time difference is determined; according to the message time difference, the inspection video frame obtained by parsing the first message, and the inspection record information and robot status information obtained by parsing the second message, the robot's inspection process is reconstructed to obtain the target inspection video. The data in the first message and the data in the second message are time-aligned by the message time difference, ensuring the time matching of the message data and the accuracy of the target inspection video. The inspection process of the target robot is replayed according to the target inspection video to determine the inspection monitoring result. Through the inspection and monitoring results, it can be determined whether the target robot needs to be adjusted, ensuring the timely handling of emergencies during the inspection process. The present invention solves the problem of delayed inspection and monitoring caused by uploading inspection data to the server and then obtaining it by the client in the prior art. A method for direct communication between the robot and the client is realized, which improves the real-time performance of inspection and monitoring.

[0094] On the basis of the above embodiment, optionally, the message acquisition module includes: a communication address acquisition unit, which is used to call the preset trajectory service interface in response to the trigger operation of the inspection processing to obtain the communication address of the target robot based on the preset trajectory service interface; a message acquisition unit, which is used to send an inspection data acquisition request to the target robot through the communication address, so that the target robot feeds back at least one inspection data message based on the inspection data acquisition request.

[0095] Optionally, a message acquisition unit is used to send an inspection data acquisition request to the target robot through a communication address to obtain a robot communication profile fed back by the target robot; and to send a local communication profile corresponding to the robot communication profile to the target robot so that the target robot feeds back at least one inspection data message according to the communication interaction mode corresponding to the local communication profile.

[0096] Optionally, the inspection video determination module is used to parse and process at least one first message to obtain at least one inspection video frame; parse and process at least one second message to determine at least one inspection position record information and at least one robot status information; based on the message time difference, mark each inspection position record information and each robot status information in the corresponding inspection video frame to obtain a marked inspection video frame; based on at least one marked inspection video frame, obtain a target inspection video.

[0097] Optionally, the device also includes: a video uploading module, which is used to encrypt the target inspection video to obtain an encrypted inspection video, and upload the encrypted inspection video to a public server, so that the target terminal can obtain the target inspection video from the public server according to the video address and video key of the encrypted inspection video.

[0098] Optionally, the device also includes: an abnormality monitoring module, which is used to determine the abnormal video frame in the target inspection video and the abnormal video timestamp corresponding to the abnormal video frame if the inspection monitoring result is an inspection abnormality; based on the abnormal video frame and the abnormal video timestamp, generate early warning prompt information, so as to perform maintenance on the target robot based on the early warning prompt information.

[0099] Optionally, the target robot is a robot that communicates based on the WebRTC communication protocol.

[0100] The inspection and monitoring device provided in the embodiment of the present invention can execute the inspection and monitoring method provided in any embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method.

[0101] Embodiment 4

[0102] Figure 51 is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0103] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0104] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0105] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the inspection monitoring method.

[0106] In some embodiments, the inspection and monitoring method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the inspection and monitoring method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the inspection and monitoring method in any other appropriate manner (e.g., by means of firmware).

[0107] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0108] The computer program for implementing the inspection and monitoring method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine, or entirely on a remote machine or server.

[0109] Embodiment 5

[0110] Embodiment 5 of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute a patrol monitoring method, the method comprising:

[0111] In response to a trigger operation of the inspection processing, at least one inspection data message corresponding to the target robot during the inspection process is obtained, wherein the inspection data message includes a first message and a second message, the first message is a message obtained after the target robot encodes the obtained inspection video frame, and the second message is a message obtained after the target robot performs data preprocessing on the obtained original inspection position record information and the original robot status information; according to the first receiving timestamp of the first message and the second receiving timestamp of the second message, the message time difference is determined; according to the message time difference, the inspection video frame obtained by parsing the first message, and the inspection record information and robot status information obtained by parsing the second message, the robot's inspection process is reconstructed to obtain a target inspection video; the inspection process of the target robot is replayed according to the target inspection video to determine the inspection monitoring result.

[0112] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0113] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0114] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0115] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0116] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0117] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A patrol monitoring method, characterized in that: include: In response to a trigger operation of the inspection process, at least one inspection data message corresponding to the target robot during the inspection process is obtained, wherein the inspection data message includes a first message and a second message, the first message is a message obtained after the target robot encodes the obtained inspection video frame, and the second message is a message obtained after the target robot performs data preprocessing on the obtained original inspection position record information and original robot state information; Determine a message time difference according to a first reception timestamp of the first message and a second reception timestamp of the second message; Performing video reconstruction processing on the inspection process of the robot according to the message time difference, the inspection video frame obtained by parsing the first message, and the inspection record information and robot state information obtained by parsing the second message to obtain a target inspection video; The inspection process of the target robot is replayed according to the target inspection video to determine the inspection monitoring result.

2. The method according to claim 1, characterized in that The step of obtaining at least one inspection data message corresponding to the target robot during the inspection process in response to the triggering operation of the inspection process includes: In response to a trigger operation of the inspection process, calling a preset trajectory service interface to obtain a communication address of the target robot based on the preset trajectory service interface; A patrol data acquisition request is sent to the target robot through the communication address, so that the target robot feeds back at least one patrol data message based on the patrol data acquisition request.

3. The method according to claim 2, characterized in that The step of sending the inspection data acquisition request to the target robot through the communication address so that the target robot feeds back at least one inspection data message based on the inspection data acquisition request includes: Sending an inspection data acquisition request to the target robot through the communication address to obtain a robot communication configuration file fed back by the target robot; A local communication configuration file corresponding to the robot communication configuration file is sent to the target robot, so that the target robot feeds back at least one inspection data message according to the communication interaction mode corresponding to the local communication configuration file.

4. The method according to claim 1, characterized in that: The method of performing video reconstruction processing on the inspection process of the robot according to the message time difference, the inspection video frame obtained by parsing the first message, and the inspection record information and robot state information obtained by parsing the second message to obtain a target inspection video includes: Parsing and processing at least one of the first messages to obtain at least one of the inspection video frames; Parsing at least one of the second messages to determine at least one inspection location record information and at least one robot status information; According to the message time difference, each inspection position record information and each robot state information are marked in a corresponding inspection video frame to obtain a marked inspection video frame; A target inspection video is obtained according to at least one of the marked inspection video frames.

5. The method according to claim 1, characterized in that After obtaining the target inspection video, the method further includes: The target inspection video is encrypted to obtain an encrypted inspection video, and the encrypted inspection video is uploaded to a public server, so that the target terminal can obtain the target inspection video from the public server according to the video address and video key of the encrypted inspection video.

6. The method according to claim 1, characterized in that Also includes: If the inspection monitoring result is an inspection abnormality, determining an abnormal video frame in the target inspection video and an abnormal video timestamp corresponding to the abnormal video frame; Based on the abnormal video frame and the abnormal video timestamp, early warning prompt information is generated, so as to perform maintenance processing on the target robot based on the early warning prompt information.

7. The method according to claim 1, characterized in that The target robot is a robot that communicates based on the WebRTC communication protocol.

8. A patrol monitoring device, characterized in that: include: A message acquisition module, used for acquiring at least one inspection data message corresponding to the target robot during the inspection process in response to a trigger operation of the inspection process, wherein the inspection data message includes a first message and a second message, the first message is a message obtained after the target robot performs encoding processing on the obtained inspection video frame, and the second message is a message obtained after the target robot performs data preprocessing on the obtained original inspection position record information and original robot state information; A time difference determination module, used to determine a message time difference according to a first reception timestamp of the first message and a second reception timestamp of the second message; An inspection video determination module is used to perform video reconstruction processing on the inspection process of the robot according to the message time difference, the inspection video frame obtained by parsing the first message, and the inspection record information and robot state information obtained by parsing the second message to obtain a target inspection video; The monitoring result determination module is used to replay the inspection process of the target robot according to the target inspection video to determine the inspection monitoring result.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the inspection and monitoring method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the inspection and monitoring method according to any one of claims 1 to 7 when executed.

Images