Mobile-based robot control method, device, equipment, and storage medium

Abstract

The present application discloses a robot control method, device, equipment and storage medium based on a mobile terminal. The method includes: receiving a robot video stream shot by a robot forwarded by a cloud server, extracting the receiving frame sequence number and cloud checksum corresponding to each receiving frame in the robot video stream, and calculating the mobile terminal checksum corresponding to each receiving frame; locating the missing receiving frame based on the receiving frame sequence number, locating the damaged receiving frame based on the cloud checksum and the mobile terminal checksum, determining the invalid image frame based on the missing receiving frame and the damaged receiving frame, and removing the invalid image frame from the robot video stream; rendering and displaying the robot video stream on the mobile terminal, responding to the control operation of the robot on the mobile terminal, obtaining the control signal corresponding to the control operation, and controlling the robot based on the control signal. This solution improves the video rendering quality by verifying the integrity of the image frames in the video stream, identifying and removing the invalid image frames generated during transmission.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a mobile-based robot control method, device, equipment, and storage medium. Background Art

[0002] With the development of mobile internet and robotics, remote robot control has been widely used in fields such as inspection, monitoring, rescue, and education. In particular, remote robot control on mobile terminals has gradually become a mainstream application due to its portability and operational flexibility.

[0003] Existing mobile-based remote control systems usually rely on video streaming technology to transmit the environmental images captured by the robot to the mobile terminal in real time through the network for the user to make operational decisions. However, in actual applications, due to network bandwidth, transmission delays or packet loss, video streams are prone to image frame loss and data corruption during transmission, resulting in incomplete image display, screen tearing or delay, which seriously affects the user's control accuracy and experience. Therefore, there is an urgent need for a robot control method that can identify and eliminate invalid image frames in real time on the mobile terminal, improve video rendering quality, and ensure the smoothness and stability of remote control interaction, so as to meet the remote control needs in complex network environments. Summary of the Invention

[0004] This application provides a mobile-based robot control method, apparatus, device, and storage medium. By receiving and processing real-time video streams captured by the robot, it achieves a deep integration of remote video display and precise control. The mobile application receives the robot video stream forwarded by a cloud server and performs a frame-by-frame integrity check on each received frame in the video stream. The frame sequence number and cloud-based checksum are extracted, and combined with the calculated mobile-side checksum, missing and damaged frames are quickly identified. Frame loss is determined based on the continuity of the frame sequence numbers, and damaged frames are further identified through checksum comparison. All invalid image frames are then removed, ensuring that the final rendered video contains only complete and valid images. The filtered video stream is rendered and displayed in real time on the mobile interface with high quality, significantly improving image consistency and viewing experience. Furthermore, users can initiate control operations through the mobile interface, generating corresponding control signals in real time and transmitting them back over the network to drive the robot to perform the corresponding actions. This application integrates strategies such as video stream integrity verification, image quality assurance, and remote control interaction to significantly improve the reliability of video transmission and the stability of user operation response during robot control. It is particularly suitable for remote robot operations in complex network environments, effectively overcoming image distortion and control delay problems caused by network jitter and packet loss, and providing a safe, stable and efficient solution for scenarios such as smart inspection, security monitoring, and remote operation and maintenance.

[0005] In a first aspect, the present application provides a mobile terminal-based robot control method, comprising:

[0006] Receive a robot video stream shot by the robot forwarded by the cloud server, extract the received frame sequence number and cloud checksum corresponding to each received frame in the robot video stream, and calculate the mobile terminal checksum corresponding to each received frame;

[0007] Locating missing received frames based on the received frame sequence numbers, locating damaged received frames based on the cloud checksum and the mobile terminal checksum, determining invalid image frames based on the missing received frames and the damaged received frames, and removing the invalid image frames from the robot video stream;

[0008] The robot video stream is rendered and displayed on a mobile terminal, and in response to a control operation on the robot on the mobile terminal, a control signal corresponding to the control operation is obtained, and the robot is controlled based on the control signal.

[0009] In a second aspect, the present application provides a mobile terminal-based robot control device, comprising:

[0010] A video stream verification module is used to receive the robot video stream shot by the robot and forwarded by the cloud server, extract the received frame sequence number and cloud checksum corresponding to each received frame in the robot video stream, and calculate the mobile terminal checksum corresponding to each received frame;

[0011] a video stream purification module, configured to locate missing received frames based on the received frame sequence numbers, locate damaged received frames based on the cloud checksum and the mobile terminal checksum, determine invalid image frames based on the missing received frames and the damaged received frames, and remove the invalid image frames from the robot video stream;

[0012] The robot control module is used to render and display the robot video stream on the mobile terminal, obtain a control signal corresponding to the control operation in response to the control operation on the robot on the mobile terminal, and control the robot based on the control signal.

[0013] In a third aspect, the present application provides a mobile-based robot control device, comprising:

[0014] one or more processors;

[0015] A memory stores one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the mobile-based robot control method as described in the first aspect.

[0016] In a fourth aspect, the present application provides a storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to execute the mobile-terminal-based robot control method as described in the first aspect.

[0017] In this application, a highly reliable integrated remote video display and control strategy is implemented by receiving and processing the real-time video stream captured by the robot on a mobile terminal and combining it with a multi-dimensional verification mechanism to identify image data integrity. The mobile terminal extracts the sequence number corresponding to each received frame and the cloud checksum, calculates a local checksum, determines whether the frame data is missing based on the sequence number continuity, and determines whether the frame data is corrupted based on the consistency of the cloud and mobile checksums. Based on the above verification results, invalid image frames are identified and removed from the robot video stream, significantly improving the data validity and image integrity during the video rendering process. On this basis, the filtered video data is rendered and displayed in real time on the mobile terminal, and corresponding control signals are generated based on user interaction and transmitted to the robot terminal to drive it to perform the corresponding actions. This method effectively addresses the problems of frame loss and data corruption that are common in network transmission, improving image display quality and remote control stability. Compared with traditional robot remote control methods, this application optimizes the real-time nature of user control responses while ensuring the continuity and validity of the video image, reducing the risk of misoperation caused by image freezes or distortion. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of a mobile terminal-based robot control method provided in an embodiment of the present application;

[0019] Figure 2 This is a flowchart of video stream resolution adjustment provided by an embodiment of the present application;

[0020] Figure 3 This is a flowchart of invalid image frame retransmission provided by an embodiment of the present application;

[0021] Figure 4 This is a flowchart of mobile terminal interface rendering provided by an embodiment of the present application;

[0022] Figure 5 This is a flowchart of stopping rendering and displaying the mobile terminal interface provided by an embodiment of the present application;

[0023] Figure 6 This is a structural diagram of a mobile terminal-based robot control device provided in an embodiment of the present application;

[0024] Figure 7 This is a structural diagram of a mobile-based robot control device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] To further clarify the objectives, technical solutions, and advantages of this application, specific embodiments of this application are described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are intended only to illustrate this application and are not intended to limit it. It should also be noted that, for ease of description, the drawings only illustrate portions relevant to this application, not all of them. Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts depict the operations (or steps) as being performed sequentially, many of the operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. A process may terminate upon completion of its operations, but may also have additional steps not shown in the accompanying drawings. A process may correspond to a method, function, procedure, subroutine, subprogram, and the like.

[0026] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the data used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than those illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects connected before and after are in an "or" relationship.

[0027] With the widespread deployment of robots in scenarios such as industrial inspections, security patrols, and remote control, the demand for flexible scheduling and high-quality image transmission in multi-terminal environments is becoming increasingly prominent. Currently, a large number of systems still rely on web-based front-ends for remote control and task scheduling of robotic equipment. This reliance on traditional web-based control methods significantly limits operational convenience and on-site adaptability, especially for key operations such as viewing angle adjustment, path planning, and behavioral instruction execution. Workers often need to frequently switch between different terminals or rely on external equipment for support, making it difficult to achieve flexible control and real-time response in mobile scenarios, which seriously restricts the efficiency of system use and the convenience of personnel operations.

[0028] Furthermore, during remote video transmission, image frame data packet loss often occurs due to factors such as network fluctuations, bandwidth jitter, or channel interference. This can lead to blurred, stagnant, or even interrupted video images, hindering operators' ability to accurately assess and respond to on-site environmental conditions. This can lead to execution errors or safety hazards, especially in critical control scenarios. Existing solutions lack mechanisms for image frame integrity verification and abnormal frame filtering, making it impossible to guarantee the quality and effectiveness of video streams from the source.

[0029] Therefore, there is an urgent need to build a remote robot scheduling system that supports high-mobility control operations and has an image quality assurance mechanism. Combining mobile intelligent perception, network transmission quality identification, and video image screening and rendering capabilities, this system can enhance operational flexibility and data transmission stability in multi-scenario deployments, laying a technical foundation for intelligent, user-friendly, and reliable applications of remote robot systems. This system will effectively address the current issues of poor hardware control portability and unstable remote video perception, supporting the remote control needs of intelligent robots in multiple scenarios and tasks.

[0030] To address the aforementioned issues, this embodiment provides a mobile-based robot control method. By establishing an image integrity recognition mechanism and a mobile control command pathway, this method constructs an integrated intelligent control process with image correction, real-time rendering, and remote control capabilities. This method receives a video stream forwarded by the robot via a cloud server, extracts the received frame sequence number and cloud checksum of each image frame, and combines this with the locally calculated mobile-side checksum to perform a double check of the image frame integrity. This accurately identifies missing and damaged frames and, accordingly, removes invalid image frames with quality issues from the video stream, effectively improving the visual clarity and frame sequence consistency of the video transmission. After image processing is complete, the mobile interface renders and presents the corrected video stream in real time, providing the user with a continuous, clear remote perspective and ensuring accurate understanding of the robot's operating environment. In response to control commands executed by the user on the mobile terminal, corresponding control signals are instantly generated and fed back to the robot via a control channel, enabling remote scheduling of its motion, posture, or task execution. This method not only ensures the authenticity and timeliness of remote visual information but also significantly enhances control stability and task interaction efficiency in weak network environments. This solution boasts excellent compatibility and network adaptability, making it suitable for remote control of various mobile robots for inspection, operation, and security applications. By integrating video quality assurance mechanisms with a highly responsive control system, it provides technical support for building a stable, efficient, and low-latency human-machine interaction control architecture. It is particularly well-suited for intelligent robot control tasks in complex environments such as new energy facilities, industrial parks, and warehousing and logistics, demonstrating significant engineering application prospects and promotional value.

[0031] The mobile-based robot control method provided in this embodiment can be executed by a mobile-based robot control device. The mobile-based robot control device can be implemented through software and / or hardware. The mobile-based robot control device can be composed of two or more physical entities, or a single physical entity. For example, the mobile-based robot control device can be an operation and maintenance server used to maintain normal business operations.

[0032] The mobile-based robot control device is installed with at least one operating system, including but not limited to Android, Linux, and Windows. The mobile-based robot control device can install at least one application based on the operating system. The application can be a native application of the operating system or an application downloaded from a third-party device or server. In this embodiment, the mobile-based robot control device has at least one application capable of executing the mobile-based robot control method.

[0033] For ease of understanding, this embodiment is described by taking the operation and maintenance server as the main body for executing the mobile terminal-based robot control method as an example.

[0034] Figure 1 A flow chart of a mobile-based robot control method provided in an embodiment of the present application is given. Figure 1 , the mobile terminal-based robot control method specifically includes:

[0035] S110, receiving a robot video stream shot by the robot and forwarded by the cloud server, extracting a received frame sequence number and a cloud checksum corresponding to each received frame in the robot video stream, and calculating a mobile terminal checksum corresponding to each received frame.

[0036] In some embodiments, a robot video stream captured by an inspection robot and forwarded by a cloud server is first received. The robot video stream represents continuous image frame data captured by the inspection robot in a specified environment. The cloud server can be a media forwarding node deployed on an edge computing platform or a remote server. After receiving the robot video stream, the corresponding received frame sequence number and cloud checksum are extracted for each received frame in the video stream. The received frame sequence number identifies the order of frames in the robot video stream, and the cloud checksum is an integrity check code calculated by the cloud server based on the content of each frame and used for subsequent data consistency verification.

[0037] In one embodiment, the mobile-end checksum may be calculated by generating a corresponding mobile-end checksum based on the pixel data content of each received frame through a preset hash function, and the mobile-end checksum is used to verify at the receiving end whether the received video frame data is erroneous or damaged during transmission.

[0038] By extracting the received frame sequence number, cloud checksum, and calculating the mobile checksum, basic data support can be provided for subsequent data consistency comparison, frame loss detection, and retransmission control, thereby improving the reliability and stability of the robot's remote video inspection process.

[0039] S120. Locate missing received frames based on the received frame sequence numbers, locate damaged received frames based on the cloud-side checksum and the mobile-end checksum, determine invalid image frames based on the missing received frames and the damaged received frames, and remove the invalid image frames from the robot video stream.

[0040] In some embodiments, the missing received frames are first located based on the received frame sequence number, wherein the received frame sequence number is used to identify the order of video image frames received by the mobile terminal, and the missing received frames refer to image frames that are not received in the sequence number continuity check; and the received frames refer to each frame of image data received by the mobile terminal during the robot video stream transmission process. After completing the missing frame location, the damaged received frames are further located based on the comparison of the cloud checksum and the mobile terminal checksum, wherein the cloud checksum is the image frame integrity check value calculated by the robot terminal, and the mobile terminal checksum is the image frame check value calculated after reception, and the damaged received frames refer to frame data with inconsistent checksums. Based on the set of missing received frames and damaged received frames, invalid image frames are determined. Invalid image frames refer to image frames that affect the accuracy of subsequent image processing due to loss or damage. These invalid image frames are then removed from the robot video stream to improve the validity of the video data.

[0041] In one embodiment, the missing received frame may be located by traversing the received frame sequence and detecting whether there is a non-continuous gap between sequence numbers. If so, the corresponding position is the missing frame.

[0042] In one embodiment, the damaged received frame may be located by calculating a checksum of the original image frame on the cloud and the image frame received on the mobile terminal, comparing the checksums to see if they are consistent, and determining the frame as damaged if they are inconsistent.

[0043] In one embodiment, the invalid image frames may be determined by combining the sequence numbers of missing received frames and damaged received frames to form an invalid frame set, which serves as a basis for subsequent image removal.

[0044] Optionally, Figure 2 A flow chart of video stream resolution adjustment provided by an embodiment of the present application is given. Figure 2 , the video stream resolution adjustment method specifically includes:

[0045] S121. Calculate an abnormal transmission ratio between the mobile terminal and the cloud server based on the invalid image frame and the robot video stream.

[0046] Exemplarily, the abnormal transmission ratio between the mobile terminal and the cloud server is first calculated based on invalid image frames and robot video streams, where invalid image frames refer to unusable image frames due to loss or damage, and robot video streams refer to continuous image data streams collected and transmitted by the robot. The abnormal transmission ratio refers to the ratio of the number of invalid image frames to the total number of transmission frames, which is used to reflect data anomalies during video transmission.

[0047] In one embodiment, the abnormal transmission ratio is calculated based on invalid image frames and robot video streams by counting the number of invalid image frames, counting the total number of frames in the robot video stream, and dividing the number of invalid frames by the total number of frames to obtain the abnormal transmission ratio.

[0048] S122. Adjust the resolution of the robot video stream forwarded by the cloud server based on the abnormal transmission ratio.

[0049] Exemplarily, the resolution of the robot video stream forwarded by the cloud server is adjusted based on the abnormal transmission ratio, where the abnormal transmission ratio refers to the ratio of invalid image frames to total transmission frames, which is used to measure the quality of video data transmission; the cloud server refers to the remote computing device responsible for receiving, processing and forwarding the robot video stream; the robot video stream resolution refers to the pixel size of the image frame in the video stream, which directly affects the video clarity and transmission bandwidth.

[0050] In one embodiment, the robot video stream resolution may be adjusted by dynamically reducing or increasing the video stream resolution according to a threshold range of an abnormal transmission ratio to balance image quality and transmission stability.

[0051] In one embodiment, the threshold range for abnormal transmission ratios can be divided into three levels: when the abnormal transmission ratio is less than 5%, the network transmission quality is considered good, and the cloud server maintains or increases the resolution of the robot video stream to 1920×1080 pixels to ensure image detail; when the abnormal transmission ratio is between 5% and 15%, the transmission quality is considered medium, and the cloud server adjusts the video stream resolution to 1280×720 pixels to reduce bandwidth pressure; when the abnormal transmission ratio is greater than 15%, the transmission quality is considered poor, and the cloud server reduces the video stream resolution to 640×360 pixels to prioritize video continuity and real-time performance. This tiered transmission strategy can achieve a dynamic balance between image quality and transmission stability under different transmission conditions.

[0052] Optionally, Figure 3 A flowchart of invalid image frame retransmission according to an embodiment of the present application is given. Figure 3 , the invalid image frame retransmission method specifically includes:

[0053] S123. Obtain a timestamp of the invalid image frame, generate an image frame retransmission request based on the timestamp, and send the image frame retransmission request to the cloud server.

[0054] Exemplarily, the system first obtains the timestamp of invalid image frames. A timestamp refers to the precise time at which each image frame was captured. Invalid image frames are those that are unusable due to loss or damage. Based on these timestamps, an image frame retransmission request is generated. This request requests the cloud server to resend the data packet corresponding to the image frame at a specified time. The generated image frame retransmission request is then sent to the cloud server to compensate for the invalid image frames.

[0055] In one embodiment, the method of obtaining the invalid image frame timestamp may be: reading the timestamp corresponding to the invalid image frame from the receiving buffer.

[0056] In one embodiment, the image frame retransmission request may be generated by constructing a retransmission request message according to a timestamp, including time information of the image frame to be retransmitted.

[0057] In one embodiment, the image frame retransmission request may be sent by transmitting the request message to the cloud server via a network communication module of the mobile terminal.

[0058] S124: Receive the retransmitted image frame corresponding to the image frame retransmission request sent by the cloud server, and correct the robot video stream based on the retransmitted image frame.

[0059] Exemplarily, the system first receives retransmitted image frames corresponding to an image frame retransmission request sent by a cloud server. Retransmitted image frames refer to missing or damaged image frame data resent by the cloud server in response to the retransmission request. Based on the retransmitted image frames, the robot video stream is corrected. Correction involves replacing or supplementing invalid frames in the original video stream with the retransmitted image frames to restore the integrity and continuity of the video stream.

[0060] In one embodiment, the retransmitted image frame may be received by receiving the image frame data packet transmitted by the cloud server through a network communication module of the mobile terminal.

[0061] In one embodiment, the robot video stream may be corrected based on the retransmitted image frame by inserting or replacing the retransmitted image frame at the corresponding invalid frame position and updating the robot video stream cache.

[0062] S130: Render and display the robot video stream on the mobile terminal, obtain a control signal corresponding to the control operation in response to a control operation on the robot on the mobile terminal, and control the robot based on the control signal.

[0063] In some embodiments, a robot video stream is first rendered and displayed on a mobile terminal. The robot video stream refers to a real-time image data stream captured by the robot and transmitted to the mobile terminal via a network. The mobile terminal refers to a smart device carried by the user, such as a mobile phone or tablet. While the video stream is being displayed on the mobile terminal, the robot is responded to by the mobile terminal's control operations on the robot. A control operation refers to a command input initiated by the user through the mobile terminal interface, and robot control refers to the robot performing corresponding actions based on the received command. Subsequently, a control signal corresponding to the control operation is obtained. A control signal refers to a digital command generated by the mobile terminal and transmitted to the robot to instruct a specific action. Based on this control signal, the robot is controlled to achieve real-time adjustment and feedback of the robot's actions.

[0064] In one embodiment, the robot video stream may be rendered and displayed by decoding the received video data stream and rendering the video frame by frame to the mobile terminal display screen.

[0065] In one embodiment, the method of responding to the mobile terminal's control operation on the robot may be: monitoring interaction events on the user interface and capturing control instruction input.

[0066] In one embodiment, the robot may be controlled based on a control signal by sending the control signal to the robot through a communication module, and the robot may receive the signal and analyze it and drive corresponding mechanical components to perform actions.

[0067] Optionally, Figure 4 A flowchart of the mobile terminal interface rendering provided by the embodiment of this application is given. Figure 4 , the mobile terminal interface rendering method specifically includes:

[0068] S1301. Obtain user information of a mobile terminal, send the user information to a cloud server, and receive permission information corresponding to the user information sent by the cloud server.

[0069] Exemplarily, the process first acquires user information from the mobile device. User information refers to a collection of data identifying the user's identity and related attributes. This user information is then sent to a cloud server, a remote computing device responsible for user information management and permission allocation. Next, the process receives permission information corresponding to the user information from the cloud server. This permission information refers to the operational permissions and access scopes determined based on the user's identity.

[0070] In one embodiment, the method of obtaining the mobile terminal user information may be: extracting the login identity and related data from the user account management module of the mobile terminal.

[0071] In one embodiment, the user information may be sent to the cloud server by uploading the user information via a mobile network communication module using an encrypted transmission protocol.

[0072] In one embodiment, the permission information may be received by monitoring the permission data packet returned by the cloud server and parsing and obtaining the specific permission content.

[0073] S1302: When the permission information includes a control permission, generate a corresponding control panel based on the control permission, and render and display the video stream and the control panel on the mobile terminal.

[0074] Exemplarily, when the permission information includes control permissions, a corresponding control panel is generated based on the control permissions. Control permissions refer to the permission that allows the user to operate the robot, and the control panel refers to the control interface displayed on the mobile device and available for user interaction. Subsequently, the robot's video stream and control panel are rendered and displayed on the mobile device. The video stream refers to the video data collected and transmitted by the robot in real time, and rendering refers to drawing the video frames and control panel interface elements onto the mobile device's display.

[0075] In one embodiment, the control panel may be generated by dynamically configuring corresponding interactive components such as buttons and sliders according to the specific content of the control authority.

[0076] In one embodiment, a method of rendering and displaying the video stream and the control panel on the mobile terminal may be: synthesizing the image after video decoding and the control interface and displaying them on the mobile terminal screen.

[0077] S1303: When the permission information does not include control permission, render and display the video stream on the mobile terminal.

[0078] For example, when the permission information does not include control permissions, only the robot video stream is rendered and displayed on the mobile device. Permission information refers to the operational permissions determined based on the user's identity, and control permissions refer to the permissions that allow the user to operate the robot. The robot video stream refers to the real-time image data stream collected and transmitted by the robot. Rendering refers to decoding the video stream and displaying it on the mobile device screen for the user to view.

[0079] In one embodiment, the robot video stream may be rendered and displayed by decoding the received video data and displaying the image frames frame by frame on the display interface of the mobile device.

[0080] Optionally, Figure 5 A flowchart of stopping rendering and displaying the mobile terminal interface provided by the embodiment of the present application is given. Figure 5 The method for stopping rendering and displaying the mobile interface specifically includes:

[0081] S131. Control the mobile terminal to capture an environmental video stream of the environment in which the mobile terminal is located, and determine an environmental distance score between the mobile terminal and the robot based on the environmental video stream.

[0082] Exemplarily, a mobile terminal is first controlled to capture an environmental video stream of the mobile terminal's environment. The environmental video stream is a continuous sequence of image data representing the mobile terminal's current environment. The mobile terminal can be a smartphone, tablet, or other mobile device with a camera. After acquiring the environmental video stream, the relative positional relationship between the mobile terminal and the robot is analyzed based on the video stream to further determine an environmental distance score between the mobile terminal and the robot. The environmental distance score is a quantitative indicator that represents the spatial distance between the mobile terminal and the robot and any obstruction by obstacles.

[0083] In one embodiment, the environmental distance score may be determined by extracting the spatial position information of the robot and the mobile terminal based on image features in the environmental video stream, and obtaining the environmental distance score by calculating the spatial distance between the two and the distribution of environmental obstacles.

[0084] In one embodiment, the environmental distance score may be determined by performing depth estimation and scene reconstruction on the environmental video stream, inferring the three-dimensional distance between the mobile terminal and the robot, and adjusting the distance score in combination with the environmental complexity.

[0085] In one embodiment, the environmental distance score can be determined by using a machine learning model to input a sequence of environmental video stream frames and output an environmental distance score between the mobile terminal and the robot. The model comprehensively evaluates distance and occlusion based on a large amount of training data.

[0086] Optionally, determining the environmental distance score between the mobile terminal and the robot based on the environmental video stream includes:

[0087] The environmental similarity between the mobile terminal and the environment of the robot is determined based on the environmental video stream and the robot video stream, and the environmental distance score between the mobile terminal and the robot is determined based on the environmental similarity.

[0088] Exemplarily, the environmental similarity between the mobile device and the robot's environment is determined based on the environment video stream and the robot video stream. The environmental similarity is an indicator that characterizes the consistency of visual features of the environments on both ends, reflecting the degree of similarity between the mobile device and the robot's environment. Based on the environmental similarity, an environmental distance score is determined between the mobile device and the robot. The environmental distance score reflects the proximity of the mobile device and the robot in the environmental space. Higher environmental similarity indicates a lower environmental distance score, indicating closer environments.

[0089] In one embodiment, the environment similarity may be determined by comparing key frame image features in the environment video stream and the robot video stream, and calculating feature matching to obtain the environment similarity.

[0090] In one embodiment, the environmental distance score may be determined by inputting the environmental similarity into a preset mapping function and converting it into an environmental distance score to quantify the spatial relationship and environmental difference between the two.

[0091] S132: When the environment distance score is less than a distance score threshold, stop rendering and displaying the robot video stream on the mobile terminal.

[0092] For example, when the environmental distance score falls below a distance score threshold, rendering and displaying the robot video stream on the mobile device is stopped. The environmental distance score is a distance evaluation metric between the mobile device and the robot, calculated based on the environmental video stream captured by the mobile device, reflecting the spatial proximity between the two. The distance score threshold is a preset safety or signal validity threshold. When the environmental distance score falls below this threshold, it indicates that the distance is too close or the signal is unstable, and the video stream display needs to be paused to avoid abnormal display or safety hazards.

[0093] In one embodiment, the environmental distance score can be calculated by extracting the relative spatial distance between the robot target and the mobile terminal based on the environmental video captured by the mobile terminal camera, combining image recognition and depth estimation algorithms, and converting it into a standardized score.

[0094] In one embodiment, the method of stopping video stream rendering can be: after the mobile terminal receives a signal that the environment distance score is lower than a threshold, it calls the stop interface of the video player or hides the video screen, and pops up a prompt message to notify the user of the current distance limit.

[0095] In one embodiment, the environmental distance score can be calculated in the range of 0 to 100, where the lower the score, the closer the distance between the robot and the mobile terminal. When the environmental distance score is lower than 30, it means that the straight-line distance between the two is less than 0.5 meters, and the user can directly observe the status of the robot with the naked eye without obtaining image information through the video stream. At this time, the mobile terminal immediately stops rendering and displaying the robot video stream to save wireless transmission bandwidth and terminal decoding resources, and displays a prompt on the screen that says "The distance is too close, the video has been paused." For example, when a robot approaches the user to 0.4 meters, the environmental distance score calculated by the depth estimation algorithm is 25, which is lower than the distance score threshold of 30. The mobile terminal triggers the video rendering to stop and releases the decoding buffer, thereby avoiding meaningless video transmission and processing overhead.

[0096] This mechanism can effectively ensure user experience and operational safety, and prevent video quality degradation or visual misleading caused by excessively close distances.

[0097] Optionally, the mobile terminal positioning of the mobile terminal is obtained, the robot positioning sent by the cloud server is received, and the positioning distance score between the mobile terminal and the robot is calculated based on the mobile terminal positioning and the robot positioning.

[0098] Exemplarily, the mobile terminal's positioning information is first obtained. The mobile terminal positioning refers to coordinate data representing the specific location of the mobile terminal in the environment, which may include GPS coordinates, indoor positioning data, or other positioning methods. The robot positioning information sent by the cloud server is received. The robot positioning refers to coordinate data representing the specific location of the robot in the current environment, which has the same or compatible coordinate system as the mobile terminal positioning. A positioning distance score is calculated based on the mobile terminal positioning and the robot positioning. The positioning distance score is a quantitative indicator representing the spatial distance between the two. The smaller the value, the closer the distance.

[0099] In one embodiment, the positioning distance score may be calculated by using the Euclidean distance formula or other distance calculation methods, combining the coordinate information of the two to derive the positioning distance score, which is used to evaluate the spatial distance between the mobile terminal and the robot.

[0100] In one embodiment, the positioning distance score is calculated on a scale of 0 to 100, with lower scores indicating closer proximity between the mobile device and the robot. For example, in an indoor positioning system, when the positioning distance score is below 30, corresponding to an actual straight-line distance of approximately 0.5 meters, the user is deemed to be interacting closely with the robot, allowing direct observation without the need for video streaming. At this point, the mobile device stops rendering the robot's video stream and displays a message on the screen stating "Video Paused, Too Close." When the positioning distance score is between 30 and 60, corresponding to distances of 0.5–1.0 meters, the mobile device maintains the video stream but automatically reduces resolution to conserve bandwidth. When the positioning distance score is above 60, meaning the distance is greater than 1.0 meters, the video resolution is maintained or increased to ensure clarity at long distances. For example, in an indoor inspection scenario, the mobile device obtains an actual distance of 0.35 meters from the robot via Bluetooth ranging. The mapped positioning distance score is 20, below the distance score threshold of 30. The mobile device immediately stops video rendering, reducing unnecessary data transmission and processing overhead.

[0101] Optionally, a comprehensive distance score between the mobile terminal and the robot is calculated based on the environment distance score and the positioning distance score. When the comprehensive distance score is less than a distance score threshold, rendering and displaying the robot video stream on the mobile terminal is stopped.

[0102] Exemplarily, a comprehensive distance score between the mobile terminal and the robot is calculated based on the environmental distance score and the positioning distance score. The comprehensive distance score is a distance evaluation metric that integrates environmental perception and positioning information to more accurately reflect the actual spatial relationship between the mobile terminal and the robot. A distance score threshold is set as a criterion for determining whether the mobile terminal and the robot are sufficiently close. When the comprehensive distance score is less than the distance score threshold, rendering and displaying the robot video stream on the mobile terminal is stopped to avoid redundant video display or conserve mobile terminal resources.

[0103] In one embodiment, the comprehensive distance score may be calculated by performing weighted averaging or other fusion algorithms on the environment distance score and the positioning distance score according to preset weights to obtain the comprehensive distance score.

[0104] In one embodiment, the comprehensive distance score is calculated in the range of 0 to 100, and is obtained by weighted fusion of the environmental distance score and the positioning distance score, where the weight of the environmental distance score is 0.6 and the weight of the positioning distance score is 0.4. When the comprehensive distance score is lower than 30, corresponding to a straight-line distance of approximately 0.5 meters, it is determined that the mobile terminal and the robot are in an extremely close interaction scene, and the user can directly observe the robot with the naked eye. At this time, the mobile terminal immediately stops rendering and displaying the video stream and releases decoding resources. For example, in an indoor inspection scenario, the environmental video analysis result obtains an environmental distance score of 28, corresponding to approximately 0.45 meters, and the indoor positioning data calculates a positioning distance score of 20, corresponding to approximately 0.35 meters. The weighted calculation results in a comprehensive distance score of:

[0105]

[0106] The calculated comprehensive distance score of 24.8 is lower than the distance score threshold of 30. The mobile terminal triggers a video rendering stop instruction and prompts "Distance is too close, video has been paused" on the mobile terminal, thereby avoiding repeated information display and saving wireless transmission bandwidth and terminal processing capabilities.

[0107] Optionally, after controlling the robot based on the control signal, the method further includes:

[0108] A heartbeat signal is periodically sent to the cloud server, and in response to not receiving a response signal corresponding to the heartbeat signal sent by the cloud server within a preset time window, a reconnection request is sent to the cloud server.

[0109] Exemplarily, after controlling the robot based on the control signal, the process further includes periodically sending a heartbeat signal to a cloud server. A heartbeat signal refers to a detection message periodically sent by a mobile terminal to maintain a connection, and a cloud server refers to a remote computing device responsible for managing communications between the robot and the mobile terminal. In response to not receiving a response signal corresponding to the heartbeat signal sent by the cloud server within a preset time window, a reconnection request is sent to the cloud server. The reconnection request refers to a request message initiated by the mobile terminal to reestablish a communication connection.

[0110] In one embodiment, the manner of periodically sending the heartbeat signal may be: setting a fixed time interval and automatically sending the heartbeat message through the network communication module.

[0111] In one embodiment, the heartbeat response signal may be detected by monitoring the response message returned by the cloud server within a preset time, and determining that the response message has not been received if the timeout expires.

[0112] In one embodiment, the reconnection request may be sent by sending a reconnection request message to the cloud server via the mobile terminal communication module to try to re-establish the communication connection.

[0113] Based on the above embodiments, Figure 6 This is a schematic diagram of the structure of the mobile terminal-based robot control device provided in the embodiment of the present application. Figure 6 The mobile terminal-based robot control device provided in this embodiment specifically includes: a video stream verification module 21, a video stream purification module 22, and a robot control module 23.

[0114] Among them, the video stream verification module 21 is configured to receive the robot video stream shot by the robot forwarded by the cloud server, extract the receiving frame sequence number and cloud checksum corresponding to each receiving frame in the robot video stream, and calculate the mobile terminal checksum corresponding to each receiving frame; the video stream purification module 22 is configured to locate the missing receiving frame based on the receiving frame sequence number, locate the damaged receiving frame based on the cloud checksum and the mobile terminal checksum, determine the invalid image frame according to the missing receiving frame and the damaged receiving frame, and remove the invalid image frame from the robot video stream; the robot control module 23 is configured to render and display the robot video stream on the mobile terminal, respond to the control operation of the robot on the mobile terminal, obtain the control signal corresponding to the control operation, and control the robot based on the control signal.

[0115] Based on the above embodiment, the mobile terminal-based robot control device also includes: an environmental distance module, configured to control the mobile terminal to shoot the environmental video stream of the environment in which the mobile terminal is located, and determine the environmental distance score between the mobile terminal and the robot based on the environmental video stream; a termination rendering module, configured to stop rendering and displaying the robot video stream on the mobile terminal when the environmental distance score is less than a distance score threshold.

[0116] Based on the above embodiment, the environmental distance module includes: an environmental distance unit, which is configured to determine the environmental similarity between the mobile terminal and the environment in which the robot is located based on the environmental video stream and the robot video stream, and determine the environmental distance score between the mobile terminal and the robot based on the environmental similarity.

[0117] Based on the above embodiment, the mobile-based robot control device also includes: a retransmission request module, configured to obtain the timestamp of the invalid image frame, generate an image frame retransmission request based on the timestamp, and send the image frame retransmission request to the cloud server; a video stream correction module, configured to receive the retransmitted image frame corresponding to the image frame retransmission request sent by the cloud server, and correct the robot video stream based on the retransmitted image frame.

[0118] Based on the above embodiment, the mobile-based robot control device also includes: a periodic heartbeat module, which is configured to periodically send a heartbeat signal to the cloud server, and in response to not receiving a response signal corresponding to the heartbeat signal sent by the cloud server within a preset time window, sends a reconnection request to the cloud server.

[0119] Based on the above embodiment, the mobile-based robot control device also includes: an abnormal transmission module, configured to calculate the abnormal transmission ratio between the mobile terminal and the cloud server based on the invalid image frame and the robot video stream; a resolution adjustment module, configured to adjust the resolution of the robot video stream forwarded by the cloud server based on the abnormal transmission ratio.

[0120] Based on the above embodiment, the robot control module 23 includes: an authority authentication unit, configured to obtain user information of the mobile terminal, send the user information to the cloud server, and receive authority information corresponding to the user information sent by the cloud server; a control authority unit, configured to generate a corresponding control panel based on the control authority when the authority information includes control authority, and render and display the video stream and control panel on the mobile terminal; a video authority unit, configured to render and display the video stream on the mobile terminal when the authority information does not include control authority.

[0121] As described above, the mobile-based robot control device provided in the embodiments of this application integrates key modules such as video stream integrity verification, image frame purification processing, video rendering and display, and interactive control to build an integrated video perception and control architecture centered on image data quality assurance, real-time video monitoring, and remote control execution. The system, composed of multiple functional units, implements a complete processing chain from image reception and verification, abnormal frame removal, to video rendering and display, and control signal generation, significantly improving the video transmission reliability and operational response efficiency during remote robot operation and maintenance.

[0122] The mobile-based robot control device provided in the embodiment of the present application can be used to execute the mobile-based robot control method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0123] Figure 7 This is a schematic diagram of a mobile-based robot control device provided in an embodiment of the present application. Figure 7 The mobile-based robot control device includes: a processor 31, a memory 32, a communication device 33, an input device 34, and an output device 35. The number of processors 31 in the mobile-based robot control device can be one or more, and the number of memories 32 in the mobile-based robot control device can be one or more. The processor 31, memory 32, communication device 33, input device 34, and output device 35 of the mobile-based robot control device can be connected via a bus or other means.

[0124] Memory 32, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the mobile-based robot control method of any embodiment of the present application (e.g., the video stream verification module 21, the video stream purification module 22, and the robot control module 23 in the mobile-based robot control device). Memory 32 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data generated based on device usage. Furthermore, memory 32 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some instances, the memory may further include memory located remotely from the processor, which can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0125] The communication device 33 is used for data transmission.

[0126] The processor 31 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 32, that is, realizes the above-mentioned mobile-based robot control method.

[0127] The input device 34 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 35 may include a display device such as a display screen.

[0128] The mobile terminal-based robot control device provided above can be used to execute the mobile terminal-based robot control method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0129] An embodiment of the present application also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to execute a mobile-based robot control method, the mobile-based robot control method comprising: receiving a robot video stream shot by a robot and forwarded by a cloud server, extracting a received frame sequence number and a cloud checksum corresponding to each received frame in the robot video stream, and calculating a mobile-side checksum corresponding to each received frame; locating missing received frames based on the received frame sequence number, locating damaged received frames based on the cloud checksum and the mobile-side checksum, determining invalid image frames based on the missing received frames and the damaged received frames, and removing the invalid image frames from the robot video stream; rendering and displaying the robot video stream on a mobile terminal, obtaining a control signal corresponding to the control operation in response to a control operation on the robot on the mobile terminal, and controlling the robot based on the control signal.

[0130] Storage medium—any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROMs, floppy disks, or tape drives; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disks or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or a combination thereof. Furthermore, a storage medium may be located in the first computer system where the program is executed, or in a different second computer system connected to the first computer system via a network (such as the Internet). The second computer system may provide program instructions to the first computer system for execution. The term "storage medium" may include two or more storage media residing in different locations (e.g., in different computer systems connected via a network). A storage medium may store program instructions (e.g., embodied as a computer program) that are executable by one or more processors.

[0131] Of course, the storage medium containing computer-executable instructions provided in an embodiment of the present application, whose computer-executable instructions are not limited to the above-mentioned mobile-based robot control method, can also execute related operations in the mobile-based robot control method provided in any embodiment of the present application.

[0132] The mobile-based robot control device, storage medium and mobile-based robot control equipment provided in the above embodiments can execute the mobile-based robot control method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the mobile-based robot control method provided in any embodiment of the present application.

[0133] The above are only preferred embodiments of the present application and the technical principles employed. The present application is not limited to the specific embodiments described herein, and any obvious changes, readjustments, and substitutions that are apparent to those skilled in the art will not depart from the scope of protection of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present application. The scope of the present application is determined by the scope of the claims.

Claims

1. A robot control method based on a mobile terminal, characterized in that: include: Receive a robot video stream shot by the robot forwarded by the cloud server, extract the received frame sequence number and cloud checksum corresponding to each received frame in the robot video stream, and calculate the mobile terminal checksum corresponding to each received frame; Locating missing received frames based on the received frame sequence numbers, locating damaged received frames based on the cloud checksum and the mobile terminal checksum, determining invalid image frames based on the missing received frames and the damaged received frames, and removing the invalid image frames from the robot video stream; The robot video stream is rendered and displayed on a mobile terminal, and in response to a control operation on the robot on the mobile terminal, a control signal corresponding to the control operation is obtained, and the robot is controlled based on the control signal.

2. The mobile terminal-based robot control method according to claim 1, characterized in that: After rendering and displaying the robot video stream on the mobile terminal, the method further includes: Controlling the mobile terminal to capture an environmental video stream of an environment in which the mobile terminal is located, and determining an environmental distance score between the mobile terminal and the robot based on the environmental video stream; When the environment distance score is less than a distance score threshold, rendering and displaying the robot video stream on the mobile terminal is stopped.

3. The mobile terminal-based robot control method according to claim 2, characterized in that: The determining of the environmental distance score between the mobile terminal and the robot based on the environmental video stream includes: The environmental similarity between the mobile terminal and the environment of the robot is determined based on the environmental video stream and the robot video stream, and the environmental distance score between the mobile terminal and the robot is determined based on the environmental similarity.

4. The mobile terminal-based robot control method according to claim 1, characterized in that: Before rendering and displaying the robot video stream on the mobile terminal, the method further includes: Obtaining a timestamp of the invalid image frame, generating an image frame retransmission request based on the timestamp, and sending the image frame retransmission request to the cloud server; Receive a retransmitted image frame corresponding to the image frame retransmission request sent by the cloud server, and correct the robot video stream based on the retransmitted image frame.

5. The mobile terminal-based robot control method according to claim 1, characterized in that: After controlling the robot based on the control signal, the method further includes: A heartbeat signal is periodically sent to the cloud server, and in response to not receiving a response signal corresponding to the heartbeat signal sent by the cloud server within a preset time window, a reconnection request is sent to the cloud server.

6. The mobile terminal-based robot control method according to claim 1, characterized in that: After removing the invalid image frame from the robot video stream, the method further includes: Calculating an abnormal transmission ratio between the mobile terminal and the cloud server based on the invalid image frame and the robot video stream; The resolution of the robot video stream forwarded by the cloud server is adjusted based on the abnormal transmission ratio.

7. The mobile terminal-based robot control method according to claim 1, characterized in that: Rendering and displaying the robot video stream on the mobile terminal includes: Obtain user information of the mobile terminal, send the user information to the cloud server, and receive permission information corresponding to the user information sent by the cloud server; In a case where the permission information includes a control permission, generating a corresponding control panel based on the control permission, and rendering and displaying the video stream and the control panel on the mobile terminal; When the permission information does not include the control permission, the video stream is rendered and displayed on the mobile terminal.

8. A robot control device based on a mobile terminal, characterized in that: include: A video stream verification module is used to receive the robot video stream shot by the robot and forwarded by the cloud server, extract the received frame sequence number and cloud checksum corresponding to each received frame in the robot video stream, and calculate the mobile terminal checksum corresponding to each received frame; a video stream purification module, configured to locate missing received frames based on the received frame sequence numbers, locate damaged received frames based on the cloud checksum and the mobile terminal checksum, determine invalid image frames based on the missing received frames and the damaged received frames, and remove the invalid image frames from the robot video stream; The robot control module is used to render and display the robot video stream on the mobile terminal, obtain a control signal corresponding to the control operation in response to the control operation on the robot on the mobile terminal, and control the robot based on the control signal.

9. A mobile-based robot control device, characterized in that: include: one or more processors; A memory storing one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the mobile terminal-based robot control method as described in any one of claims 1-7.

10. A storage medium containing computer-executable instructions, characterized in that: When executed by a computer processor, the computer executable instructions are used to execute the mobile terminal-based robot control method as described in any one of claims 1 to 7.

Images