Control method of self-moving equipment and related equipment

Abstract

The embodiment of the invention provides a control method of self-moving equipment and related equipment, and the method comprises the steps: collecting video stream data, packaging the video stream data into a real-time transport protocol (RTP) data packet based on a user datagram protocol (UDP), coding and transmitting the RTP data packet to a server through a real-time communication (RTC) channel of the server, and transmitting the RTP data packet to the server through the RTC channel of the server. Transmitting the RTP data packet to the electronic equipment through the server, so that the electronic equipment displays the video stream data; receiving a control instruction of the electronic equipment forwarded by the server, wherein the control instruction is generated based on the video stream data; and in response to the control instruction, executing a corresponding operation. According to the embodiment of the invention, the video data and the control instruction between the self-moving equipment and the electronic equipment are forwarded through the server, so that when the electronic equipment is used for remotely controlling the self-moving equipment, the data transmission time delay is reduced, and the remote control efficiency is effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of self-moving equipment, and in particular to a control method for a self-moving equipment and related equipment. Background Art

[0002] With the continuous advancement of science and technology and the rapid development of artificial intelligence, the use of autonomous devices (such as mowing robots, cleaning robots, food delivery robots, etc.) to replace manual work can greatly improve work efficiency.

[0003] Remote monitoring, route adjustment, and equipment failure scenarios require users to remotely control their mobile devices. Remote control of mobile devices is typically achieved using wireless communication technologies such as 5G+WebSocket / MQTT (Message Queuing Telemetry Transport) protocols, RTC (Real-Time Communications) services, and UDP (User Datagram Protocol). However, these wireless communication technologies suffer from high latency and unstable networks, resulting in low communication reliability and impacting remote control of mobile devices. Summary of the Invention

[0004] In view of this, the embodiments of the present application provide a control method and related equipment for a self-mobile device to solve the problem that wireless communication technology has high communication delay, weak network instability, resulting in low communication reliability and affecting the remote control effect of the self-mobile device.

[0005] In a first aspect, an embodiment of the present application provides a control method for a self-mobile device, which is applied to the self-mobile device, and the method includes: collecting video stream data, encapsulating the video stream data into a real-time transport protocol RTP data packet based on the user datagram protocol UDP, encoding and transmitting the RTP data packet to the server through the real-time communication RTC channel of the server, and transmitting the RTP data packet to the electronic device through the server so that the electronic device displays the video stream data; receiving a control instruction of the electronic device forwarded by the server, the control instruction being generated based on the video stream data; and performing a corresponding operation in response to the control instruction.

[0006] In a possible implementation, executing a corresponding operation in response to the control instruction includes: parsing the control instruction and performing an integrity check, and if the control instruction passes the integrity check, executing the operation corresponding to the control instruction.

[0007] In a second aspect, an embodiment of the present application provides a control method for a self-mobile device, which is applied to a server, and the method includes: receiving video stream data transmitted from a mobile device; determining a first transmission node based on first location information of the electronic device, and transmitting the video stream data to the electronic device through the first transmission node; receiving a control instruction of the electronic device, and forwarding the control instruction to the self-mobile device, so that the self-mobile device performs a corresponding operation in response to the control instruction, and the control instruction is generated based on the video stream data.

[0008] In one possible implementation, determining the first transmission node based on the first location information of the electronic device includes: determining that the transmission node closest to the electronic device is the first transmission node based on the first location information of the electronic device and the second location information of multiple transmission nodes of the software-defined real-time network of the server.

[0009] In one possible implementation, transmitting the video stream data to the electronic device through the first transmission node includes: adopting a forward error correction method and / or an automatic retransmission method to transmit the video stream data to the first transmission node, and transmitting the video stream data to the electronic device through the first transmission node.

[0010] In one possible implementation, the method further includes: adjusting at least one of the bit rate, frame rate, and resolution of the video stream data based on the network status between the server and the electronic device; and encrypting the video stream data using a secure real-time transport protocol.

[0011] In one possible implementation, forwarding the control instruction to the self-mobile device includes: determining that the transmission node closest to the self-mobile device is the second transmission node based on the third location information of the self-mobile device and the second location information of multiple transmission nodes of the software-defined real-time network of the server; and forwarding the control instruction to the self-mobile device through the second transmission node using a dynamic redundancy method.

[0012] In a third aspect, an embodiment of the present application provides a method for controlling a mobile device, which is applied to an electronic device, the method comprising: Receive and display video stream data transmitted by the server, the video stream data coming from the mobile device; A control instruction is generated based on the video stream data, and the control instruction is transmitted to the server. The server forwards the control instruction to the mobile device, so that the mobile device performs a corresponding operation in response to the control instruction.

[0013] In one possible implementation, generating control instructions based on the video stream data and sending the control instructions to the server includes: converting user operations during the display of the video stream data into the control instructions; serializing the control instructions, adding redundant data to the control instructions using dynamic redundancy technology, encrypting the control instructions based on the Transport Layer Security protocol or the Datagram Transport Layer Security protocol, and transmitting the control instructions to the server through a reliable data transmission channel.

[0014] In a fourth aspect, an embodiment of the present application provides a self-mobile device, which includes: a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the computer program, the self-mobile device implements the above-mentioned self-mobile device control method.

[0015] In a fifth aspect, an embodiment of the present application provides a server, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the server implements the above-mentioned method for controlling a mobile device.

[0016] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the above-mentioned method for controlling a self-mobile device.

[0017] In the seventh aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor in a mobile device, the computer program implements the above-mentioned method for controlling the mobile device, or when the computer program is executed by a processor in a server, the computer program implements the above-mentioned method for controlling the mobile device, or when the computer program is executed by a processor in an electronic device, the computer program implements the above-mentioned method for controlling the mobile device.

[0018] The control method of the self-mobile device, the self-mobile device, the electronic device, the server and the computer-readable storage medium provided in the embodiments of the present application forward video data and control instructions between the self-mobile device and the electronic device through the server, which can reduce data transmission delay and effectively improve remote control efficiency when the electronic device is used to remotely control the self-mobile device. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0020] Figure 1 This is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application.

[0021] Figure 2 This is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application.

[0022] Figure 3 This is a schematic diagram of the hardware structure of a mobile device provided in one embodiment of the present application.

[0023] Figure 4 This is a flowchart of a method for controlling a mobile device provided by an embodiment of the present application.

[0024] Figure 5 This is a flowchart of a method for controlling a mobile device provided by another embodiment of the present application.

[0025] Figure 6 This is a flowchart of a method for controlling a mobile device provided by another embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical solutions and advantages of this application clearer, this application is described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] It should be noted that, in this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A alone, A and B together, and B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," and so on (if any) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or precedence.

[0028] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete manner. The following embodiments and features in the embodiments may be combined with each other unless there is a conflict.

[0029] With the continuous advancement of science and technology and the rapid development of artificial intelligence, the use of autonomous devices (such as mowing robots, cleaning robots, food delivery robots, etc.) to replace manual work can greatly improve work efficiency.

[0030] In application scenarios such as remote monitoring, route adjustment, and equipment failure, users need to remotely control their mobile devices. Remote control of mobile devices is typically achieved using wireless communication technologies such as 5G+WebSocket / MQTT (Message Queuing Telemetry Transport) protocols, RTC (Real-Time Communications) services, and UDP (User Datagram Protocol). However, these wireless communication technologies suffer from high latency and unstable networks, resulting in low communication reliability and impacting the effectiveness of remote control of users' mobile devices.

[0031] Among them, the 5G+WebSocket / MQTT protocol lacks stability in weak network environments and relies on a single communication link, making it susceptible to signal fluctuations. Traditional RTC services have poor cross-platform compatibility, such as insufficient adaptation to vehicle-mounted systems. They also have weak anti-packet loss capabilities, and when the packet loss rate exceeds 30%, the screen is prone to freezing. The development cost of private UDP protocols is high, and the lack of standardized security mechanisms increases the difficulty of maintenance and expansion. C / S architecture remote control requires customized client software, making it difficult to support multi-terminal collaborative operations and limiting the flexibility of the system.

[0032] Related technologies for remote control from mobile devices suffer from shortcomings such as high latency, unstable networks, frequent security vulnerabilities, insufficient cross-platform compatibility, and excessive resource consumption. 5G solutions are significantly affected by signal attenuation in complex terrain (such as mining areas and tunnels), leading to command loss. Measured end-to-end latency on 5G networks can reach up to 300ms, impacting the real-time nature of remote control. Security vulnerabilities in modern automotive APIs have been repeatedly exposed. Attackers can exploit techniques such as CRLF (carriage return / linefeed) injection to spoof identities and hijack vehicle control, posing serious security risks. Traditional solutions lack consistent compatibility across Linux-based car computers, Android, and iOS devices, extending development cycles by over 30% and increasing project delivery challenges. Architectures based on Netty (an asynchronous event-driven network application framework) experience significant server load surges in high-concurrency scenarios (e.g., with tens of thousands of users online), with CPU utilization exceeding 80%, impacting system stability and scalability. These shortcomings severely impact the stability, security, and scalability of remote vehicle control, necessitating more optimized technical solutions to enhance system reliability and user experience.

[0033] In order to solve the problem that the above-mentioned wireless communication technology has high communication delay, weak network instability, low communication reliability, and affects the remote control effect of the mobile device, the embodiment of the present application provides a control method for the mobile device, which forwards the video data and control instructions between the mobile device and the electronic device through the server. When the electronic device is used to remotely control the mobile device, the data transmission delay can be reduced, and the remote control efficiency can be effectively improved.

[0034] See Figure 1 FIG. 1 is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. The electronic device 100 includes, but is not limited to, a processor 110 , a memory 120 , and a display screen 140 connected via a communication bus 130 . Figure 1 This is merely an example of an electronic device and does not constitute a limitation. In other embodiments of the present application, the electronic device may include more components than shown. The control method provided in the embodiments of the present application can be applied to an electronic device 100, which can be a smartphone, a personal computer, a wearable device, a vehicle remote control FPV (First Person View), etc.

[0035] See Figure 2 FIG2 is a schematic diagram of the hardware structure of a server according to an embodiment of the present application. The server 200 includes, but is not limited to, a processor 210 and a memory 220 connected via a communication bus 230 . Figure 2This is merely an example of a server and does not constitute a limitation. In other embodiments of the present application, the server may include more components than shown. The control method provided in the embodiment of the present application from a mobile device can be applied to the server 200, which can be a single server, a server cluster, a cloud server, etc.

[0036] See Figure 3 The figure is a schematic diagram of the hardware structure of the self-moving device provided in one embodiment of the present application. The control method of the self-moving device in the embodiment of the present application can also be applied to Figure 3 The illustrated autonomous device 1 includes a main body, and disposed therein are a memory 11, a processor 12, a power supply 13, a sensor 14, an operating mechanism 15, a communication module 16, a positioning module 17, a drive wheel 18, and a bus 19. Processor 12 is coupled to memory 11, power supply 13, sensor 14, operating mechanism 15, communication module 16, positioning module 17, and drive wheel 18 via bus 19. Autonomous device 1 is an electronic device with autonomous mobility, such as a lawn mower robot, a cleaning robot, or a food delivery robot.

[0037] The aforementioned memory may include one or more random access memories (RAMs) and one or more non-volatile memories (NVMs). RAM can be directly read and written by the processor and can be used to store executable programs (e.g., machine instructions) for the operating system or other running programs, as well as user and application data. RAM may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), and double data rate synchronous dynamic random access memory (DDR SDRAM).

[0038] Non-volatile memory can also store executable programs and user and application data. It can be pre-loaded into random access memory for direct reading and writing by the processor. Non-volatile memory can include disk storage devices and flash memory.

[0039] The memory is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor. The one or more computer programs include multiple instructions. When the multiple instructions are executed by the processor, the control method of the self-mobile device can be implemented on the self-mobile device 1.

[0040] In other embodiments, the electronic device, server, or mobile device may further include an external memory interface for connecting to an external memory to expand storage capacity.

[0041] The aforementioned processor may include one or more processing units, for example, an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0042] The processor provides computing and control capabilities. For example, the processor is used to execute a computer program stored in the memory to implement the above-mentioned method for controlling the mobile device.

[0043] The power supply 13 is used to power the mobile device. In one embodiment of the present application, the power supply 13 may include any one or more power supply devices such as a battery, a fuel generator, a solar power generation module, a wind power generation module, etc.

[0044] The sensor 14 is used to obtain information from the mobile device 1, such as environmental information and movement information of the mobile device 1. In one embodiment of the present application, the sensor 14 may include one or more sensors such as a laser radar, a camera, an infrared sensor, and an encoder.

[0045] The operating mechanism 15 is used to perform corresponding operating tasks, such as mowing, patrolling, sweeping, and spraying pesticides. In some embodiments of the present application, the operating mechanism 15 may include a motor, a transmission mechanism, and a cutter disc. In some embodiments of the present application, the motor can drive the cutter disc to rotate via the transmission mechanism to achieve the mowing function. The motor can also control the movement of the blade to adjust the mowing height and mowing area.

[0046] The communication module 16 is used to enable communication between the mobile device and other devices. In one embodiment of the present application, the communication module 16 can exchange data with other devices via wired and / or wireless communication. Such wireless communication can include one or a combination of Bluetooth, Wi-Fi, and Near Field Communication (NFC).

[0047] Positioning module 17 is used to determine the location of the mobile device. In some embodiments of the present application, positioning module 17 may include one or more positioning modules such as a Global Positioning System (GPS), an inertial navigation system, a real-time kinematic (RTK) carrier phase differential system, and the like.

[0048] The drive wheels 18 are used to enable the self-moving device to move. In some embodiments of the present application, the drive wheels 18 can achieve the movement function of the self-moving device according to the control of the processor 12. In some embodiments of the present application, the drive wheels 18 may include a left drive wheel and a right drive wheel.

[0049] The bus 19 is at least used to provide a channel for mutual communication between the memory 11 , the processor 12 , the power supply 13 , the sensor 14 , the operating mechanism 15 , the communication module 16 , the positioning module 17 , and the driving wheel 18 in the mobile device 1 .

[0050] In other embodiments of the present application, the self-propelled device 1 may further include an anti-collision portion and a steering assembly. The anti-collision portion may be used to prevent the driving wheel 18 from colliding with obstacles in front of the self-propelled device. The steering assembly may be used to adjust the driving wheel 18 to adjust the driving direction.

[0051] It should be understood that the structures illustrated in the embodiments of this application do not constitute specific limitations on the self-moving device 1. In other embodiments of this application, the self-moving device 1 may include more or fewer components than shown, or may combine or separate certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0052] See Figure 4 The flowchart of the control method of the self-moving device provided in one embodiment of the present application is shown in FIG. The control method of the self-moving device in this embodiment can be applied to Figure 3 The self-moving device 1 shown. The control method of the self-moving device includes the following steps: S101, collecting video stream data, and encapsulating the video stream data into a real-time transport protocol RTP data packet based on the user datagram protocol UDP.

[0053] In one embodiment of the present application, a mobile device captures image frames of its surroundings at predetermined intervals using a camera, and records audio frames of the surroundings at predetermined intervals using a microphone. The plurality of consecutive image frames and audio frames form video stream data. For example, the predetermined interval may be 30 milliseconds, 60 milliseconds, 120 milliseconds, or another interval.

[0054] In one embodiment of the present application, a coding standard such as H.264 / H.265 is used to encode and compress the collected video stream to reduce the amount of data, and the encoded and compressed video stream is decomposed into units suitable for network transmission (such as network abstraction layer NAL units) to avoid the maximum transmission unit exceeding the limit due to the excessive amount of data in a single packet. A protocol header is added to the NAL unit, and the protocol header contains timing and control information. Specifically, a buffer for the data packet is reserved, and header information such as the sequence number, timestamp, payload type, and marker bit is written into the packet header based on the offset, and the NAL unit is attached to the packet header, thereby encapsulating the video stream data into a UDP-based real-time transport protocol RTP data packet (RTO over RTP).

[0055] S102 , encoding an RTP data packet and transmitting it to the server via a real-time communication RTC channel of the server, and transmitting the RTP data packet to the electronic device via the server, so that the electronic device displays the video stream data.

[0056] In one embodiment of the present application, the server is an Agora server, which provides an RTC channel for transmitting RTP packets. An electronic device encodes the RTP packets and transmits them to the Agora server via the RTC channel provided by the Agora server. The Agora server transmits the RTP packets to the electronic device. After receiving the RTP packets, the electronic device decodes the RTP packets to obtain video stream data and displays it on a display screen.

[0057] S103: Receive a control instruction of the electronic device forwarded by the server, where the control instruction is generated based on the video stream data.

[0058] In one embodiment of the present application, the electronic device can convert user operations during the display of video streaming data into control instructions, and forward the control instructions to the mobile device through the Agora server.

[0059] S104: Execute corresponding operations in response to the control instruction.

[0060] In one embodiment of the present application, the control instruction is parsed and an integrity check is performed. For example, a check value of the control instruction can be calculated using a hash algorithm or a cyclic redundancy algorithm. If the calculated check value is the same as a preset value, the control instruction is determined to have passed the integrity check. If the calculated check value is different from the preset value, the control instruction is determined to have failed the integrity check.

[0061] In one embodiment of the present application, when the control instruction passes the integrity check, the operation corresponding to the control instruction is executed. For example, the control instruction may include a charging instruction, a moving instruction, a stop instruction, a steering instruction, an operating mechanism opening instruction, an operating mechanism closing instruction, etc. The operation corresponding to the charging instruction is to move to the charging pile for charging. The operation corresponding to the moving instruction is to move in the current direction. The operation corresponding to the stop instruction is to stop moving. The operation corresponding to the steering instruction is to adjust the moving direction. The operation corresponding to the operating mechanism opening instruction is to open the operating mechanism, such as opening the cutter disc. The operation corresponding to the operating mechanism closing instruction is to close the operating mechanism, such as closing the cutter disc.

[0062] See Figure 5 As shown, it is a flow chart of a control method of a self-moving device provided by another embodiment of the present application. The control method of the self-moving device in this embodiment can be applied to Figure 2 The server 200 shown is an aural network server. The control method from a mobile device includes the following steps: S201: Receive video stream data transmitted from a mobile device.

[0063] In one embodiment of the present application, the server receives video stream data encoded and transmitted from the mobile device through an RTC channel.

[0064] S202: Determine a first transmission node based on first location information of the electronic device, and transmit the video stream data to the electronic device through the first transmission node.

[0065] In one embodiment of the present application, a server is connected to multiple transmission nodes to form an SD-RTN. TM(Software Defined Real-Time Network) network, and the server stores the second location information of each transmission node. When an electronic device is used to remotely control a mobile device, the electronic device pre-transmits the first location information to the server. Therefore, the server also stores the first location information of the electronic device. After receiving the video stream data transmitted from the mobile device, the server calculates the distance between each transmission node and the electronic device based on the second location information of each transmission node and the first location information of the electronic device, and determines the transmission node closest to the electronic device as the first transmission node. The first transmission node is the target transmission node for forwarding the video stream data to the electronic device. In this way, the video stream data is transmitted to the electronic device via the first transmission node.

[0066] In one embodiment of the present application, the server uses a forward error correction method and / or an automatic retransmission method to transmit the video stream data to the first transmission node, and transmits the video stream data to the electronic device through the first transmission node. The forward error correction method is to use a preset algorithm to operate on the original data packet to obtain an error correction code. For example, the preset algorithm is a Reed-Solomon code algorithm, a convolutional code algorithm, etc. The error correction code and the data packet are transmitted to the electronic device together. The electronic device calculates a check value for the received data and compares it with the received error correction code. If the check value is different from the error correction code, the erroneous data is located based on the error correction code and corrected. The automatic retransmission method is that after the server transmits the data packet to the electronic device, it determines whether a confirmation frame is received within a preset time length. If the confirmation frame is not received within a certain time length, the current data packet is retransmitted. If the confirmation frame is received within the preset time length, it is determined that the data has been received correctly.

[0067] In one embodiment of the present application, at least one of the bit rate, frame rate, and resolution of the video stream data is adjusted based on the network status between the server and the electronic device. For example, the network bandwidth between the server and the electronic device is detected. If the network bandwidth is greater than or equal to a preset threshold, it is determined that the network status between the server and the electronic device is good, and the bit rate of the video stream data is set to a first bit rate, the bit rate of the video stream data is set to a first frame rate, and the resolution of the video stream data is set to the first resolution. If the network bandwidth is less than the preset threshold, it is determined that the network status between the server and the electronic device is poor, and the bit rate of the video stream data is set to a second bit rate, the bit rate of the video stream data is set to a second frame rate, and the resolution of the video stream data is set to the second resolution. The second bit rate is lower than the first bit rate, for example, the first bit rate is 1000 kbps and the second bit rate is 500 kbps. The second frame rate is lower than the first frame rate, for example, the first frame rate is 60 FPS and the second frame rate is 30 FPS. The second resolution is lower than the first resolution, for example, the first resolution is 1280*720 and the second resolution is 640*360.

[0068] In another embodiment of the present application, the network delay between the server and the electronic device may also be detected. If the network delay is less than or equal to a preset delay threshold, the network status between the server and the electronic device is determined to be good. If the network delay is greater than the preset delay threshold, the network status between the server and the electronic device is determined to be poor.

[0069] In one embodiment of the present application, the Secure Real-time Transport Protocol (SRTP) may be used to encrypt the video stream data.

[0070] S203, receiving a control instruction of the electronic device, and forwarding the control instruction to the mobile device, so that the mobile device performs a corresponding operation in response to the control instruction, where the control instruction is generated based on the video stream data.

[0071] In one embodiment of the present application, based on the third position information of the self-mobile device and the second position information of multiple transmission nodes, the transmission node closest to the self-mobile device is determined to be the second transmission node. The self-mobile device can transmit its real-time third position information along with the video stream data to the server, so that the server also stores the third position information of the self-mobile device. After receiving the control instruction generated based on the video stream data sent by the electronic device, the server calculates the distance between each transmission node and the self-mobile device based on the second position information of each transmission node and the third position information of the self-mobile device, and determines the transmission node closest to the self-mobile device as the second transmission node. The second transmission node is the target transmission node for forwarding the control instruction to the self-mobile device. In this way, the control instruction is forwarded to the self-mobile device through the second transmission node.

[0072] In one embodiment of the present application, the server uses dynamic redundancy to forward control instructions to the mobile device via a second transmission node. The dynamic redundancy is a reliable data transfer (RDT) method. If the network status between the server and the mobile device is poor, redundant packets are added to the control instructions to improve the instruction arrival rate. For example, if the packet loss rate between the server and the mobile device is greater than or equal to a first preset percentage, redundant data of a second preset percentage is added to the control instructions. For example, the first preset percentage is 30%, and the second preset percentage is 50%.

[0073] In one embodiment of the present application, after receiving the control instruction, the mobile device executes the operation corresponding to the control instruction.

[0074] See Figure 6As shown, it is a flow chart of a control method of a self-moving device provided by another embodiment of the present application. The control method of the self-moving device in this embodiment can be applied to Figure 1 The electronic device 100 shown in the figure may be a smart phone, a personal computer, a vehicle remote control FPV (First Person View), etc. The control method from the mobile device includes the following steps: S301, receiving and displaying video stream data transmitted by a server, where the video stream data comes from a mobile device.

[0075] In one embodiment of the present application, after the electronic device receives the video stream data packet transmitted by the server, it uses the SDK (software development kit) interface provided by the Agora server to hard-decode the video stream data packet to obtain video stream data, and displays the video stream data in real time on the display screen of the electronic device.

[0076] S302: Generate a control instruction based on the video stream data, transmit the control instruction to the server, and forward the control instruction to the mobile device through the server, so that the mobile device performs a corresponding operation in response to the control instruction.

[0077] In one embodiment of the present application, user operations during the display of video stream data are converted into control instructions. Specifically, the user can perform touch operations on the display screen based on the real-time video stream data displayed by the electronic device to control the vehicle, and the electronic device converts the touch operations into standardized control instructions. The touch operations are operations performed by the user on the virtual joystick or controls displayed on the display screen. For example, the virtual remote control is used to control the movement and steering of the self-mobile device, and the controls are used to control the self-mobile device to go to the charging station for charging and to open or close the operating mechanism.

[0078] In one embodiment of the present application, after an electronic device generates a control instruction based on a touch operation, it performs Protocol Buffers serialization on the control instruction. Specifically, a proto file is created and a specified Protocol Buffers structure is defined. The proto file contains fields such as the instruction type, parameters, and timestamp. The proto file is compiled to generate language-specific code, which is then enumerated and assigned values, thereby encoding and serializing the instructions and reducing the instruction capacity.

[0079] In one embodiment of the present application, the electronic device uses dynamic redundancy technology to add redundant data to the control instructions, encrypts the control instructions based on the Secure Transaction Layer Protocol or the Datagram Secure Transaction Layer Protocol, and transmits the control instructions to the server through the reliable data transmission RDT channel.

[0080] The above-described embodiments of the present application enable ultra-low-latency data transmission. Using a private UDP protocol, communication overhead is reduced, resulting in a data transmission latency of less than 50ms, significantly lower than the 200-500ms of traditional RTC protocols. This improves the real-time nature of data transmission, reducing the time lag between user actions and their execution on the mobile device, making remote control smoother. Stable audio and video transmission is also achieved, utilizing RTC services (WebRTC standard protocols) to ensure efficient encoding and decoding of audio and video data, providing clear and smooth remote visual feedback. Optimizing the transmission path through 5G networks + WebSocket / MQTT reduces jitter and improves audio and video stability, maintaining high transmission quality even in complex network environments. Through the bidirectional, independent design of the video stream channel (from the mobile device to the Agora server to the electronic device) and the RDT control channel (from the electronic device to the Agora server to the mobile device), remote operators can clearly see the real-time image, make accurate decisions, and ensure fast and stable transmission of control commands. This is independent of the video stream, preventing fluctuations in audio and video bandwidth from impacting control data transmission. The embodiments of the present application can also resist packet loss and enhance stability, and adopt forward error correction, adaptive retransmission and other mechanisms to improve the reliability of data transmission, and ensure a stable remote control experience even in the case of high network packet loss rate. The embodiments of the present application can also optimize the quality of audio and video transmission through adaptive bit rate technology, and can still maintain smooth audio and video images in bandwidth-constrained environments. The embodiments of the present application can also use end-to-end encryption to ensure the security of transmitted data, prevent remote control signals from being stolen or tampered with, and combine device authentication and permission management to ensure the security of remote control and avoid unauthorized access.

[0081] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the methods in the above-mentioned embodiments of the present application.

[0082] The computer-readable storage medium may be the internal memory of the mobile device described in the above embodiment, such as the hard disk or memory of the mobile device. The computer-readable storage medium may also be an external storage device of the mobile device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the mobile device.

[0083] In one embodiment of the present application, the computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created based on the use of the mobile device, etc.

[0084] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0085] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0086] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0087] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0088] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A control method for a self-moving device, applied to a self-moving device, characterized in that: The method comprises: Collecting video stream data, encapsulating the video stream data into a real-time transport protocol (RTP) data packet based on the user datagram protocol (UDP), encoding and transmitting the RTP data packet to the server via a real-time communication (RTC) channel of the server, and transmitting the RTP data packet to an electronic device via the server, so that the electronic device displays the video stream data; receiving a control instruction of the electronic device forwarded by the server, where the control instruction is generated based on the video stream data; In response to the control instruction, a corresponding operation is performed.

2. The method for controlling a self-moving device according to claim 1, wherein: The performing of a corresponding operation in response to the control instruction includes: The control instruction is parsed and an integrity check is performed. If the control instruction passes the integrity check, an operation corresponding to the control instruction is executed.

3. A control method for a mobile device, applied to a server, characterized in that: The method comprises: receiving video stream data transmitted from a mobile device; determining a first transmission node based on first location information of the electronic device, and transmitting the video stream data to the electronic device through the first transmission node; A control instruction of the electronic device is received, and the control instruction is forwarded to the self-mobile device, so that the self-mobile device performs a corresponding operation in response to the control instruction, where the control instruction is generated based on the video stream data.

4. The method for controlling a self-moving device according to claim 3, wherein: The determining the first transmission node based on the first location information of the electronic device includes: Based on the first location information of the electronic device and the second location information of multiple transmission nodes of the software-defined real-time network of the server, it is determined that the transmission node closest to the electronic device is the first transmission node.

5. The method for controlling a self-moving device according to claim 3, wherein: The transmitting the video stream data to the electronic device through the first transmission node includes: The video stream data is transmitted to the first transmission node using a forward error correction method and / or an automatic retransmission method, and the video stream data is transmitted to the electronic device through the first transmission node.

6. The method for controlling a self-moving device according to claim 3, wherein: The method further comprises: adjusting at least one of a bit rate, a frame rate, and a resolution of the video stream data based on a network status between the server and the electronic device; The video stream data is encrypted using a secure real-time transport protocol.

7. The method for controlling a self-moving device according to claim 3, wherein: The forwarding of the control instruction to the mobile device includes: Based on the third location information of the self-mobile device and the second location information of multiple transmission nodes of the software-defined real-time network of the server, determining the transmission node closest to the self-mobile device as the second transmission node; The control instruction is forwarded to the mobile device through the second transmission node in a dynamic redundancy manner.

8. A method for controlling a self-moving device, applied to an electronic device, characterized in that: The method comprises: Receive and display video stream data transmitted by the server, the video stream data coming from the mobile device; A control instruction is generated based on the video stream data, and the control instruction is transmitted to the server. The server forwards the control instruction to the mobile device, so that the mobile device performs a corresponding operation in response to the control instruction.

9. The method for controlling a self-moving device according to claim 8, wherein: The generating a control instruction based on the video stream data and sending the control instruction to the server includes: converting user operations during the display of the video stream data into the control instructions; The control instructions are serialized, redundant data is added to the control instructions using dynamic redundancy technology, the control instructions are encrypted based on the Secure Transaction Layer protocol or the Datagram Transport Layer Security protocol, and the control instructions are transmitted to the server through a reliable data transmission channel.

10. A self-propelled device, characterized in that: The self-mobile device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the self-mobile device implements the self-mobile device control method according to any one of claims 1 to 2.

11. A server, characterized in that: The server includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the server implements the control method for a mobile device according to any one of claims 3 to 7.

12. An electronic device, characterized in that: The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the electronic device implements the control method for a self-mobile device as described in any one of claims 8 to 9.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor in a mobile device, implements the control method of a mobile device as described in any one of claims 1 to 2, or, when executed by a processor in a server, implements the control method of a mobile device as described in any one of claims 3 to 7, or, when executed by a processor in an electronic device, implements the control method of a mobile device as described in any one of claims 8 to 9.

Images