Robot remote debugging method and system, electronic device, storage medium and product

Abstract

The application provides a robot remote debugging method, system, electronic equipment, storage medium and product, comprising: receiving a starting instruction for at least one target debugging object, creating a virtual display screen corresponding to each target debugging object, and displaying each target debugging object currently to be cooperatively debugged; each virtual display screen is displayed on the same robot device screen; receiving a debugging instruction, displaying a debugging operation corresponding to the debugging instruction on the corresponding virtual display screen, and uniformly recording and encoding each virtual display screen to obtain a mixed screen video stream; the mixed screen video stream is used to instruct the debugging platform to cut the mixed screen video stream when a stream pulling request is received, obtain a screen video stream corresponding to a debugging end, and send the screen video stream to the debugging end. The application enables each debugging object to have an independent running screen stream display, overcomes the physical screen limitation, realizes that different debugging objects are not disturbed when multiple people cooperatively debug the different debugging objects in parallel, and improves the debugging efficiency of the robot equipment.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, system, electronic device, storage medium, and product for remote debugging of a robot. Background Technology

[0002] With the rapid development of the Internet of Things and the continuous breakthroughs in cutting-edge technologies such as artificial intelligence and machine learning, the robotics industry has ushered in unprecedented development opportunities. Robotics technology is demonstrating broad application prospects in areas such as home automation, personal services, education and entertainment, and logistics warehousing, offering a superior user experience and higher efficiency. However, during project development or after-sales service, when robot equipment malfunctions, location limitations or resource constraints often prevent the timely acquisition of critical information, hindering rapid problem localization and resolution, and severely impacting fault handling efficiency.

[0003] During the project development phase, the high cost of robotic equipment necessitates that R&D teams in different regions share the same equipment. This typically involves remotely operating a host computer to debug the robot. However, this method does not support simultaneous operation by multiple people. Even when R&D personnel are physically present at the robot, the limitations of the robot's screen force them to take turns operating it, significantly reducing R&D efficiency.

[0004] For malfunctions in after-sales service robot equipment, the current debugging process mainly relies on customers manually operating the physical robot, with software developers analyzing and judging based on the manual operation results and uploaded logs. However, this method has several challenges: due to time zone and working hour differences, customers may not be able to perform necessary operations in a timely manner; uploaded log information may be incomplete or lack key data. For some complex or unrecorded problems, the above method often fails to provide effective solutions, thus affecting the debugging progress.

[0005] Therefore, improving the debugging efficiency of robot equipment is an urgent problem to be solved. Summary of the Invention

[0006] This invention provides a method, system, electronic device, storage medium, and product for remote debugging of robots, which can improve the debugging efficiency of robot equipment.

[0007] This invention provides a remote debugging method for robots, applied to robot equipment, the method comprising: Upon receiving a start command for at least one target debugging object, a virtual display screen corresponding to each target debugging object is created, and the corresponding target debugging object is displayed on each virtual display screen; wherein, the target debugging object is the debugging object to be collaboratively debugged, and the virtual display screens are displayed on the same robot device screen; Upon receiving a debugging instruction for the target debugging object, the debugging operation corresponding to the debugging instruction is displayed on the virtual display screen corresponding to the target debugging object, and the debugging operation displayed on each virtual display screen is uniformly screen-recorded and encoded to obtain a mixed-screen video stream, which is then pushed to the debugging platform. The mixed-screen video stream is used to instruct the debugging platform to trim the mixed-screen video stream upon receiving a pull request from any debugging end, thereby obtaining the screen video stream corresponding to any debugging end and sending it to that debugging end.

[0008] According to a remote robot debugging method provided by the present invention, the step of creating a virtual display screen corresponding to each target debugging object and displaying the corresponding target debugging object on each virtual display screen includes: For each target debugging object, after starting the target debugging object, obtain the task stack ID of the target debugging object; Create a virtual display screen and obtain the display screen ID of the virtual display screen; The task stack ID of the target debugging object is associated with the display ID of the virtual display screen to display the corresponding target debugging object on each virtual display screen.

[0009] According to the robot remote debugging method provided by the present invention, after creating virtual display screens corresponding to each target debugging object and displaying the corresponding target debugging objects on each virtual display screen, the method further includes: For each virtual display screen, the screen percentage coordinates corresponding to the virtual display screen are determined based on the display screen ID of the virtual display screen, the screen width ratio of the virtual display screen, and the total number of virtual display screens. The screen percentage coordinates corresponding to each virtual display screen are reported to the debugging platform; the screen percentage coordinates are used to instruct the debugging platform to determine the target virtual display screen corresponding to any debugging terminal when it receives a pull request from any debugging terminal, and to trim the mixed video stream based on the screen percentage coordinates corresponding to the target virtual display screen to obtain the screen video stream corresponding to any debugging terminal.

[0010] According to a remote robot debugging method provided by the present invention, the step of displaying the debugging operation corresponding to the debugging instruction on the virtual display screen corresponding to the target debugging object includes: Based on the display ID of the virtual display screen corresponding to the target debugging object, construct an input event corresponding to the debugging instruction carrying the display screen ID; Based on the display screen ID carried by the input event, the input event is forwarded to the virtual display screen corresponding to the display screen ID, so as to display the debugging operation corresponding to the debugging command on the virtual display screen.

[0011] According to a robot remote debugging method provided by the present invention, the resolution of the mixed-screen video stream and the resource allocation of each target debugging object are determined based on the system resource usage; the virtual display screens corresponding to each target debugging object and the processes corresponding to each target debugging object are deleted after debugging is completed.

[0012] This invention also provides another method for remote robot debugging, applied at the debugging end, the method comprising: In response to a user's startup operation on a target debugging object, a startup command for the target debugging object is generated and sent to the robot device through the debugging platform; the startup command is used to instruct the robot device to create a virtual display screen corresponding to the target debugging object and display the corresponding target debugging object on the virtual display screen; In response to a user's debugging operation on the target debugging object, a debugging instruction for the target debugging object is generated and sent to the robot device through the debugging platform. The debugging instruction is used to instruct the robot device to display the debugging operation corresponding to the debugging instruction on the virtual display screen corresponding to the target debugging object, and to uniformly record and encode the debugging operations displayed on each virtual display screen to obtain a mixed screen video stream and push it to the debugging platform. The target debugging object is the debugging object to be collaboratively debugged, and the virtual display screens are displayed on the same robot device screen. Send a pull request to the debugging platform, receive the screen video stream corresponding to the debugging end returned by the debugging platform, and display the screen video stream.

[0013] According to a remote robot debugging method provided by the present invention, the step of generating a debugging instruction for the target debugging object in response to a user's debugging operation on the target debugging object and sending it to the robot device through a debugging platform includes: In response to a user's debugging operation on the target debugging object, determine the input event corresponding to the debugging operation; The input events are converted into debugging instructions based on a multi-person collaborative remote debugging protocol format and sent to the robot device through the debugging platform; The header of the multi-person collaborative remote debugging protocol format includes a source address, a destination address, control bits, a cyclic redundancy checksum, a timestamp, and a unique identifier; the unique identifier includes an operation timestamp, the display ID of the virtual display screen corresponding to the debugging terminal, and a randomly generated string; The display ID of the virtual display screen corresponding to the debugging terminal is obtained by the debugging terminal from the debugging platform.

[0014] According to a robot remote debugging method provided by the present invention, the step of receiving the screen video stream corresponding to the debugging terminal returned by the debugging platform and displaying the screen video stream includes: The screen video stream corresponding to the debugging terminal returned by the debugging platform is decoded to obtain the video encoding time; If the time difference between the video encoding time and the current decoding time is greater than a preset difference threshold, a screen lock command is generated; Based on the screen lock command, the debugging screen is locked.

[0015] According to a remote debugging method for a robot provided by the present invention, after displaying the screen video stream, the method further includes: In response to a resolution adjustment operation, a resolution adjustment request for the screen video stream is generated; The resolution adjustment request is sent to the debugging platform; the resolution adjustment request is used to instruct the debugging platform to determine the target debugging end that meets the resolution adjustment requirements, determine the first display screen ID of the virtual display screen corresponding to the target debugging end, and re-cut the mixed screen video stream based on the screen percentage coordinates corresponding to the first display screen ID to obtain the updated first screen video stream and send it to the debugging end. Receive and display the updated first-screen video stream.

[0016] According to a remote debugging method for a robot provided by the present invention, after displaying the screen video stream, the method further includes: In response to a viewing operation targeting a video stream on a target screen, a viewing command is generated; The viewing instruction is sent to the debugging platform; the viewing instruction is used to instruct the debugging platform to determine the second display ID corresponding to the target screen video stream, and to re-cut the mixed screen video stream based on the screen percentage coordinates corresponding to the second display ID to obtain the updated second screen video stream and send it to the debugging terminal; Receive and display the updated second-screen video stream.

[0017] This invention also provides another method for remote robot debugging, applied to a debugging platform, the method comprising: The robot receives start commands from multiple debugging terminals for at least one target debugging object and sends the start commands to the robot device. The start commands instruct the robot device to create virtual display screens corresponding to each target debugging object and display the corresponding target debugging object on each virtual display screen. The target debugging object is the debugging object to be collaboratively debugged, and the virtual display screens are displayed on the same robot device screen. The system receives debugging instructions from multiple debugging terminals for at least one target debugging object, and sends the debugging instructions to the robot device. The debugging instructions are used to instruct the robot device to display the debugging operation corresponding to the debugging instructions on the virtual display screen corresponding to the target debugging object, and to uniformly record and encode the debugging operations displayed on each virtual display screen to obtain a mixed screen video stream and push it to the debugging platform. Upon receiving a pull request from any debugging terminal, the mixed video stream is trimmed to obtain the screen video stream corresponding to the debugging terminal and sent to the debugging terminal.

[0018] According to a remote debugging method for a robot provided by the present invention, the step of trimming the mixed-screen video stream to obtain the screen video stream corresponding to any debugging end includes: For any debugging terminal, obtain the screen percentage coordinates corresponding to the virtual display screen based on the virtual display screen corresponding to the debugging terminal; Calculate the pixel coordinates and the width and height of the screen video stream corresponding to the debugging terminal based on the screen mixing resolution and screen mixing mode. Based on the screen percentage coordinates corresponding to the virtual display screen, the pixel coordinates, and the width and height of the screen video stream corresponding to the debugging terminal, the mixed screen video stream is trimmed to obtain the screen video stream corresponding to any debugging terminal.

[0019] According to the robot remote debugging method provided by the present invention, after obtaining the screen video stream corresponding to any of the debugging terminals, the method further includes: Based on the rendering requirements and network conditions of any of the debugging terminals, adjust the video parameters of the screen video stream corresponding to any of the debugging terminals, so as to send the screen video stream with adjusted parameters to any of the debugging terminals.

[0020] According to a robot remote debugging method provided by the present invention, the step of receiving start commands sent by multiple debugging terminals for at least one target debugging object, and sending the start commands to the robot device, includes: When multiple debugging endpoints send debugging commands for the same target debugging object, within each future time slice, grant any one debugging endpoint access to the target debugging object and disable the other debugging endpoints' access to the target debugging object.

[0021] This invention also provides a remote robot debugging system, the system comprising: The debugging end is used to respond to the user's start operation for the target debugging object, generate a start command for the target debugging object, and send it to the robot device; A debugging platform is used to send the start command to the robot device; A robotic device is used to create virtual display screens corresponding to each target debugging object, and to display the corresponding target debugging object on each virtual display screen; wherein, the target debugging object is the debugging object to be collaboratively debugged, and the virtual display screens are displayed on the same robotic device screen; The debugging terminal is also used to respond to the user's debugging operation on the target debugging object, generate debugging instructions for the target debugging object, and send them to the debugging platform; The debugging platform is also used to send the debugging instructions to the robot device; The robot device is also used to, upon receiving a debugging instruction for the target debugging object, display the debugging operation corresponding to the debugging instruction on the virtual display screen corresponding to the target debugging object, and uniformly record and encode the debugging operations displayed on each virtual display screen to obtain a mixed screen video stream and push it to the debugging platform. The debugging platform is also used to trim the mixed video stream when it receives a pull request from any debugging terminal, to obtain the screen video stream corresponding to any debugging terminal, and send it to the debugging terminal. The debugging terminal is also used to receive and display the screen video stream sent by the debugging platform.

[0022] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the robot remote debugging method described above.

[0023] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robot remote debugging method as described above.

[0024] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the robot remote debugging method as described above.

[0025] The robot remote debugging method, system, electronic device, storage medium and product provided by the present invention dynamically create and manage virtual display screens corresponding to each target debugging object, so that each target debugging object has an independent running screen stream display, overcoming the limitations of physical screens, realizing that multiple people can debug different target debugging objects in parallel without disturbing each other, which is conducive to improving the debugging efficiency of robot equipment. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This is one of the flowcharts illustrating the remote debugging method for robots provided in this embodiment of the invention.

[0028] Figure 2 This is a schematic diagram of dynamically creating a virtual display screen according to an embodiment of the present invention.

[0029] Figure 3 This is the virtual screen mixing mode for four people provided in the embodiments of the present invention.

[0030] Figure 4 This is the virtual screen mixing mode for ten people provided in the embodiments of the present invention.

[0031] Figure 5 This is the second flowchart illustrating the robot remote debugging method provided in this embodiment of the invention.

[0032] Figure 6 This is a schematic diagram of the debugging terminal operation application provided in the embodiment of the present invention.

[0033] Figure 7 This is a schematic diagram of the debugging terminal simulation provided in an embodiment of the present invention.

[0034] Figure 8 This is the third flowchart of the robot remote debugging method provided in this embodiment of the invention.

[0035] Figure 9 This is a timing diagram of remote robot debugging provided in an embodiment of the present invention.

[0036] Figure 10 This is a schematic diagram of the structure of the robot remote debugging system provided in an embodiment of the present invention.

[0037] Figure 11 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0039] In the description of embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Those skilled in the art will understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0040] Remote debugging methods offer significantly improved efficiency compared to traditional methods. Several technologies can be used to implement remote debugging: 1) Intranet Penetration Technology: Also known as NAT (Network Address Translation Traversal) penetration, this technology allows devices without a public IP (Internet Protocol) address to establish connections and communicate with external networks through an intermediate server with a public IP address. By setting up an intranet penetration service, the public IP address of the terminal being debugged is exposed, allowing remote external personnel to connect to the terminal.

[0041] However, this method directly exposes the device's public IP address and does not implement ADB (Android Debug Bridge) authentication. Any user participating in the test can use the terminal as long as they know the network address and port of the terminal being debugged, which poses a risk of theft and has low security. At the same time, it relies on a relay server as a bridge between the internal network and the external network. If the relay server fails or the network is unstable, it will affect the transmission speed and effectiveness of the internal network penetration.

[0042] 2) VPN (Virtual Private Network) technology: A private network is established on a public network, placing the terminal to be debugged and the remote personnel under the same local area network. Under the same local area network, the remote personnel can debug the terminal through ADB wireless connection.

[0043] However, VPN applications on the Android platform may contain security vulnerabilities that could be exploited to steal sensitive information. VPN connections require data to be transmitted through encrypted channels, which is typically time-consuming and can lead to increased network latency and bandwidth consumption. Setting up a dedicated VPN service requires managing multiple network devices and configurations, which is costly, and improper configuration or overload can also affect the speed and stability of remote access.

[0044] 3) Remote Desktop Assistance: Remote personnel can view the desktop recording stream of the terminal being debugged in real time using professional remote assistance software to identify any abnormalities. This method allows users to remotely operate the assisted device by sharing their screen; during this process, the remote operator can control the target device through gesture guidance or by enabling accessibility mode.

[0045] However, this method is usually limited to accessing surface-level information in the user interface and cannot be used to view logs in real time or perform in-depth debugging tasks.

[0046] 4) Signaling transmission and reception method: A stable long-term connection communication mechanism is established between the terminal being debugged and the server. The server sends specific debugging commands to the terminal, and the terminal executes the corresponding operations one by one according to the command requirements, and promptly reports the operation results back to the server.

[0047] However, each command execution involves starting a separate child process to execute the command or call the program, making it impossible to perform coherent operations within the context. For example, if you want to enter / data / android / , you cannot execute cd / data first and then cd android, which is cumbersome.

[0048] 5) Host computer mode: This mode relies on a computer device near the terminal. The computer device is connected to the terminal being debugged via wired connection or wireless connection via ADB within the same local area network. Remote personnel then use remote desktop tools to control the computer device to debug the terminal.

[0049] However, this method requires deploying a host computer next to the robot and building a complete debugging environment. Due to many physical limitations, it is difficult to apply to real user environments.

[0050] In summary, existing remote debugging technologies have certain problems in terms of debugging convenience, variety, and security.

[0051] To address the aforementioned issues, this invention addresses these problems by dynamically creating and managing virtual displays corresponding to each target debugging object. This allows each target debugging object to have an independent running screen stream, overcoming the limitations of physical screens and enabling multiple people to collaboratively debug different target debugging objects in parallel without interfering with each other. This is beneficial for improving the debugging efficiency of robot equipment.

[0052] Figure 1 This is one of the flowcharts illustrating the remote debugging method for robots provided in this embodiment of the invention. (Refer to...) Figure 1 This invention provides a method for remote debugging of a robot, which may specifically include the following steps: Step 101: Upon receiving a start command for at least one target debugging object, create a virtual display screen corresponding to each target debugging object and display the corresponding target debugging object on each virtual display screen; wherein, the target debugging object is the debugging object to be collaboratively debugged, and the virtual display screens are displayed on the same robot device screen.

[0053] The execution subject of the robot remote debugging method provided in this embodiment of the invention can be a robot device, a component in the robot device, an integrated circuit, or a chip.

[0054] In some embodiments, the debugging object may include applications, simulation terminals, and peripheral devices of the robot device, etc., and the embodiments of the present invention do not specifically limit this. When the debugging personnel need to debug the running status of the application, they can enter the application package name on the control terminal to initiate an application debugging request; when command-line debugging is required, they can initiate a simulation terminal debugging request; when peripheral devices on the robot device side need to be debugged, they can enter the IP address of the specified peripheral device on the control terminal to initiate a peripheral debugging request.

[0055] The following embodiments of the present invention use an application as an example to illustrate the debugging object.

[0056] In this embodiment of the invention, multiple debugging terminals can simultaneously initiate startup commands for at least one target application; the debugging platform can send startup commands for at least one target application to the robot device; after receiving the startup command for at least one target application, the robot device can create virtual displays corresponding to each target application and display the corresponding target application on each virtual display, thereby realizing the parallel collaborative operation and debugging of multiple target applications.

[0057] In this embodiment of the invention, different virtual displays are used to display different target applications. All virtual displays are displayed on the same screen, thereby overcoming the limitations of physical screens while enabling the parallel collaborative running and debugging of multiple target applications, which is beneficial to improving debugging efficiency.

[0058] The target applications targeted by different debugging terminals can be completely different, partially the same, or the same.

[0059] It should be noted that the virtual display screen in this embodiment of the invention is dynamically created. The number of virtual display screens created will vary depending on the number of debugging objects (i.e., target debugging objects) to be collaboratively debugged.

[0060] Figure 2 This is a schematic diagram illustrating the dynamic creation of a virtual display screen according to an embodiment of the present invention. (Refer to...) Figure 2 In some embodiments, the robot device may have a virtual screen management function for dynamically managing virtual displays and application distribution mechanisms on the robot device's end side. Compared with existing multi-user remote debugging technologies, the embodiments of the present invention are not limited to multiple people simultaneously sending debugging commands to obtain instantaneous status information of the robot device. Instead, by dynamically creating and managing one or more virtual displays on the robot device, each display is equipped with an independent display layer (Surface), and the display layers do not interfere with each other, thereby realizing the parallel collaborative operation of multiple interactive applications, overcoming the limitations of physical screens, and improving the debugging efficiency of the robot device.

[0061] Step 102: Upon receiving a debugging instruction for the target debugging object, display the debugging operation corresponding to the debugging instruction on the virtual display screen corresponding to the target debugging object, and uniformly record and encode the debugging operation displayed on each virtual display screen to obtain a mixed screen video stream and push it to the debugging platform; the mixed screen video stream is used to instruct the debugging platform to trim the mixed screen video stream upon receiving a pull stream request from any debugging terminal to obtain the screen video stream corresponding to any debugging terminal and send it to the any debugging terminal.

[0062] In this embodiment of the invention, multiple debugging terminals can simultaneously initiate debugging commands for at least one target application; the debugging platform can send the debugging commands for at least one target application to the robot device; after receiving the debugging commands for at least one target application, the robot device can display the debugging operations corresponding to the debugging commands on the virtual display screen corresponding to the target application, and uniformly record and encode the debugging operations displayed on each virtual display screen to obtain a mixed screen video stream, and push the mixed screen video stream to the debugging platform.

[0063] In some embodiments, to address the problem of limited device encoding resources (generally only one channel of encoding is possible), embodiments of the present invention can perform unified screen recording capture and video stream encoding processing on each display layer configured on each virtual display screen, and dynamically generate mixed screen video streams with different mixed screen modes and different resolution sizes according to the number of screens on the virtual display screen, thereby pushing streaming media data in real time.

[0064] Figure 3This is the virtual screen mixing mode for four people provided in the embodiments of the present invention. Figure 4 This is the virtual screen mixing mode for ten people provided in this embodiment of the invention. (Refer to...) Figure 3 and Figure 4 In some embodiments, when each user initiates parallel collaborative debugging for a different application, the created virtual display screen corresponds one-to-one with the debugging user and the target application. The virtual screen mixing mode can also differ depending on the number of users coordinating in parallel. Different virtual screen mixing modes can output mixed video streams with different resolutions; for example, the mixed video stream output can be 1280×720, 1920×1080, or other resolutions supported by the robot device.

[0065] In this embodiment of the invention, after receiving the mixed-screen video stream pushed in real time by the robot device, if the debugging platform receives a pull request sent by a certain debugging end, it will trim the mixed-screen video stream to obtain the screen video stream corresponding to the debugging end and send it to the debugging end, thereby displaying the screen video stream corresponding to the target application currently being remotely debugged by the debugging end on the debugging end.

[0066] This invention, upon receiving a start command for at least one target debugging object, creates a virtual display screen corresponding to each target debugging object and displays the corresponding target debugging object on each virtual display screen. The target debugging object is the object to be collaboratively debugged, and all virtual display screens are displayed on the same robot device screen. Upon receiving a debugging command for a target debugging object, the debugging operation corresponding to the debugging command is displayed on the virtual display screen corresponding to the target debugging object. The debugging operations displayed on each virtual display screen are uniformly recorded and encoded to obtain a mixed-screen video stream, which is then pushed to the debugging platform. This instructs the debugging platform to trim the mixed-screen video stream upon receiving a pull request from any debugging end, obtaining the screen video stream corresponding to any debugging end, and sending it to that debugging end. This invention, by dynamically creating and managing virtual display screens corresponding to each target debugging object, allows each target debugging object to have an independent running screen stream display, overcoming the limitations of physical screens. It enables multiple people to collaboratively debug different target debugging objects in parallel without interference, thus improving the debugging efficiency of the robot device.

[0067] In one optional embodiment, the step of creating a virtual display screen corresponding to each target debugging object and displaying the corresponding target debugging object on each virtual display screen may specifically include: Step S11: For each target debugging object, after starting the target debugging object, obtain the task stack ID of the target debugging object; Step S12: Create a virtual display screen and obtain the display screen ID of the virtual display screen; Step S13: Associate the task stack ID of the target debugging object with the display ID of the virtual display screen, so as to display the corresponding target debugging object on each virtual display screen.

[0068] In some embodiments, taking an Android platform device as an example, this embodiment of the invention uses the creation of a virtual display screen as the basis for remote robot debugging. For each target application, after the target application is launched, the STACK_ID (task stack ID) of the application can be obtained in real time. A virtual display screen is created to obtain the DISPLAY_ID (display ID). The task stack ID is associated with the display ID, and the process is performed via am display move-stack.<STACK_ID><DISPLAY_ID> Displaying the application's STACK_ID on the associated virtual display's DISPLAY_ID allows different target applications to be displayed on different virtual displays, avoiding screen overlap issues during multi-user collaboration.

[0069] In an optional embodiment, after creating virtual displays corresponding to each target debugging object and displaying the corresponding target debugging object on each virtual display, the method may further include: Step S21: For each virtual display screen, determine the screen percentage coordinates corresponding to the virtual display screen based on the display screen ID of the virtual display screen, the screen width ratio of the virtual display screen, and the total number of virtual display screens; Step S22: Report the screen percentage coordinates corresponding to each virtual display screen to the debugging platform; the screen percentage coordinates are used to instruct the debugging platform to determine the target virtual display screen corresponding to any debugging terminal when it receives a pull request from any debugging terminal, and to trim the mixed video stream based on the screen percentage coordinates corresponding to the target virtual display screen to obtain the screen video stream corresponding to any debugging terminal.

[0070] In this embodiment of the invention, the screen flow coordinates of the display ID of the virtual display screen corresponding to each target application can be recorded in units of screen percentage, thereby ensuring compatibility with different resolutions.

[0071] In some embodiments, a mapping relationship can be established between the top-left corner of the virtual display screen corresponding to the application and the virtual display screen in the form of (left, top, right, bottom) to obtain the screen percentage coordinates corresponding to the virtual display screen. Here, left can refer to the percentage distance from the top-left corner of the virtual display screen to the left edge of the real device screen, top can refer to the percentage distance from the top-left corner of the virtual display screen to the top edge of the real device screen, right can refer to the percentage distance from the top-left corner of the virtual display screen to the right edge of the real device screen, and bottom can refer to the percentage distance from the top-left corner of the virtual display screen to the bottom edge of the real device screen.

[0072] For example, refer to Figure 4 The screen percentage coordinates for application 6 can be (0.75, 0.25, 0.25, 0.75).

[0073] In some embodiments, the coordinate formula for the screen stream corresponding to the application can be as follows: x = Fun1(s, w, n), w = 0.5 (s = 1 or s = 2), w = 0.25 (when n > 4 and s > 2); Where x is the coordinate of the top left corner of the virtual display screen, w is the screen width ratio, s is the display ID of the virtual display screen corresponding to the application, and n is the total number of application screens (i.e., the total number of virtual displays).

[0074] Among them, the screen width ratio can refer to the ratio of the width of the virtual display screen to the width of the actual screen of the robot device.

[0075] In this embodiment of the invention, the screen percentage coordinates corresponding to each virtual display screen can be reported to the debugging platform. When the debugging platform receives a streaming request from a debugging terminal, it can determine the target virtual display screen corresponding to the debugging terminal, and based on the screen percentage coordinates corresponding to the target virtual display screen, trim the mixed video stream to obtain the screen video stream corresponding to the debugging terminal and distribute it to the debugging terminal.

[0076] In one optional embodiment, displaying the debugging operation corresponding to the debugging command on the virtual display screen corresponding to the target debugging object may specifically include: Step S31: Based on the display screen ID of the virtual display screen corresponding to the target debugging object, construct an input event corresponding to the debugging instruction carrying the display screen ID; Step S32: Based on the display screen ID carried by the input event, forward the input event to the virtual display screen corresponding to the display screen ID, so as to display the debugging operation corresponding to the debugging command on the virtual display screen.

[0077] In some embodiments, compared to existing debugging signaling processing technologies, the robot device can have fusion sensing capabilities to support multiple users simultaneously and collaboratively touching different applications. After receiving a debugging command from a remote debugging peripheral, the robot device can classify the commands by virtual display ID and construct a KeyEvent or MotionEvent (i.e., the input event corresponding to the debugging command) carrying the virtual display ID. Specifically, the Display of the InputEvent event can be set to the Virtual Display ID using the private function setDisplay() of InputEvent and forwarded to different virtual displays; the simulated input of the remote peripheral can be achieved through event injection using injectInputEvent(), enabling the input event on the remote debugging terminal's display to be forwarded to the corresponding Virtual Display on the robot device, and then the corresponding debugging operation (such as clicking a button, entering text, etc.) can be executed.

[0078] In one optional embodiment, the resolution of the mixed video stream and the resource allocation of each target debugging object are determined based on the system resource usage; the virtual display screens corresponding to each target debugging object and the processes corresponding to each target debugging object are deleted after debugging is completed.

[0079] The creation and management of virtual displays and simulated terminals can consume significant system resources, leading to a decline in the performance of robotic equipment.

[0080] In this embodiment of the invention, a dynamic creation and destruction mechanism for virtual displays is introduced (i.e., virtual displays are dynamically created during parallel remote collaborative debugging; after disconnecting remote debugging, the virtual displays corresponding to each target application are deleted, and the corresponding application processes are destroyed). At the same time, the system resource usage is monitored, and the resolution of the mixed-screen video stream output is changed in a timely manner based on the system resource usage. The secondary resource allocation for application debugging weight is also adjusted, for example, the resolution of the mixed-screen stream is adjusted from 1920×1080 to 1280×720, or some applications are stopped or exited from the background, thereby avoiding the consumption of a large amount of system resources and improving the performance of the robot equipment.

[0081] In one optional embodiment, unlike existing graphical applications for simulated terminals installed on the remote debugging end, this embodiment of the invention, when the debugging object is a simulated terminal, can localize the simulated terminal command generator. That is, the simulated terminal is installed on the robot device side, and the context environment is maintained on the robot device side. This effectively solves the problem of inconsistent context operations in traditional signaling transmission and reception (traditional signaling transmission and reception is based on executing one instruction after another in the same initial environment; to enter / data / android / , it is impossible to first execute cd / data and then cd android). Through the virtual screen stream of the simulated terminal on the robot device side, the remote debugging end can view the debugging results in real time and input debugging commands through the screen stream, providing the debugger with an immersive Linux command-line experience.

[0082] In one optional embodiment, when the robot device receives a request to debug the peripheral device, the simulated terminal manager forks a child process to act as an adb server, allowing the debugger to use adb connect... <ip>It connects to devices running on the same local area network and acts as a host computer to remotely debug peripheral devices.

[0083] Figure 5 This is the second flowchart illustrating the remote robot debugging method provided in this embodiment of the invention. (Refer to...) Figure 5 This invention provides a method for remote debugging of a robot, which may specifically include the following steps: Step 501: In response to the user's start operation for the target debugging object, a start command for the target debugging object is generated and sent to the robot device through the debugging platform; the start command is used to instruct the robot device to create a virtual display screen corresponding to the target debugging object and display the corresponding target debugging object on the virtual display screen; Step 502: In response to the user's debugging operation on the target debugging object, a debugging instruction for the target debugging object is generated and sent to the robot device through the debugging platform; the debugging instruction is used to instruct the robot device to display the debugging operation corresponding to the debugging instruction on the virtual display screen corresponding to the target debugging object, and to uniformly record and encode the debugging operations displayed on each virtual display screen to obtain a mixed screen video stream and push it to the debugging platform; wherein, the target debugging object is the debugging object to be collaboratively debugged, and each virtual display screen is displayed on the same robot device screen; Step 503: Send a pull stream request to the debugging platform, receive the screen video stream corresponding to the debugging terminal returned by the debugging platform, and display the screen video stream.

[0084] The remote debugging method for robots provided in this invention can be applied to debugging terminals. Each debugging terminal can respond to a user's startup operation for a target application, generate a startup command for the target application, and send it to the robot device through a debugging platform. The debugging platform can send startup commands for at least one target application initiated simultaneously by multiple debugging terminals to the robot device. After receiving a startup command for at least one target application, the robot device can create virtual displays corresponding to each target application and display the corresponding target application on each virtual display, thereby realizing the parallel collaborative debugging of multiple target applications.

[0085] In this embodiment of the invention, different virtual displays are used to display different target applications. All virtual displays are displayed on the same screen, thereby overcoming the limitations of physical screens while enabling the parallel collaborative running and debugging of multiple target applications, which is beneficial to improving debugging efficiency.

[0086] In this embodiment of the invention, after the debugging platform receives the mixed-screen video stream pushed in real time by the robot device, the debugging terminal can send a pull request to the debugging platform. The debugging platform can trim the mixed-screen video stream to obtain the screen video stream corresponding to the debugging terminal and send it to the debugging terminal. The debugging terminal can receive the screen video stream corresponding to the debugging terminal returned by the debugging platform and display the screen video stream.

[0087] In some embodiments, after the debugging terminal pulls and decodes the screen video stream of the corresponding virtual display, it can adaptively render and display the video stream based on bitrate, resolution, and frame rate, and supports direct operation of the display video stream by peripheral devices. The current remote debugging application can be selected for synchronous viewing to understand the current device operating status.

[0088] This invention dynamically creates and manages virtual displays corresponding to each target debugging object, enabling each target debugging object to have an independent running screen stream display. This overcomes the limitations of physical screens and allows multiple people to debug different target debugging objects in parallel without interfering with each other, which is beneficial to improving the debugging efficiency of robot equipment.

[0089] In one optional embodiment, the step of generating debugging instructions for the target debugging object in response to a user's debugging operation on the target debugging object and sending them to the robot device through the debugging platform may specifically include: Step S41: In response to the user's debugging operation on the target debugging object, determine the input event corresponding to the debugging operation; Step S42: The input event is converted into a debugging instruction based on a multi-person collaborative remote debugging protocol format and sent to the robot device through the debugging platform; wherein, the header of the multi-person collaborative remote debugging protocol format includes a source address, a destination address, control bits, a cyclic redundancy checksum, a timestamp, and a unique identifier; the unique identifier includes an operation timestamp, the display ID of the virtual display screen corresponding to the debugging end, and a randomly generated string; wherein, the display ID of the virtual display screen corresponding to the debugging end is obtained by the debugging end from the debugging platform.

[0090] In some embodiments, users can perform debugging operations on the target application through peripherals such as keyboards and mice connected to the control terminal; the debugging terminal can receive input events such as keyboard and mouse clicks and touch, convert the input events into debugging instructions based on the multi-person collaborative remote debugging protocol format, and send them to the robot device through the debugging platform.

[0091] In some embodiments, the multi-user collaborative remote debugging protocol is a custom lightweight remote debugging control protocol. The protocol uses the ProtoBuff serialization protocol to define lightweight gRPC (Remote Procedure Call) negotiation and interaction signaling, and predefines control bits (including remote control commands, text input, touch swipe commands, long press commands, etc.) and control data (remote control KeyCode values, text input content, touch trajectory point data, long press coordinate data, etc.). The format of the multi-user collaborative remote debugging protocol can be shown in the table below: Table 1

[0092] Among them, the source address indicates where the data comes from, and the destination address indicates where the data goes; ControlID represents a unique identifier for each debug command sent, which can be composed of the display ID of the virtual display screen corresponding to the debug terminal, the operation timestamp, and a randomly generated string; the control bit indicates the type of control to be executed, which can include screen switching, resolution switching, remote control buttons, text input, touch swiping, long press, etc.; the control data represents the details of the content to be operated, and different control bits have different data content, such as screen ID, whether the resolution is increased, remote control key value, input string content, left and right swipe touch trajectory points, and long press coordinates, etc.; CRC (Cyclic Redundancy Check) is a checksum generated by the sending end for the header data and the data part, which is then filled and verified at the receiving end.

[0093] In some embodiments, after the robot device creates virtual displays corresponding to each target application, the robot device can report the mapping relationship between the display IDs of each virtual display and the debugging terminal to the debugging platform. The debugging terminal can send a request to the debugging platform to obtain the display ID of the virtual display corresponding to the debugging terminal, thereby generating debugging instructions based on the multi-person collaborative remote debugging protocol format.

[0094] In one optional embodiment, receiving the screen video stream corresponding to the debugging terminal returned by the debugging platform and displaying the screen video stream may specifically include: Step S51: Decode the screen video stream corresponding to the debugging terminal returned by the debugging platform to obtain the video encoding time; Step S52: If the time difference between the video encoding time and the current decoding time is greater than a preset difference threshold, a screen lock command is generated. Step S53: Lock the debugging screen based on the screen lock command.

[0095] The synchronous display of various virtual displays on the robot device screen may be delayed or out of sync with the actual operation, leading to problems of chaotic multi-person parallel debugging and collaboration.

[0096] This invention introduces a time-based virtual display synchronization mechanism. When decoding the screen video stream returned by the debugging platform at the debugging end, the video encoding time is obtained and compared with the current decoding time. If the time difference between the video encoding time and the current decoding time is greater than a preset difference threshold (e.g., 350ms), it indicates that the screen video stream delay is too long. A screen lock command is then generated, and the debugging end screen is temporarily locked based on the screen lock command to avoid accidental operation. If the delay persists for an extended period, the network, bitrate, and frame rate can be adjusted adaptively to meet the requirements of real-time remote video stream preview and display with low latency.

[0097] In an alternative embodiment, after displaying the screen video stream, the process may further include: Step S61: In response to the resolution adjustment operation, generate a resolution adjustment request for the screen video stream; Step S62: Send the resolution adjustment request to the debugging platform; the resolution adjustment request is used to instruct the debugging platform to determine the target debugging end that meets the resolution adjustment requirements, determine the first display screen ID of the virtual display screen corresponding to the target debugging end, and re-cut the mixed screen video stream based on the screen percentage coordinates corresponding to the first display screen ID to obtain the updated first screen video stream and send it to the debugging end. Step S63: Receive and display the updated first-screen video stream.

[0098] With a fixed resolution in the mixed-screen video stream, as the number of simultaneous remote collaborators increases, when the number exceeds four, the display area of ​​some applications will significantly decrease. (Refer to...) Figure 4 Application 3's size area is only 1 / 4 that of Application 1. At this point, if the debugger of Application 3 wants to improve the clarity of the screen video stream, they can request a resolution adjustment. With the consent of either the debugger of Application 1 or Application 2, the display screen encoding coordinates corresponding to the applications are swapped, allowing the output resolution of the virtual display to change dynamically, thus enabling the debugger to view the screen video stream with the updated resolution.

[0099] Resolution adjustment requests can include resolution increase requests and resolution decrease requests. In some embodiments, taking an example where a debugger wants to improve the clarity of the screen video stream, the debugger can perform a resolution increase operation on the debugger end. The debugger end can generate a resolution increase request for the currently displayed screen video stream and send it to the debug platform. The debug platform can identify a target debugger end that meets the resolution adjustment requirements (i.e., the screen video stream has a higher resolution). If the target debugger end agrees to adjust the resolution, the platform re-trimmes the mixed screen video stream based on the screen percentage coordinates corresponding to the first display ID of the virtual display screen corresponding to the target debugger end, obtaining an updated first screen video stream, which is then sent to the debugger end. The debugger end can receive and display the updated, higher-resolution first screen video stream.

[0100] This invention allows users to switch resolutions at different resolutions by clicking on the debugging end, enabling dynamic changes in the output resolution of the virtual display screen and thus improving the user's debugging experience.

[0101] In an alternative embodiment, after displaying the screen video stream, the process may further include: Step S71: In response to a viewing operation on the target screen video stream, a viewing instruction is generated; Step S72: Send the viewing instruction to the debugging platform; the viewing instruction is used to instruct the debugging platform to determine the second display ID corresponding to the target screen video stream, and to re-cut the mixed screen video stream based on the screen percentage coordinates corresponding to the second display ID to obtain the updated second screen video stream and send it to the debugging terminal; Step S73: Receive and display the updated second screen video stream.

[0102] In this embodiment of the invention, the debugging platform can trim the mixed video stream to obtain the screen video stream corresponding to each target application, and send it to any debugging terminal. This allows the screen video stream corresponding to the target application currently being remotely debugged by the debugging terminal, as well as the screen video stream corresponding to other target applications currently being remotely debugged, to be displayed on any debugging terminal. This enables users to view the total number of collaborative debugging operators and other remote debugging applications running on the virtual display screen on the debugging terminal.

[0103] Figure 6 This is a schematic diagram illustrating the debugging terminal operation application provided in an embodiment of the present invention. (Refer to...) Figure 6 During multi-person collaborative remote debugging, users can view the screen video stream of the application being remotely debugged, as well as the screen video streams of the applications being remotely debugged by other operators, on the debugging end.

[0104] Figure 7 This is a schematic diagram of a debugging terminal simulation provided in an embodiment of the present invention. (Refer to...) Figure 7 During multi-person collaborative remote debugging, users can view the screen video stream corresponding to the currently remotely debugged simulated terminal manager, as well as the screen video stream corresponding to the simulated terminal manager of other operators.

[0105] In some embodiments, users can click on the screen video streams of other applications on the graphical user interface displayed on the debugging terminal to view the running status of other remotely debugged applications. The debugging terminal can respond to the viewing operation for the target screen video stream by generating a viewing command and sending it to the debugging platform. The debugging platform can determine the second display ID corresponding to the target screen video stream, and based on the screen percentage coordinates corresponding to the second display ID, re-trimmed the mixed-screen video stream to obtain an updated second screen video stream, which is then sent to the debugging terminal. The debugging terminal can receive and display the updated second screen video stream.

[0106] The embodiments of the present invention allow users to view the total number of current operators, the screen stream corresponding to the target application being debugged, or switch to other operators' debugging screen streams for simultaneous viewing, thereby achieving dynamic management of screen video streams.

[0107] In some embodiments, the debugging terminal can establish remote control with the robot equipment, thereby enabling multiple people to operate simultaneously, preventing conflicts arising from multiple operations, and managing the video stream acquisition strategy for the display screen.

[0108] Figure 8 This is the third flowchart illustrating the robot remote debugging method provided in this embodiment of the invention. (Refer to...) Figure 8 This invention provides a method for remote debugging of a robot, which may specifically include the following steps: Step 801: Receive start commands from multiple debugging terminals for at least one target debugging object, and send the start commands to the robot device; the start commands are used to instruct the robot device to create virtual display screens corresponding to each target debugging object, and display the corresponding target debugging object on each virtual display screen; wherein, the target debugging object is the debugging object to be collaboratively debugged, and the virtual display screens are displayed on the same robot device screen; Step 802: Receive debugging instructions sent by multiple debugging terminals for at least one target debugging object, and send the debugging instructions to the robot device; the debugging instructions are used to instruct the robot device to display the debugging operation corresponding to the debugging instructions on the virtual display screen corresponding to the target debugging object, and uniformly record and encode the debugging operations displayed on each virtual display screen to obtain a mixed screen video stream and push it to the debugging platform; Step 803: Upon receiving a pull request from any debugging terminal, trim the mixed video stream to obtain the screen video stream corresponding to any debugging terminal and send it to the debugging terminal.

[0109] The remote debugging method for robots provided in this invention can be applied to a debugging platform. Multiple debugging terminals can simultaneously initiate start commands for at least one target application; the debugging platform can send the start commands for at least one target application to the robot device; after receiving the start commands for at least one target application, the robot device can create virtual displays corresponding to each target application and display the corresponding target application on each virtual display, thereby realizing the parallel collaborative operation and debugging of multiple target applications.

[0110] In this embodiment of the invention, different virtual displays are used to display different target applications. All virtual displays are displayed on the same screen, thereby overcoming the limitations of physical screens while enabling the parallel collaborative running and debugging of multiple target applications, which is beneficial to improving debugging efficiency.

[0111] In this embodiment of the invention, after receiving a streaming request from any debugging terminal, the debugging platform can trim the mixed video stream to obtain the screen video stream corresponding to that debugging terminal and send it to that debugging terminal. The debugging terminal can receive the screen video stream corresponding to that debugging terminal returned by the debugging platform and display the screen video stream.

[0112] This invention dynamically creates and manages virtual displays corresponding to each target debugging object, enabling each target debugging object to have an independent running screen stream display. This overcomes the limitations of physical screens and allows multiple people to debug different target debugging objects in parallel without interfering with each other, which is beneficial to improving the debugging efficiency of robot equipment.

[0113] In one optional embodiment, the step of trimming the mixed video stream to obtain the screen video stream corresponding to any debugging end may specifically include: Step S81: For any debugging terminal, obtain the screen percentage coordinates corresponding to the virtual display screen based on the virtual display screen corresponding to the debugging terminal; Step S82: Calculate the pixel coordinates and the width and height of the screen video stream corresponding to the debugging terminal based on the screen mixing resolution and screen mixing mode. Step S83: Based on the screen percentage coordinates corresponding to the virtual display screen, the pixel coordinates, and the width and height of the screen video stream corresponding to the debugging terminal, the mixed screen video stream is trimmed to obtain the screen video stream corresponding to any debugging terminal.

[0114] In some embodiments, the screen video stream corresponding to the corresponding virtual display ID can be recorded in units of screen percentage. The coordinates of the top left corner of the virtual display screen are recorded in the form of (left, top, right, bottom) to form a mapping relationship. Database fields may include robot device identifier, operator user identifier, display ID (i.e., screen ID) of the virtual display screen corresponding to the debugging application, display ID of the currently previewed virtual display screen, coordinate point, mixed screen video stream resolution, mixed screen mode identifier, etc.

[0115] The formula for obtaining the clipping region using coordinate points can be as follows: c = Fun2(s, r, m); Where 'c' represents the data required for the cropping area, which may include the actual coordinates of the top-left and right corners, as well as the cropped width and height; 's' is the screen ID corresponding to the application; 'r' is the mixed-stream resolution; and 'm' is the mixed-stream mode identifier. For example, by querying the corresponding coordinates through the screen ID, the actual pixel coordinates and screen stream width and height are calculated based on the mixed-stream resolution 'r' and the mixed-stream mode 'm'. This allows for the cropping and distribution of the mixed-stream video, enabling multi-user parallel collaborative operation. Different users see different screen video streams corresponding to different applications.

[0116] In one optional embodiment, after obtaining the screen video stream corresponding to any of the debugging terminals, the method may further include: adjusting the video parameters of the screen video stream corresponding to any of the debugging terminals according to the rendering requirements and network conditions of the debugging terminals, so as to send the screen video stream with adjusted parameters to the debugging terminals.

[0117] In some embodiments, the debugging platform can trim the mixed-screen video stream to obtain screen video streams corresponding to each debugging end and distribute them. During distribution, the debugging platform can intelligently adjust video parameters such as resolution, bitrate, and frame rate of the screen video stream according to the rendering requirements or network conditions of the debugging end, thereby distributing the parameter-adjusted screen video stream to the corresponding debugging end and achieving adaptive distribution of the screen video stream.

[0118] In one optional embodiment, receiving start instructions from multiple debugging terminals for at least one target debugging object and sending the start instructions to the robot device may specifically include: when multiple debugging terminals send debugging instructions for the same target debugging object, in each future time slice, granting any one debugging terminal access to the target debugging object and disabling other debugging terminals access to the target debugging object.

[0119] If multiple people simultaneously initiate remote debugging commands for the same application, conflicts may occur when inputting commands, leading to a chaotic debugging process and potentially causing data inconsistencies or application crashes.

[0120] Based on this, embodiments of the present invention manage the operation permissions of multiple users on the same application by introducing a mutex lock (to obtain exclusive operation rights, at which time other users will be temporarily prevented from operating on the application) or a time slice turn-around mechanism (dividing a period of time into several fixed-length time slices, in which only one user can control the application within each time slice). Thus, when the debugging platform receives debugging instructions for the same target application from multiple debugging terminals, the debugging platform can sequentially open the operation permissions of each debugging terminal to the target application in each future time slice. That is, in each future time slice, only one debugging terminal is allowed to operate on the target application, while the operation permissions of other debugging terminals to the target application are closed, thus avoiding input conflicts of instructions for the same application.

[0121] In one optional embodiment, in the multi-user mode, the debugging operator can request an increase in screen display resolution. The debugging platform can reorder the applications, update the coordinate point information corresponding to the debugging applications, and then recalculate and distribute the corresponding screen video streams. When a debugging operator wants to view the debugging screen streams of other operators, they can recalculate and crop the streams based on the screen IDs of the other operators' screen video streams, or broadcast the streams based on the already distributed screen streams, thereby switching to the screen stream with the corresponding screen ID in real time.

[0122] Figure 9 This is a timing diagram illustrating remote robot debugging provided in an embodiment of the present invention. (Refer to...) Figure 9 In some embodiments, remote robot debugging may include the following steps: Step 1: The robot device is powered on and running, generating a unique device code and establishing a long connection with the debugging platform. It also reports local MAC (Media Access Control Address), serial number, and other device information. At this time, the robot device is online. The operator uses the debugging terminal to request the robot device to start the background remote debugging service program, or enters a specific password or enters a specific interface on the robot device to wake up the background remote debugging service program.

[0123] Step 2: If the operator wants to debug the application's running status, they can enter the application package name to initiate an application debugging request; if they want to debug the command line, they can initiate a simulated terminal debugging request; if they want to debug the robot's peripheral devices, they can enter the IP address of the specified device to initiate a peripheral debugging request.

[0124] After receiving a debugging request, the robot device can dynamically create a virtual screen, launch the corresponding application on the virtual screen, report the mapping relationship between the virtual screen ID and the debugger, perform unified screen recording encoding on each display layer, dynamically generate mixed video streams of different screen resolution sizes of the application according to the number of screens, and push the video streams to the debugging platform.

[0125] Step 3: After receiving the streaming request from the debugging end, the remote debugging platform can query the corresponding screen ID and its corresponding percentage coordinates (left, top, right, bottom) based on the device identifier, trim and distribute the mixed stream, and return the screen stream corresponding to the device identifier to the debugging end. After decoding the screen stream, the debugging end can adaptively render and display it according to the resolution, bitrate, etc.

[0126] Step 4: Each operator manipulates the input of peripheral devices (including keyboard, mouse clicks, and touch) on the screen stream rendered and displayed on the debugging end. The input events can be converted into a custom lightweight remote debugging control protocol. After receiving the instructions based on the control protocol, the device end classifies them by virtual screen ID and constructs virtual button events or touch events to realize the simulated input of remote peripheral devices.

[0127] Step 5: On the debugging end, you can view the total number of current operators and other remote debugging applications running on the virtual screen; you can click on other applications to view the running status of other virtual screens in real time, and you can click to switch resolutions to render screen streams of different sizes.

[0128] The embodiments of the present invention have the following advantages: (1) Parallel collaborative debugging capability: It supports multiple people to operate at the same time, dynamically create and manage one or more virtual displays, and each person has their own independent running result screen stream display. The operation between multiple people will not interfere with each other, breaking the screen display limitation of remote debugging of devices with screens.

[0129] (2) Enhanced realism: Through dynamic management and distribution of screen streams, a native application experience is provided, closely resembling the actual feeling of local operation, thus enhancing the realism and intuitiveness of debugging. Among them, the terminal emulation manager saves the context environment before and after command execution, which can solve the problem that each instruction execution can only create a new process to run in the background without context.

[0130] (3) Flexibility and scalability: While the robot equipment is being debugged, it can also act as a host to control and debug peripheral equipment, thereby increasing the overall flexibility and debugging range of the system.

[0131] Figure 10 This is a schematic diagram of the robot remote debugging system provided in an embodiment of the present invention. (Refer to...) Figure 10 This invention provides a remote debugging system for robots, comprising: The debugging terminal 1010 is used to respond to the user's startup operation on the target debugging object, generate a startup command for the target debugging object, and send it to the robot device; The debugging platform 1020 is used to send the start command to the robot device; Robot device 1030 is used to create virtual display screens corresponding to each target debugging object, and to display the corresponding target debugging object on each virtual display screen; wherein, the target debugging object is the debugging object to be collaboratively debugged, and the virtual display screens are displayed on the same robot device screen; The debugging terminal 1010 is also used to respond to the user's debugging operation on the target debugging object, generate a debugging instruction for the target debugging object, and send it to the debugging platform; The debugging platform 1020 is also used to send the debugging instructions to the robot device; The robot device 1030 is also used to, upon receiving a debugging instruction for the target debugging object, display the debugging operation corresponding to the debugging instruction on the virtual display screen corresponding to the target debugging object, and uniformly record and encode the debugging operation displayed on each virtual display screen to obtain a mixed screen video stream and push it to the debugging platform. The debugging platform 1020 is also used to trim the mixed video stream when it receives a pull request sent by any debugging terminal, to obtain the screen video stream corresponding to any debugging terminal and send it to the debugging terminal. The debugging terminal 1010 is also used to receive and display the screen video stream sent by the debugging platform.

[0132] Remote debugging technology provides developers with an effective means to monitor program execution status in real time. By analyzing log information and execution results, problems encountered by programs running on remote devices can be quickly identified and resolved. However, for robotic devices with screens, no current remote debugging technology supports simultaneous debugging of interactive applications by multiple users. Even when developers are physically present, screen limitations force them to take turns operating the software. To address this technical challenge, this invention comprehensively considers the convenience, richness, and security of remote debugging, providing an efficient and secure multi-user collaborative debugging platform.

[0133] In some embodiments, the system architecture involves the robot device end-side, remote debugging and collaboration platform, remote control and debugging terminal and peripheral devices, mainly including key components such as virtual screen management module, fusion perception module, analog terminal management module, screen streaming media framework, ADB host module, collaboration service module, rendering module and reverse control adaptation module.

[0134] This invention, through the dynamic creation and management of virtual displays corresponding to each target debugging object, enables each target debugging object to have an independent running screen stream, overcoming the limitations of physical screens and achieving multi-person parallel collaborative debugging of different target debugging objects without interference, which is beneficial to improving the debugging efficiency of robot equipment.

[0135] Figure 11 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 11 As shown, the electronic device may include: a processor 1110, a communications interface 1120, a memory 1130, and a communications bus 1140, wherein the processor 1110, the communications interface 1120, and the memory 1130 communicate with each other through the communications bus 1140. The processor 1110 can call logical instructions in the memory 1130 to execute a robot remote debugging method. This method includes: upon receiving a start instruction for at least one target debugging object, creating virtual display screens corresponding to each target debugging object, and displaying the corresponding target debugging object on each virtual display screen; wherein the target debugging object is the debugging object currently to be collaboratively debugged, and the virtual display screens are displayed on the same robot device screen; upon receiving a debugging instruction for the target debugging object, displaying the debugging operation corresponding to the debugging instruction on the virtual display screen corresponding to the target debugging object, and uniformly recording and encoding the debugging operation displayed on each virtual display screen to obtain a mixed-screen video stream and pushing it to the debugging platform; the mixed-screen video stream is used to instruct the debugging platform, upon receiving a pull request from any debugging end, to trim the mixed-screen video stream to obtain a screen video stream corresponding to the any debugging end and send it to the any debugging end.

[0136] Furthermore, the logical instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the robot remote debugging method provided by the above methods. The method includes: upon receiving a start instruction for at least one target debugging object, creating virtual display screens corresponding to each target debugging object, and displaying the corresponding target debugging object on each virtual display screen; wherein the target debugging object is the debugging object to be collaboratively debugged, and the virtual display screens are displayed on the same robot device screen; upon receiving a debugging instruction for the target debugging object, displaying the debugging operation corresponding to the debugging instruction on the virtual display screen corresponding to the target debugging object, and uniformly recording and encoding the debugging operation displayed on each virtual display screen to obtain a mixed screen video stream and pushing it to the debugging platform; the mixed screen video stream is used to instruct the debugging platform to trim the mixed screen video stream upon receiving a pull request from any debugging end, to obtain a screen video stream corresponding to the debugging end and send it to the debugging end.

[0138] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the robot remote debugging method provided by the above methods. The method includes: upon receiving a start command for at least one target debugging object, creating virtual display screens corresponding to each target debugging object, and displaying the corresponding target debugging object on each virtual display screen; wherein the target debugging object is the debugging object to be collaboratively debugged, and the virtual display screens are displayed on the same robot device screen; upon receiving a debugging command for the target debugging object, displaying the debugging operation corresponding to the debugging command on the virtual display screen corresponding to the target debugging object, and uniformly recording and encoding the debugging operation displayed on each virtual display screen to obtain a mixed-screen video stream and pushing it to the debugging platform; the mixed-screen video stream is used to instruct the debugging platform to trim the mixed-screen video stream upon receiving a pull request from any debugging end, to obtain a screen video stream corresponding to the any debugging end, and sending it to the any debugging end.

[0139] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / ip>

Claims

1. A method for remote debugging of a robot, characterized in that, Applied to robotic devices, the method includes: Upon receiving a start command for at least one target debugging object, a virtual display screen corresponding to each target debugging object is created, and the corresponding target debugging object is displayed on each virtual display screen; wherein, the target debugging object is the debugging object to be collaboratively debugged, and the virtual display screens are displayed on the same robot device screen; Upon receiving a debugging instruction for the target debugging object, the debugging operation corresponding to the debugging instruction is displayed on the virtual display screen corresponding to the target debugging object, and the debugging operation displayed on each virtual display screen is uniformly screen-recorded and encoded to obtain a mixed-screen video stream, which is then pushed to the debugging platform. The mixed-screen video stream is used to instruct the debugging platform to trim the mixed-screen video stream upon receiving a pull request from any debugging end, thereby obtaining the screen video stream corresponding to any debugging end and sending it to that debugging end.

2. The method according to claim 1, characterized in that, The step of creating a virtual display screen corresponding to each target debugging object and displaying the corresponding target debugging object on each virtual display screen includes: For each target debugging object, after starting the target debugging object, obtain the task stack ID of the target debugging object; Create a virtual display screen and obtain the display screen ID of the virtual display screen; The task stack ID of the target debugging object is associated with the display ID of the virtual display screen to display the corresponding target debugging object on each virtual display screen.

3. The method according to claim 2, characterized in that, After creating a virtual display screen for each target debugging object and displaying the corresponding target debugging object on each virtual display screen, the process also includes: For each virtual display screen, the screen percentage coordinates corresponding to the virtual display screen are determined based on the display screen ID of the virtual display screen, the screen width ratio of the virtual display screen, and the total number of virtual display screens. The screen percentage coordinates corresponding to each virtual display screen are reported to the debugging platform; the screen percentage coordinates are used to instruct the debugging platform to determine the target virtual display screen corresponding to any debugging terminal when it receives a pull request from any debugging terminal, and to trim the mixed video stream based on the screen percentage coordinates corresponding to the target virtual display screen to obtain the screen video stream corresponding to any debugging terminal.

4. The method according to claim 2, characterized in that, The step of displaying the debugging operation corresponding to the debugging command on the virtual display screen corresponding to the target debugging object includes: Based on the display ID of the virtual display screen corresponding to the target debugging object, construct an input event corresponding to the debugging instruction carrying the display screen ID; Based on the display screen ID carried by the input event, the input event is forwarded to the virtual display screen corresponding to the display screen ID, so as to display the debugging operation corresponding to the debugging command on the virtual display screen.

5. The method according to claim 1, characterized in that, The resolution of the mixed video stream and the resource allocation of each target debugging object are determined based on the system resource usage; the virtual display screens corresponding to each target debugging object and the processes corresponding to each target debugging object are deleted after debugging is completed.

6. A method for remote debugging of a robot, characterized in that, Applied to the debugging end, the method includes: In response to a user's startup operation on a target debugging object, a startup command for the target debugging object is generated and sent to the robot device through the debugging platform; the startup command is used to instruct the robot device to create a virtual display screen corresponding to the target debugging object and display the corresponding target debugging object on the virtual display screen; In response to a user's debugging operation on the target debugging object, a debugging instruction for the target debugging object is generated and sent to the robot device through the debugging platform. The debugging instruction is used to instruct the robot device to display the debugging operation corresponding to the debugging instruction on the virtual display screen corresponding to the target debugging object, and to uniformly record and encode the debugging operations displayed on each virtual display screen to obtain a mixed screen video stream and push it to the debugging platform. The target debugging object is the debugging object to be collaboratively debugged, and the virtual display screens are displayed on the same robot device screen. Send a pull request to the debugging platform, receive the screen video stream corresponding to the debugging end returned by the debugging platform, and display the screen video stream.

7. The method according to claim 6, characterized in that, The step of responding to a user's debugging operation on the target debugging object, generating debugging instructions for the target debugging object, and sending them to the robot device through the debugging platform includes: In response to a user's debugging operation on the target debugging object, determine the input event corresponding to the debugging operation; The input events are converted into debugging instructions based on a multi-person collaborative remote debugging protocol format and sent to the robot device through the debugging platform; The header of the multi-person collaborative remote debugging protocol format includes a source address, a destination address, control bits, a cyclic redundancy checksum, a timestamp, and a unique identifier; the unique identifier includes an operation timestamp, the display ID of the virtual display screen corresponding to the debugging terminal, and a randomly generated string; The display ID of the virtual display screen corresponding to the debugging terminal is obtained by the debugging terminal from the debugging platform.

8. The method according to claim 6, characterized in that, Receiving the screen video stream corresponding to the debugging terminal returned by the debugging platform and displaying the screen video stream includes: The screen video stream corresponding to the debugging terminal returned by the debugging platform is decoded to obtain the video encoding time; If the time difference between the video encoding time and the current decoding time is greater than a preset difference threshold, a screen lock command is generated; Based on the screen lock command, the debugging screen is locked.

9. The method according to claim 6, characterized in that, After displaying the screen video stream, the process also includes: In response to a resolution adjustment operation, a resolution adjustment request for the screen video stream is generated; The resolution adjustment request is sent to the debugging platform; the resolution adjustment request is used to instruct the debugging platform to determine the target debugging end that meets the resolution adjustment requirements, determine the first display screen ID of the virtual display screen corresponding to the target debugging end, and re-cut the mixed screen video stream based on the screen percentage coordinates corresponding to the first display screen ID to obtain the updated first screen video stream and send it to the debugging end. Receive and display the updated first-screen video stream.

10. The method according to claim 6, characterized in that, After displaying the screen video stream, the process also includes: In response to a viewing operation targeting a video stream on a target screen, a viewing command is generated; The viewing instruction is sent to the debugging platform; the viewing instruction is used to instruct the debugging platform to determine the second display ID corresponding to the target screen video stream, and to re-cut the mixed screen video stream based on the screen percentage coordinates corresponding to the second display ID to obtain the updated second screen video stream and send it to the debugging terminal; Receive and display the updated second-screen video stream.

11. A method for remote debugging of a robot, characterized in that, Applied to a debugging platform, the method includes: The robot receives start commands from multiple debugging terminals for at least one target debugging object and sends the start commands to the robot device. The start commands instruct the robot device to create virtual display screens corresponding to each target debugging object and display the corresponding target debugging object on each virtual display screen. The target debugging object is the debugging object to be collaboratively debugged, and the virtual display screens are displayed on the same robot device screen. The system receives debugging instructions from multiple debugging terminals for at least one target debugging object, and sends the debugging instructions to the robot device. The debugging instructions are used to instruct the robot device to display the debugging operation corresponding to the debugging instructions on the virtual display screen corresponding to the target debugging object, and to uniformly record and encode the debugging operations displayed on each virtual display screen to obtain a mixed screen video stream and push it to the debugging platform. Upon receiving a pull request from any debugging terminal, the mixed video stream is trimmed to obtain the screen video stream corresponding to the debugging terminal and sent to the debugging terminal.

12. The method according to claim 11, characterized in that, The step of trimming the mixed-screen video stream to obtain the screen video stream corresponding to any debugging end includes: For any debugging terminal, obtain the screen percentage coordinates corresponding to the virtual display screen based on the virtual display screen corresponding to the debugging terminal; Calculate the pixel coordinates and the width and height of the screen video stream corresponding to the debugging terminal based on the screen mixing resolution and screen mixing mode. Based on the screen percentage coordinates corresponding to the virtual display screen, the pixel coordinates, and the width and height of the screen video stream corresponding to the debugging terminal, the mixed screen video stream is trimmed to obtain the screen video stream corresponding to any debugging terminal.

13. The method according to claim 11, characterized in that, After obtaining the screen video stream corresponding to any of the debugging terminals, the process further includes: Based on the rendering requirements and network conditions of any of the debugging terminals, adjust the video parameters of the screen video stream corresponding to any of the debugging terminals, so as to send the screen video stream with adjusted parameters to any of the debugging terminals.

14. The method according to claim 11, characterized in that, The step of receiving start commands from multiple debugging terminals for at least one target debugging object and sending the start commands to the robot device includes: When multiple debugging endpoints send debugging commands for the same target debugging object, within each future time slice, grant any one debugging endpoint access to the target debugging object and disable the other debugging endpoints' access to the target debugging object.

15. A remote debugging system for robots, characterized in that, The system includes: The debugging end is used to respond to the user's start operation for the target debugging object, generate a start command for the target debugging object, and send it to the robot device; A debugging platform is used to send the start command to the robot device; A robotic device is used to create virtual display screens corresponding to each target debugging object, and to display the corresponding target debugging object on each virtual display screen; wherein, the target debugging object is the debugging object to be collaboratively debugged, and the virtual display screens are displayed on the same robotic device screen; The debugging terminal is also used to respond to the user's debugging operation on the target debugging object, generate debugging instructions for the target debugging object, and send them to the debugging platform; The debugging platform is also used to send the debugging instructions to the robot device; The robot device is also used to, upon receiving a debugging instruction for the target debugging object, display the debugging operation corresponding to the debugging instruction on the virtual display screen corresponding to the target debugging object, and uniformly record and encode the debugging operations displayed on each virtual display screen to obtain a mixed screen video stream and push it to the debugging platform. The debugging platform is also used to trim the mixed video stream when it receives a pull request from any debugging terminal, to obtain the screen video stream corresponding to any debugging terminal, and send it to the debugging terminal. The debugging terminal is also used to receive and display the screen video stream sent by the debugging platform.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the robot remote debugging method as described in any one of claims 1-5, 6-10, or 11-14.

17. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot remote debugging method as described in any one of claims 1-5, 6-10, or 11-14.

18. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the robot remote debugging method as described in any one of claims 1-5, 6-10, or 11-14.

Images