Data transmission method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a data transmission method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: in response to a received screen capture instruction, capturing a plurality of screen images in the running process of a cloud real machine through proxy service; synthesizing a video stream based on the plurality of screen images; and in response to a received video stream acquisition request, sending the video stream. According to the application, the fluency and the real-time performance of the mobile phone picture in the cloud measurement platform can be improved.

Description

Technical Field

[0001] The present application relates to computer network communication technology, and in particular to a data transmission method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] The cloud testing platform can display the screen image of the cloud real machine it provides through a web page. At the same time, users can remotely control the cloud real machine by operating the cloud real machine screen. In order to improve the realism of user operations, the web page is required to obtain the screen image of the cloud real machine in real time, and the displayed image must also be smooth. Summary of the invention

[0003] The embodiments of the present application provide a data transmission method, device, electronic device, computer-readable storage medium and computer program product, which can improve the fluency and real-time performance of mobile phone screens in a cloud testing platform.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a data transmission method, the method comprising:

[0006] In response to the received screenshot instruction, multiple screen images of the cloud real machine running process are captured through the proxy service;

[0007] synthesizing a video stream based on the multiple screen images;

[0008] In response to the received video stream acquisition request, the video stream is sent.

[0009] The present application provides a data transmission device, including:

[0010] The capture module is used to capture multiple screen images during the operation of the cloud real machine through the proxy service in response to the received screenshot instruction;

[0011] A synthesis module, used for synthesizing a video stream based on the multiple screen images;

[0012] The sending module is used to send the video stream in response to the received video stream acquisition request.

[0013] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0014] A memory for storing computer executable instructions;

[0015] The processor is used to implement the data transmission method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0016] An embodiment of the present application provides a computer-readable storage medium storing a computer program or a computer-executable instruction for implementing the data transmission method provided in the embodiment of the present application when executed by a processor.

[0017] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the data transmission method provided in the embodiment of the present application is implemented.

[0018] The embodiments of the present application have the following beneficial effects:

[0019] After receiving the screenshot instruction, multiple screen images of the cloud real machine are captured through the proxy service, and the video stream is synthesized. After receiving the video stream acquisition request, the video stream is sent. Compared with the method of transmitting the captured screen image by the proxy service in the related technology, the number of data transmissions is reduced, and the delay time in the transmission process is reduced, thereby improving the efficiency of data transmission, and then being able to improve the fluency and real-time performance of the mobile phone screen in the cloud testing platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the architecture of the data transmission system provided in the embodiment of the present application;

[0021] Figure 2 is a schematic diagram of the structure of an electronic device for implementing a data transmission method provided in an embodiment of the present application;

[0022] Figure 3 It is a flowchart of a data transmission method provided in an embodiment of the present application;

[0023] Figure 4 This is a schematic diagram of a first process of synthesizing a video stream provided in an embodiment of the present application;

[0024] Figure 5 It is a schematic diagram of a process for obtaining a media data stream provided in an embodiment of the present application;

[0025] Figure 6 It is a schematic diagram of the principle of the cloud testing platform provided in the embodiment of the present application;

[0026] Figure 7 is a schematic diagram of the media data flow provided by an embodiment of the present application;

[0027] Figure 8 is a schematic diagram of point-to-point transmission provided in an embodiment of the present application;

[0028] Fig. 9 is a schematic diagram of a second process of synthesizing a video stream provided in an embodiment of the present application;

[0029] Fig.10It is a schematic diagram of the principle of establishing a point-to-point connection provided by an embodiment of the present application;

[0030] Fig.11 It is a schematic diagram of the principle of synthesizing video streams on an iOS terminal device provided in an embodiment of the present application.

[0031] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0033] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0034] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0035] In the embodiments of the present application, the terms "module" and "unit" refer to a computer program or a part of a computer program with a predetermined function, and work together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0036] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0037] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0038] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0039] 1) Cloud testing platform: Cloud testing platform is a cloud-based service that mainly provides automated software testing services. The cloud platform deploys and uses testing tools and testing equipment to meet the full-cycle testing needs of software and system functions, compatibility, performance, security, etc. For example, the cloud testing platform can automatically run scripts, or provide remote rental of real machine services for manual testing, or real people and real machines can be used for testing.

[0040] 2) Cloud real machine: a real cloud mobile phone provided on the cloud testing platform, for example, used to run the user's automated running scripts, or remotely rented for manual testing, etc.

[0041] 3) WebSocket, a full-duplex communication protocol based on the Transmission Control Protocol, is a protocol for full-duplex communication on a single TCP (Transmission Control Protocol) connection, supporting real-time data transmission. Through WebSocket, a persistent connection is established between the client and the server to transmit data in real time. For example, WebSocket can be used in the cloud testing platform to transmit data such as operation instructions or parameters in real time.

[0042] 4) Web Real-Time Communication (WebRTC): WebRTC is a technology that supports real-time audio and video communication between browsers. With WebRTC, you can conduct real-time audio and video calls or video conferences directly in the browser. For example, by establishing real-time web communication between two terminals, real-time audio or video transmission between terminals can be achieved.

[0043] 5) Pixel bitmap (CGImageRef), CGImageRef is part of the Core Graphics framework (CoreGraphics is a graphics processing framework used to process graphics and image data on the iPhone Operating System (iOS) and macOS), and is a reference to image data. It is usually used to create bitmap images, such as User Interface Image (UIImage). Through CGImageRef, you can get information such as the width and height of the image. For example, load a picture from the local and display it on the interface of an iOS application. You can use a UIImage object to load a picture file, then convert it to a CGImageRef object, and finally draw it to the screen through a drawing context.

[0044] 6) Pixel buffer (CVPixelBufferRef) is a memory buffer used to store image data. It is a core data type in the Core Video framework, used to represent the pixel data of an image. CVPixelBufferRef can store image data of different formats and resolutions, and is commonly used in graphics processing, video editing, and real-time video communication. For example, to process a picture, you first need to load the picture into the application. You can use a pixel buffer to store the pixel data of this picture. Use the Application Programming Interface (API) of Core Video or other graphics processing libraries to rotate, scale, filter, and perform other operations on this picture. After the operation is completed, save the modified picture locally or upload it to the server.

[0045] 7) Sample buffer (CMSampleBufferRef), a memory buffer used to store multimedia sample data. It is a core data type in the Core Media framework, used to represent the sampling of audio, video, or combined audio and video data. CMSampleBufferRef can store audio and video data of different formats and sampling rates, and is commonly used in multimedia processing, audio and video editing, and real-time multimedia communications. For example, to process a video, you first need to load the video file into the application. Use the sample buffer to store the sample data of the video, and use the API of Core Media (CoreMedia) or other video processing libraries to edit the video, add effects, and other operations. After the operation is completed, save the modified video locally or upload it to the server.

[0046] 8) Web Driver Agent (WDA), an automated testing framework for developers that allows developers to automate testing of iOS applications through the WebDriver protocol. WDA is an automation framework based on Safari and can be integrated with Xcode Test (XCTest) for more complex user interface (UI) testing.

[0047] During the research process, the applicant found that the relevant technology generally uses WDA to capture the picture from the mobile phone interface to the screen, and then the proxy server (Agent) pulls the picture through the Hyper Text Transfer Protocol (HTTP) protocol and pushes the picture to the browser of the cloud testing platform. The browser receives the picture stream and displays it to the user through the canvas tag.

[0048] The applicant found that the relevant technology still has the following technical problems in the actual application process:

[0049] 1. The mobile phone screen in the cloud testing platform has low fluency and poor real-time performance;

[0050] 2. The captured screen images need to be forwarded multiple times through the proxy service, and the browser processing performance is low, resulting in image delays.

[0051] The embodiments of the present application provide a data transmission method, device, electronic device, computer-readable storage medium and computer program product, which can improve the fluency and real-time performance of mobile phone screens in a cloud testing platform.

[0052] See also Figure 1 , Figure 1 1 is a schematic diagram of the architecture of the data transmission system provided by the embodiment of the present application. To support an application, in the system 10, the test terminal 100 and the cloud real machine (cloud real machine 400-1 and cloud real machine 400-2 are shown as examples) are connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. The cloud real machines (400-1 and 400-2) are provided with a proxy service, which can capture the screen images of the cloud real machines (400-1 and 400-2).

[0053] The graphical interface 110 in the test terminal 100 is used to display the cloud real machines in the cloud testing platform for the tester to select the cloud real machine. After the cloud real machine (such as 400-1 and 400-2) is selected, the test terminal 100 sends a screenshot instruction to the selected cloud real machine through the server 200.

[0054] The cloud real machine (400-1 and 400-2) is used to capture multiple screen images during the operation of the cloud real machine (400-1 and 400-2) through the proxy service after receiving the screenshot instruction, and synthesize the video stream.

[0055] The test terminal 100 is also used to send a video stream acquisition request to the cloud real machine (400-1 and 400-2), and receive the video stream sent by the cloud real machine (400-1 and 400-2).

[0056] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0057] The following describes exemplary applications of the electronic device provided in the embodiments of the present application. The electronic device provided in the embodiments of the present application can be implemented as various types of terminals such as laptop computers, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, and vehicle-mounted terminals, and can also be implemented as servers.

[0058] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device for implementing a data transmission method provided in an embodiment of the present application, the electronic device may be a terminal or a server, Figure 2 The electronic device 500 shown includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 540 is not described in detail. Figure 2 Various buses are labeled as bus system 540 .

[0059] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0060] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0061] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices that are physically remote from the processor 510.

[0062] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0063] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0064] The operating system 551 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and process hardware-based tasks.

[0065] The network communication module 552 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 520. Exemplary network interfaces 520 include: Bluetooth, Wireless Fidelity (WiFi), and Universal Serial Bus (USB).

[0066] The presentation module 553 is used to enable presentation of information (eg, a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (eg, a display screen, a speaker, etc.) associated with the user interface 530 .

[0067] The input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 532 and translate the detected inputs or interactions.

[0068] In some embodiments, the data transmission device provided in the embodiments of the present application can be implemented in software. Figure 2 The data transmission device 555 stored in the memory 550 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: an interception module 5551, a synthesis module 5552 and a sending module 5553. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0069] In other embodiments, the data transmission device provided in the embodiments of the present application can be implemented in hardware. As an example, the data transmission device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data transmission method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0070] The data transmission method provided by the embodiment of the present application will be described in combination with the exemplary application and implementation of the electronic device provided by the embodiment of the present application. As mentioned above, the electronic device implementing the data transmission method of the embodiment of the present application can be a terminal, a server, or a combination of the two.

[0071] Taking the electronic device as a cloud real machine as an example, the cloud real machine can be a terminal, and the data transmission method of the embodiment of the present application is described, see Figure 3 , Figure 3 is a flow chart of the data transmission method provided in the embodiment of the present application, which will be combined with Figure 3 Steps 101 to 103 are shown for explanation.

[0072] In step 101, in response to a received screenshot instruction, a plurality of screen images of the cloud real machine during operation are captured through a proxy service.

[0073] In some embodiments, the cloud real machine can be an iOS terminal device, and the test terminal can be a computer device. The user selects the cloud real machine in the browser interface of the computer device (cloud testing platform interface). After selecting the cloud real machine, the computer device immediately sends a screenshot instruction to the proxy server (a server that controls and captures terminal devices in the cloud testing platform and can communicate with the cloud real machine). The proxy server forwards the screenshot instruction to the iOS terminal device. The screenshot instruction starts WDA, and WDA captures the screen image.

[0074] It should be noted that when WDA captures screen images, it directly uses the XCTest framework (XCTest is a framework provided for developers to write and run automated tests) from the bottom layer of WDA to capture images. The specific process of capturing images is to obtain the binary data of the screen image, and then compress it into an image based on the binary data. The image can be compressed into images of different formats. The choice of specific formats depends on application requirements and usage scenarios. The image format can be Joint Photographic Experts Group (JPEG), Portable Network Graphics (PNG), Graphics Interchange Format (GIF), Bitmap (BMP), etc. Different image formats are suitable for different application scenarios and requirements. In the process of compressing binary data into images, the image size can be reduced by more than 50% by setting parameters such as image quality and thumbnail ratio, thereby reducing the amount of video data for subsequent synthesis, thereby increasing the transmission speed and further improving the fluency and real-time performance of the mobile phone screen in the cloud testing platform.

[0075] In some embodiments, before receiving the screenshot instruction, the screenshot instruction is received through a WebSocket connection. A WebSocket connection can be established between the cloud real machine and the test terminal through a proxy server, a WebSocket connection can be established between the test terminal and the proxy server, and a WebSocket connection can be established between the cloud real machine and the proxy server. The screenshot instruction can be received by the cloud real machine through the WebSocket connection. A WebSocket connection can also be established between the test terminal and the proxy server, and the cloud real machine and the proxy server are connected via a USB data cable. The purpose of connecting the cloud real machine to the proxy server via a USB data cable is that the proxy server needs to call the WDA in the cloud real machine via the USB data cable to capture the screen image.

[0076] The test terminal sends the screenshot command to the proxy server, and the proxy server forwards the screenshot command to the cloud real machine. After receiving the screenshot command, the cloud real machine captures multiple screen images during the operation of the cloud real machine through WDA.

[0077] In some embodiments, the above-mentioned capturing of multiple screen images during the operation of the cloud real machine through a proxy service can be achieved through the following technical solution: according to the screenshot frequency of the screenshots, multiple screen images are periodically captured through the proxy service during the operation of the cloud real machine.

[0078] As an example, different screenshot frequencies can be dynamically set according to changes in network conditions or user needs. In the embodiment of the present application, the frame rate of the synthesized video stream can be modified by setting the screenshot frequency. For example, the screenshot frequency can be set to capture images 60 times per second according to actual conditions.

[0079] For example, the screenshot frequency is set to capture images 60 times per second according to actual conditions. Then, according to the screenshot frequency of 60 times per second, multiple screen images are periodically captured with 16.7 microseconds as a cycle during the operation of the cloud real machine through WDA.

[0080] Through the above technical solution, the frame rate of the composite video stream can be modified by setting the screenshot frequency, and the method of capturing the screen image will not stop capturing the screen after being disturbed. All displayed contents on the screen will be captured to ensure the integrity of the picture.

[0081] Continue to see Figure 3 , and the description will continue with step 101 above.

[0082] In step 102, a video stream is synthesized based on a plurality of screen images.

[0083] In some embodiments, see Figure 4 , Figure 4 is a schematic diagram of a first process of synthesizing a video stream provided in an embodiment of the present application, Figure 3 The illustrated step 102 may be implemented by following the steps 1021 to 1022, which are described in detail below.

[0084] In step 1021, the image formats of the plurality of screen images are converted into a target format to obtain a plurality of target images.

[0085] As an example, in the embodiment of the present application, the image formats of multiple screen images can be obtained by checking the file extensions. Different image formats have different file extensions, for example, JPEG files have .jpg or .jpeg extensions, PNG files have .png extensions, GIF files have .gif extensions, etc.

[0086] The above-mentioned conversion of the image formats of multiple screen images into a target format to obtain multiple target images can be achieved through the following technical solution: for each screen image among the multiple screen images, the following processing is performed: the binary data of the screen image is determined according to the image format; the binary data is decoded to obtain the image information of the screen image; according to the image information, a target format bitmap is obtained; according to the target format bitmap, a target image of the target format is obtained.

[0087] For example, the image format of the screen image is JPEG format. According to the image format being JPEG format, the JPEG binary data corresponding to the screen image can be determined, and then the JPEG binary data source can be obtained according to the JPEG binary data. The original image information of the screen image can be restored by decoding the JPEG binary data source, and the image information includes pixel data, color space information, resolution, ICC color management information, other metadata, etc. The target format bitmap is constructed according to the image information, and the target format bitmap can be a pixel bitmap. The target image of the target format is constructed according to the information of the pixel bitmap, that is, the pixel picture object, and the width and height, pixel format type and compatible type of the pixel picture object are set according to the information of the pixel bitmap. For example, the cloud real machine is an iOS mobile phone. A CGIma geRef bitmap (pixel bitmap) is created according to the image information, and the pixel data of CGImageRef is copied to CVPixelBufferRef (target image). At the same time, the properties of CVPixelBufferRef are set according to the color space, pixel format and other information in the CGImageRef bitmap information.

[0088] In step 1022, multiple target images are synthesized into a video stream.

[0089] As an example, the above-mentioned synthesis of multiple target images into a video stream can be achieved through the following technical solution: synthesizing the video stream according to the multiple target images, the timestamp information of the multiple target images, and the format description information of the multiple target images.

[0090] For example, the current time of the system is obtained as the display time of the video frame (target image) as the timestamp information. Since the final synthesized video stream is the video stream, the duration of the video (the length of time the video takes from the beginning to the end) needs to be set to invalid, and the decoding timestamp is not required. The format description information corresponding to the target image is obtained. The format description information includes: codec type (such as H.264, HEVC, VP9, ​​etc.), video frame size and bit rate, color space information, frame rate, and image direction. After obtaining multiple target images, the multiple target images are sorted according to the timestamp information of each target image, and then the sorted multiple target images are synthesized according to the format description information into a video stream that can be transmitted by the communication connection method used.

[0091] For example, if the cloud real machine is an iOS phone, CVPixelBufferRef is added to CMSampleBuffer as the data of the sampling buffer. According to CVPixelBufferRef, a timing information object and a video format description object are created, and the properties of CMSampleBufferRef are set through the timing information object and the video format description object. Then, according to the communication connection method used to transmit the data stream, the video data stored in CMSampleBufferRef is converted into a video stream that can be transmitted by the communication connection used. For example, if the communication connection used is a WebRTC connection, the video data stored in CMSampleBufferRef is encoded into a video frame object required by the WebRTC framework (Real-Time Communication Video Frame (RTCVideoFrame) video stream).

[0092] In some embodiments, the above-mentioned synthesis of video streams based on multiple screen images can also be achieved through the following technical solutions: deduplication processing is performed on multiple screen images to obtain multiple non-repetitive screen images; based on multiple non-repetitive screen images, the video stream is synthesized.

[0093] As an example, before storing a newly captured screen image, the newly captured screen image is compared with the last stored image. The pixel data of the two images can be compared to determine whether the two images are the same. If the pixel data of the two images are the same, the newly captured screen image is determined to be a duplicate image and deleted. If the pixel data of the two images are different, the newly captured screen image is determined to be a non-duplicate image, saved and used as a comparison image for the next newly captured screen image. A similarity threshold can also be set to define the similarity between the two images. If the similarity exceeds the similarity threshold, the newly captured screen image is determined to be a duplicate image and deleted. If the similarity does not exceed the similarity threshold, the newly captured screen image is determined to be a non-duplicate image, saved and used as a comparison image for the next newly captured screen image. In this way, multiple non-duplicate screen images can be obtained. The above-mentioned implementation method of synthesizing video streams based on multiple non-duplicate screen images can refer to steps 1021 to 1022, which will not be repeated here.

[0094] Through step 1021 to step 1022, the obtained screenshots can be deduplicated, duplicate images can be removed, and non-duplicate images can be synthesized into a video stream in a target format. The video stream does not contain audio information, which reduces the amount of transmitted data and increases the transmission speed. By transmitting the video stream, the smoothness and real-time performance of the mobile phone screen in the cloud testing platform can be further improved.

[0095] Continue to see Figure 3 , and the description continues with step 102 above.

[0096] In step 103, in response to the received video stream acquisition request, the video stream is sent.

[0097] In some embodiments, the sending of the video stream can be achieved through the following technical means: sending the video stream through a point-to-point connection.

[0098] As an example, a point-to-point connection generally refers to a connection method established directly between two devices without going through any intermediate device or server. Sending a video stream through the point-to-point connection can improve the transmission efficiency of the video stream.

[0099] In some embodiments, a point-to-point connection can be established through the following technical solution, which is specifically described below: sending a first connection parameter of the cloud real machine; receiving a second connection parameter returned based on the first connection parameter; and establishing a point-to-point connection based on the first connection parameter and the second connection parameter.

[0100] As an example, in order to send the first connection parameter of the cloud real machine and receive the second connection parameter returned based on the first connection parameter, before establishing the point-to-point connection, a WebSocket connection is established between the cloud real machine and the signaling server and between the test terminal and the signaling server, respectively, as described in detail below. The cloud real machine and the test terminal use a client library or tool that supports WebSocket, such as the WebSocket application program interface in Java Script, the WebSocket client library in Python, etc., to create a WebSocket client object, and specify the WebSocket address of the signaling server, that is, the uniform resource locator (URL) of the WebSocket server, and initiate a connection request to the signaling server through the connection method of the WebSocket client object (such as the 'open()' method of WebSocket in JavaScript). Once the connection is successfully established, the cloud real machine and the test terminal can send messages to the signaling server and receive messages from the signaling server.

[0101] After establishing the WebSocket connection between the cloud real machine and the signaling server and between the test terminal and the signaling server, the cloud real machine sends the first connection parameter of the cloud real machine to the signaling server, and the signaling server forwards the first connection parameter of the cloud real machine to the test terminal. The first connection parameter includes the media parameter and network parameter of the cloud real machine. After receiving the first connection parameter of the cloud real machine, the test terminal configures according to the second connection parameter and sends the second connection parameter of the test terminal to the signaling server. The cloud real machine receives the second connection parameter of the test terminal forwarded by the signaling server, and configures according to the second connection parameter. The second connection parameter includes the media parameter and network parameter of the test terminal.

[0102] Media parameters include: codec (audio and video codecs that need to be agreed upon by the cloud device and the test terminal, such as VP8, VP9, ​​H.264, etc.), resolution (target resolution of the video stream, such as 1280x720), frame rate (frame rate of the video stream, such as 30fps), audio sampling rate (sampling rate of the audio stream, such as 44100Hz), number of audio channels (number of channels of the audio stream, such as mono or stereo), etc. Network parameters include: Network Address Translation (NAT) / firewall information (the cloud device and the test terminal need to understand the NAT and firewall of the other party in order to select the appropriate Session Traversal Utilities for NAT (STUN) / Traversal Using Relays around NAT (TURN) server), STUN / TURN server address (used to help both parties resolve each other's public Internet Protocol (IP) addresses and ports), Interactive Connectivity Establishment (ICE) candidates (both parties need to exchange ICE candidates for possible NAT penetration methods to establish connections), network quality (network bandwidth, latency, packet loss rate and other information), etc.

[0103] Based on the first connection parameter and the second connection parameter, a point-to-point connection is established, which can be based on the first connection parameter and the second connection parameter, to establish a point-to-point connection between the cloud real machine and the test terminal. Specifically, the point-to-point connection can be a WebRTC connection. After the cloud real machine and the test terminal are configured according to the second connection parameter and the first connection parameter respectively, a WebRTC connection between the cloud real machine and the test terminal is established using the user datagram protocol. A real-time audio and video communication connection based on WebRTC can be successfully established between the cloud real machine and the test terminal through the user datagram protocol.

[0104] In some embodiments, after obtaining the video stream of the cloud real machine desktop, the video stream is sent to the test terminal through a point-to-point connection. The sending process uses WebRTC connection technology to realize the active push of the cloud real machine and the active pull of the test terminal. For example, set the RTCVideoFrame video stream to the WebRTC track and wait for the test terminal to receive the video stream. After the test terminal receives the video stream, it can play the video stream on the media display interface.

[0105] The following is a detailed description of the process of pulling streams by terminal devices: First, the terminal (such as a computer, mobile phone, or dedicated video conferencing equipment) sends a video stream request to the server through the network. The request usually contains information about the video stream requirements, such as the terminal's IP address, port number, and the requested video stream type. Then, after the server receives the video stream request from the terminal, it will process it according to the content of the request, which may include verifying the terminal's request authority, finding and preparing the corresponding video stream data, etc. Then, the server sends the prepared video stream data back to the terminal, which usually involves data encoding, packaging, and transmission. The video stream data may be transmitted in the form of real-time streaming media, such as using Real-time Streaming Protocol (RTSP), Hypertext Transfer Protocol Streaming Video (HTTP-FLV), WebRTC, Dynamic Adaptive Streaming over HTTP (DASH), Hypertext Transfer Protocol Real-time Streaming Protocol (HTTP Live Streaming, HLS) and other protocols. Finally, after the terminal receives the video stream data returned by the server, it will decode and render it, and then play the video on the user's device. The implementation methods of active streaming by cloud real devices and active streaming by test terminals can refer to the process of streaming by terminal devices mentioned above.

[0106] Through step 103, the video stream corresponding to the cloud real machine screen image can be sent directly to the test terminal through a point-to-point connection, which only requires one network transmission, greatly improving the video stream transmission rate, thereby improving the smoothness and real-time performance of the mobile phone screen in the cloud testing platform.

[0107] In some embodiments, see Figure 5 , Figure 5 is a schematic diagram of a process for obtaining a media data stream provided in an embodiment of the present application. Figure 3 After step 103 shown, a media data stream may be obtained, which will be described in detail below in conjunction with steps 201 to 202.

[0108] In step 201, an audio stream corresponding to a video stream is obtained.

[0109] As an example, the cloud real machine can obtain the audio stream of the local microphone through the API. For example, if the cloud real machine is an iOS phone, the audio session application programming interface (AVAudioSession API) can be used to obtain the audio stream of the cloud real machine.

[0110] In step 202, the video stream and the audio stream are combined to obtain a media data stream.

[0111] As an example, a video stream and an audio stream are data-encapsulated to obtain a media data stream.

[0112] For example, the RTCVideoFrame video stream and audio stream are placed in the corresponding video track and audio track for data encapsulation to obtain a media data stream.

[0113] In some embodiments, the sending of the video stream can be implemented through the following technical solution: sending a media data stream.

[0114] As an example, after obtaining the media data of the cloud real machine device desktop, the media data stream is sent to the test terminal through a point-to-point connection. The sending process uses WebRTC connection technology to realize the cloud real machine's active push stream and the test terminal's active pull stream. For example, if the cloud real machine is an iOS mobile phone, the RTCVideoFrame video stream and audio stream are placed in the corresponding video track and audio track for data encapsulation to obtain the media data stream, and the media data stream video stream and audio stream are respectively set to the corresponding video track and audio track of WebRTC, waiting for the test terminal to receive the media data stream. After the test terminal receives the media data stream, it can play the media data stream in the media display interface.

[0115] In the embodiment of the present application, after receiving the screenshot command, multiple screen images are captured during the operation of the cloud real machine, and the frame rate of the synthesized video stream is modified by setting the screenshot frequency. The method of capturing the screen image will not stop capturing after being disturbed. All displayed contents on the screen will be captured to ensure the integrity of the picture. The acquired screenshots are deduplicated once, and audio information is not encapsulated when synthesizing the video stream, which reduces the amount of transmitted data. The audio stream and the video stream are set to the corresponding tracks for separate transmission, which further improves the transmission speed. The video stream is sent to the test terminal through a point-to-point connection, which only requires one network transmission, further improving the efficiency of data transmission. Through this application, the fluency and real-time performance of the mobile phone screen in the cloud testing platform can be improved.

[0116] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0117] The embodiments of the present application can be applied to sharing screen images between two terminals, and can also be applied to browsers sharing cloud real machine screen images in a cloud testing platform.

[0118] See also Figure 6 , Figure 6 It is a schematic diagram of the principles of the cloud testing platform provided in the embodiment of the present application.

[0119] In step 301, after the browser selects the cloud real machine, it establishes a WebSocket connection with the signaling server (waiting for receiving and sending the parameters necessary to establish a web page instant messaging connection), locks the mobile phone, and can lock the mobile phone as an iOS terminal device of the iOS operating system. After that, it establishes a WebSocket connection with the proxy server, and the browser forwards the command to start WDA through the proxy server. After the WDA is successfully started, it establishes a WebSocket connection with the signaling server (waiting for receiving and sending the parameters necessary to establish a web page instant messaging connection), and at the same time, it captures the screen picture at a fixed time (waiting for the web page instant messaging peer connection to be established and then performing video stream synthesis).

[0120] In step 302, the iOS terminal device sends local media parameters and network parameters to the browser through the signaling server. After the browser sets the relevant information, it also sends the local media parameters and network parameters to the iOS terminal device through the signaling server. After the iOS terminal device sets the relevant information, the point-to-point connection for web instant messaging is established.

[0121] In step 303, after the point-to-point connection is established, the WDA will be regularly captured to synthesize the video stream, and the system audio stream will be captured, and the data stream and the audio stream will be encapsulated to obtain the media data stream. The browser side pulls the media data stream as needed, transmits the screen image of the cloud real machine to the browser page, and renders it in the browser so that the user can remotely watch the screen image of the cloud real machine. The browser side plays the media data in real time to achieve the visual effect of the browser on the screen of the iOS terminal device.

[0122] In the embodiment of the present application, the user datagram protocol is used for data transmission. Figure 7 , Figure 7 Schematic diagram of the media data flow provided by the embodiment of the present application. Figure 7 As shown, the audio and video streams of the mobile phone system are transmitted to the browser via the user datagram protocol, and then rendered by the browser. The efficiency of the user datagram protocol in transmitting data is higher than the combination of the http protocol and the tcp / ip protocol.

[0123] In the embodiment of the present application, the media data stream can be transmitted through a point-to-point connection. Figure 8 , Figure 8 Schematic diagram of point-to-point transmission provided by the embodiment of the present application. Figure 8As shown, the communication between the browser and the terminal device needs to establish a point-to-point connection, and the terminal device screen directly transmits the media data stream to the browser through point-to-point transmission. Based on WebRTC technology, the terminal actively pushes the stream and the personal computer (PC) browser actively pulls the stream, directly obtains the media data stream of the terminal device desktop, and sends it to the browser point-to-point. In addition to transmitting video streams, WebRTC can also transmit audio streams. The timed screenshot function based on WDA shields some defects of iOS mobile phone screen capture itself (for example, if an iOS mobile phone wants to capture the system screen, it needs to start the screen recording (BroadCast) process, which exists in the mobile phone process through WebSocket communication (an additional communication process), and the intermediate links are: BroadCast process->Application holding WebRTC connection->PC browser->rendering; when BroadCast captures the screen, once the mobile phone screen is black, it will be interrupted, and it cannot be turned on at the remote end, so the mobile phone needs to be kept on, which seriously affects the life of the mobile phone screen; when BroadCast captures the screen, there will be a small red dot or red bar on the top, and once the user makes a mistake, the remote end cannot be restored; when BroadCast captures the screen, some system permission pop-ups cannot be captured, which will give the user a false death or freeze phenomenon at the remote end; it takes 3 seconds to wait after initiation, resulting in delay).

[0124] In the embodiment of the present application, the method of synthesizing the video stream is shown in Fig. 9 , Fig. 9 2 is a schematic diagram of a second process of synthesizing a video stream provided in an embodiment of the present application. Fig. 9 As shown, after the synthesis starts, proceed to step 401.

[0125] In step 401 , Joint Photographic Experts Group binary data is obtained, and the process proceeds to step 402 .

[0126] In step 402 , a Joint Photographic Experts Group binary data source is created, and the process proceeds to step 403 .

[0127] In step 403 , the Joint Photographic Experts Group binary data source is decoded. If the decoding fails, an exception is thrown. If the decoding succeeds, the process proceeds to step 404 .

[0128] In step 404 , a pixel bitmap is created from the Joint Photographic Experts Group binary data source. If the creation fails, an exception is thrown. If the creation succeeds, the process proceeds to step 405 .

[0129] In step 405, a pixel buffer is created to store image data, the width and height of the pixel buffer are set according to the bitmap information of the pixel bitmap generated in step 405, the pixel format type is set, the compatible type is set, and the pixel bitmap data is filled into the constructed pixel buffer through the canvas. If the creation fails, an exception is thrown. If the creation is successful, the process proceeds to step 406.

[0130] In step 406 , the timestamp information of the video frame is created, the current system time is obtained as the video frame display time, the duration is set to be invalid, and it is set that the decoding timestamp is not required, and the process proceeds to step 407 .

[0131] In step 407 , the format description information of the video stream is created, and according to the pixel buffer generated in step 406 , a video format description object (including description of the format of the encoded and decoded video and parameters such as pixels, width and height of the image) is created, and the process proceeds to step 408 .

[0132] In step 408, a sampling buffer video stream is created. The core basic object in the audio and video processing pipeline transmission, namely, the sampling buffer video stream, is created through the timestamp information and the video format description object. It should be noted that the sampling buffer video stream is essentially the data of the video stream stored in the sampling buffer. The video stream data needs to be transcoded into the target video stream according to the specific application in order to successfully create and complete the synthesis of the video stream. If the creation fails, an exception is thrown.

[0133] For example, if a video stream needs to be transmitted through a WebRTC connection, the video stream data stored in the sampling buffer is transcoded into a video frame object (RTCVideoFrame) required by the WebRTC framework, and then the video frame object is manually added to the video data acquisition source of WebRTC.

[0134] In the embodiment of the present application, a peer-to-peer connection (RTCPeerConnection) is established to establish a peer-to-peer communication and implement peer-to-peer screen projection. Fig.10 , Fig.10 This is a schematic diagram of the principle of establishing a point-to-point connection provided by an embodiment of the present application. The browser in the figure is a PC browser, and the mobile terminal is an iOS terminal device.

[0135] In step 501A, the browser sends the occupied device information to the signaling server, and the signaling server sends the information of initiating screen projection to the mobile terminal. While executing step 501A, step 501B is also executed. In step 501B, the browser sends the operation instruction of starting WDA to the proxy server, and the proxy server forwards the start instruction, operates the mobile terminal to start WDA, and WDA regularly captures the screen image of the mobile terminal. The mobile terminal synthesizes the video stream ready for projection based on the captured screen image, and then goes to step 502.

[0136] In step 502 , the mobile terminal sends a connection intention (ie, a connection request) to the signaling server. The connection intention includes the format of the video stream. The signaling server forwards the connection intention to the browser and the process proceeds to step 503 .

[0137] In step 503 , the browser sends a response to the signaling server. The response includes the format of the video stream that can be parsed by the browser. The signaling server forwards the response to the mobile terminal and the process proceeds to step 504 .

[0138] In step 504 , the mobile terminal and the browser respectively connect to the relay server, obtain their respective external IP addresses and port numbers, and then proceed to step 505 .

[0139] In step 505, the mobile terminal sends a candidate to the signaling server, the candidate includes the external IP address and port number of the mobile terminal, and the browser sends a candidate to the signaling server, the candidate includes the external IP address and port number of the browser. The signaling server forwards the candidate sent by the mobile terminal to the browser, and forwards the candidate sent by the browser to the mobile terminal, so that the mobile terminal and the browser can try to initiate a point-to-point connection. When the format, IP address and port number of the video stream are all negotiated in agreement, a point-to-point network connection will be truly established. The connection is successful, and the process proceeds to step 506.

[0140] In step 506, a peer-to-peer connection (point-to-point connection) is established between the browser and the mobile terminal.

[0141] The mobile terminal captures the screen images and merges them into a video stream, which is set to the point-to-point connection of WebRTC. If the mobile terminal system has audio, the audio stream is also set to the point-to-point connection of WebRTC, and the browser will automatically pull the stream to the local for rendering and playback.

[0142] It should be noted that after establishing a point-to-point connection in step 506 and transmitting the mobile terminal media data stream, the browser can send other instructions for operating the mobile terminal to the proxy server, and the proxy continues to forward the operating instructions to operate the mobile terminal. The screen image of the mobile terminal is always continuously transmitted to the browser end via the video stream.

[0143] In the embodiment of the present application, the process implemented on the iOS terminal device is shown in Fig.11 , Fig.11 It is a schematic diagram of the principle of synthesizing video streams on an iOS terminal device provided in an embodiment of the present application.

[0144] After receiving the start command, the Agent forwards the command to the iOS terminal device, and the iOS terminal device starts the WDA of the iOS terminal device; after starting, the WDA will take a screenshot; after starting, the WDA will also establish a WebRTC connection; the screen image captured by WDA is not in a video format that can be sent directly by WebRTC, and the captured screen image needs to be merged into a video frame object (ImageBuffer) and other video streams that can be transmitted by WebRTC; the video stream is set to the WebRTC track and waits for reading.

[0145] The following further describes an exemplary structure of the data transmission device 555 provided in the embodiment of the present application implemented as a software module. In some embodiments, for example Figure 2 As shown, the software modules stored in the data transmission device 555 of the memory 550 may include:

[0146] The capture module 5551 is used to respond to the received screenshot instruction and capture multiple screen images during the operation of the cloud real machine through the proxy service.

[0147] The synthesis module 5552 is used to synthesize the video stream based on the multiple screen images.

[0148] The sending module 5553 sends the video stream in response to the received video stream acquisition request.

[0149] In some embodiments, the capture module 5551 is also used to periodically capture multiple screen images through the proxy service during the operation of the cloud real machine according to the screenshot frequency.

[0150] In some embodiments, the synthesis module 5552 is further used to convert the image format of the multiple screen images into a target format to obtain multiple target images; and synthesize the multiple target images into a video stream.

[0151] In some embodiments, the synthesis module 5552 is also used to perform the following processing for each of the multiple screen images: determine the binary data of the screen image according to the image format; decode the binary data to obtain image information of the screen image; obtain a target format bitmap based on the image information; and obtain a target image in a target format based on the target format bitmap.

[0152] In some embodiments, the synthesis module 5552 is further used to synthesize a video stream according to the multiple target images, the timestamp information of the multiple target images and the format description information of the multiple target images.

[0153] In some embodiments, the sending module 5553 is further used to send the video stream via a point-to-point connection.

[0154] In some embodiments, the sending module 5553 is also used to send the first connection parameter of the cloud real machine; receive the second connection parameter returned based on the first connection parameter; and establish the point-to-point connection based on the first connection parameter and the second connection parameter.

[0155] In some embodiments, the capture module 5551 is further used to receive the screenshot instruction via a WebSocket connection.

[0156] In some embodiments, the sending module 5553 is further used to establish the WebRTC connection based on the first connection parameter and the second connection parameter using the User Datagram Protocol.

[0157] In some embodiments, the synthesis module 5552 is further used to obtain an audio stream corresponding to the video stream; and merge the video stream with the audio stream to obtain a media data stream.

[0158] In some embodiments, the sending module 5553 is also used to send the media data stream.

[0159] In some embodiments, the synthesis module 5552 is further used to perform deduplication processing on the multiple screen images to obtain multiple non-duplicate screen images; and synthesize a video stream based on the multiple non-duplicate screen images.

[0160] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the data transmission method described in the embodiment of the present application.

[0161] An embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the data transmission method provided by the embodiment of the present application.

[0162] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0163] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0164] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0165] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0166] In summary, after receiving the screenshot instruction, according to the embodiment of the present application, in the process of compressing the acquired binary data into an image, the picture quality, thumbnail ratio and other parameters are set to reduce the size of the image, thereby reducing the amount of video data and improving the transmission speed; the screenshot frequency can be set according to changes in network conditions or user needs to modify the frame rate of the synthesized video stream; and the method of capturing the screen image will not stop the screenshot after being disturbed, and all displayed content on the screen will be captured to ensure the integrity of the picture; the acquired screenshot is deduplicated once, and the audio information is not encapsulated when synthesizing the video stream, which reduces the amount of transmitted data and improves the transmission speed; the video stream is sent through a point-to-point connection, only one network transmission is required, and the udp protocol is used for transmission, and the transmission efficiency is significantly improved. The video synthesized by a small number of screenshots is clearer and more stable at the playback end after being transmitted through WebRT C, and there will be no image snowflake phenomenon; and the audio stream is obtained separately and set to be transmitted on a different track from the video stream, which can improve the transmission speed; finally, based on the above, the embodiment of the present application can improve the fluency and real-time performance of the mobile phone screen in the cloud testing platform.

[0167] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data transmission method, characterized in that: The method comprises: In response to the received screenshot instruction, multiple screen images of the cloud real machine running process are captured through the proxy service; synthesizing a video stream based on the multiple screen images; In response to the received video stream acquisition request, the video stream is sent.

2. The method according to claim 1, characterized in that: The multiple screen images captured during the operation of the cloud real machine through the proxy service include: According to the screenshot frequency, during the operation of the cloud real machine, multiple screen images are periodically captured through the proxy service.

3. The method according to claim 1, characterized in that The synthesizing a video stream based on the multiple screen images comprises: Converting the image formats of the plurality of screen images into a target format to obtain a plurality of target images; The multiple target images are synthesized into a video stream.

4. The method according to claim 3, characterized in that The converting the image formats of the plurality of screen images into a target format to obtain a plurality of target images comprises: For each of the plurality of screen images, the following processing is performed: Determining binary data of the screen image according to the image format; Decoding the binary data to obtain image information of the screen image; According to the image information, a target format bitmap is obtained; A target image in a target format is obtained according to the target format bitmap.

5. The method according to claim 3, characterized in that: The step of synthesizing the plurality of target images into a video stream comprises: A video stream is synthesized according to the multiple target images, the timestamp information of the multiple target images, and the format description information of the multiple target images.

6. The method according to claim 1, characterized in that The sending of the video stream comprises: The video stream is sent via a point-to-point connection.

7. The method according to claim 6, characterized in that The method further comprises: Sending a first connection parameter of the cloud real machine; receiving a second connection parameter returned based on the first connection parameter; The point-to-point connection is established based on the first connection parameter and the second connection parameter.

8. The method according to claim 7, characterized in that Before intercepting multiple screen images during the operation of the cloud real machine through the proxy service, the method further includes: Receiving the screenshot instruction via a WebSocket connection; The point-to-point connection is a WebRTC connection, and establishing the point-to-point connection based on the first connection parameter and the second connection parameter includes: The WebRTC connection is established based on the first connection parameter and the second connection parameter using the User Datagram Protocol.

9. The method according to claim 1, characterized in that: After synthesizing the video stream based on the multiple screen images, the method further includes: Obtaining the audio stream corresponding to the video stream; Merging the video stream and the audio stream to obtain a media data stream; The sending of the video stream comprises: The media data stream is sent.

10. The method according to claim 1, characterized in that The synthesizing a video stream based on the multiple screen images comprises: Performing deduplication processing on the multiple screen images to obtain multiple non-duplicate screen images; A video stream is synthesized based on the multiple non-repetitive screen images.

11. A data transmission device, characterized in that: The device comprises: The capture module is used to capture multiple screen images during the operation of the cloud real machine through the proxy service in response to the received screenshot instruction; A synthesis module, used for synthesizing a video stream based on the multiple screen images; The sending module is used to send the video stream in response to the received video stream acquisition request.

12. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions; A processor, configured to implement the data transmission method according to any one of claims 1 to 10 when executing the computer executable instructions stored in the memory.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the data transmission method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the data transmission method according to any one of claims 1 to 10 is implemented.

Images