High-performance and high-compatibility cloud mobile phone camera flow pushing method and device

Abstract

The invention provides a high-performance and high-compatibility cloud mobile phone camera stream pushing method and device, and the method comprises the steps: a cloud mobile phone sends a camera opening instruction and video collection information to a browser for applying to open a local equipment camera, and transmits the video collection information; video data are acquired according to the video acquisition information, a video track is acquired from the video data through navigator.media Devices. GetUserMedia, and the video track is called back to a browser; performing H.264 coding on the video track to obtain coded data; pushing the coded data to the cloud mobile phone frame by frame; the cloud mobile phone decodes the coded data and injects the coded data into the virtual camera of the cloud mobile phone, the compatibility is high, the delay is low, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video streaming, and in particular to a high-performance and highly compatible cloud phone camera streaming method and device. Background Art

[0002] The cloud phone service allows customers to access cloud phone devices through local devices such as PC, Android, iPhone, etc. In some scenarios (such as face recognition, code scanning login, live broadcast, etc. of cloud phone applications), customers need to push the real-time images captured by the local device camera to the cloud phone virtual camera. Since video encoding consumes a lot of hardware resources and needs to be compatible with different devices, systems and browsers, the video in the cloud phone is delayed. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a high-performance and highly compatible cloud phone camera streaming method and device with high compatibility and low latency to improve user experience.

[0004] In a first aspect, the present invention provides a high-performance and highly compatible cloud phone camera streaming method, comprising the following steps:

[0005] Step 1: The cloud phone sends a camera opening instruction and video acquisition information to the browser, wherein the video acquisition information includes: video width, video height and frame rate;

[0006] Step 2: Call navigator.mediaDevices.getUserMedia to open the local device camera and pass in the video acquisition information;

[0007] Step 3: Obtain video data according to the video acquisition information, obtain the video track MediaStreamTrack through navigator.mediaDevices.getUserMedia, and call back to the browser; perform H.264 encoding on the video track MediaStreamTrack to obtain encoded data;

[0008] Step 4: Push the encoded data frame by frame to the cloud phone;

[0009] Step 5: The cloud phone decodes the encoded data and injects it into the cloud phone virtual camera.

[0010] In a second aspect, the present invention provides a high-performance and highly compatible cloud phone camera streaming device, comprising:

[0011] In the preparation module, the cloud phone sends a camera opening instruction and video acquisition information to the browser, wherein the video acquisition information includes: video width, video height and frame rate;

[0012] Apply to open the module, call navigator.mediaDevices.getUserMedia to open the local device camera, and pass in the video acquisition information;

[0013] The encoding module obtains video data according to the video acquisition information, obtains the video track MediaStreamTrack through navigator.mediaDevices.getUserMedia, and calls back to the browser; performs H.264 encoding on the video track MediaStreamTrack to obtain encoded data;

[0014] The push module pushes the encoded data frame by frame to the cloud phone;

[0015] Decoding module, the cloud phone decodes the encoded data and injects it into the cloud phone virtual camera.

[0016] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:

[0017] High compatibility: compatibility is improved through the combination of multiple encoding schemes and degradation strategies;

[0018] High performance: Dynamically select the best-performing encoding and streaming based on the user's device environment;

[0019] Low latency: Real-time image acquisition and transmission are achieved to provide users with an interactive experience of the process.

[0020] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0022] Figure 1 is a flow chart of the method in Embodiment 1 of the present invention;

[0023] Figure 2 It is a schematic diagram of the structure of the device in the second embodiment of the present invention. DETAILED DESCRIPTION

[0024] The technical solution in the embodiment of the present application has the following general idea:

[0025] H.264: A video compression standard widely used in streaming media transmission and video storage, with high compression rate and excellent image quality.

[0026] WebCodecs: A high-performance Web API that provides direct access to device audio and video codecs, suitable for low-latency video processing scenarios.

[0027] WASM H.264 Encoder: A WebAssembly-based H.264 video encoder that supports efficient running of video encoding tasks in the browser.

[0028] WebAssembly (WASM): A binary instruction format designed to run complex applications in browsers with high performance.

[0029] VideoEncoderAPI

[0030] WebCodecs: Provides video encoding interfaces for hardware-accelerated and low-latency video encoding.

[0031] MediaStream: A Web API that represents real-time media data streams. It is widely used for the collection and transmission of audio and video streams.

[0032] MediaStreamTrackProcessor is a Web API that allows you to extract video or audio data streams frame by frame from MediaStreamTrack.

[0033] MediaStreamTrackGenerator is a Web API that is used to create writable tracks and write encoded or processed audio and video data into new tracks.

[0034] TransformStream is a Web API that supports the step-by-step processing and transformation of streaming data.

[0035] The MediaRecorder API is a web API for capturing media streams and encoding them into a specific format, such as H.264.

[0036] Worker is a multithreading technology provided by browsers that allows JavaScript to run outside the main thread (UI thread). It is suitable for processing time-consuming tasks without affecting the responsiveness of the page.

[0037] Encoding and streaming SDK: Select the video encoding scheme according to the environment. Collect the camera data of the user's local device in real time, encode it in H.264, and push the encoded data to the cloud phone.

[0038] Cloud phone: In the current solution, it is mainly responsible for receiving H.264 video stream data. After decoding, it is injected into the virtual camera of the cloud phone, so that the APP in the cloud phone can display the images collected by the local device in real time.

[0039] The main process is as follows:

[0040] 1. The user opens the APP camera in the cloud phone

[0041] 1. The user clicks on functions in the cloud phone app that require the use of the camera, such as taking photos, recording videos, scanning codes, live streaming, etc.;

[0042] 2. The cloud phone sends the camera opening command to the front-end encoding and streaming SDK

[0043] 1. The cloud phone monitors the operation of the APP opening the camera;

[0044] 2. The cloud phone sends a command to open the camera to the front-end encoding and streaming SDK to collect video information including: video width, video height, and frame rate;

[0045] 3. Apply to open the user's local device camera

[0046] 1. Call navigator.mediaDevices.getUserMedia API and pass in the width, height, and frame rate of the camera image to be captured, and ask the user to open the local device camera.

[0047] 2. At this time, the browser will automatically pop up a confirmation window, asking the customer to confirm whether to allow the local camera to be opened;

[0048] 4. Start collecting camera images

[0049] 1. After the user allows access to the camera, start capturing real-time images

[0050] The real-time data is captured and the video track MediaStreamTrack is called back to the encoding and streaming SDK through navigator.mediaDevices.getUserMediaAPI;

[0051] Select the video encoding scheme and encode the video in H.264;

[0052] The following encoding schemes are arranged in the order of performance first and compatibility second. When the first scheme that meets the conditions is detected, it is adopted and the detection process is terminated. Combined with environmental detection, a technical method of combining multiple schemes and gradually degrading is used to ensure a balance between compatibility and performance.

[0053] Solution 1: WebCodecs encoding solution: Use the VideoEncoderAPI provided by WebCodecs for hardware accelerated encoding.

[0054] Main steps of the program:

[0055] 1. Use MediaStreamTrackProcessor to extract the video stream from MediaStreamTrack as input;

[0056] 2. Create a TransformStream to process the video frame stream read in MediaStreamTrackProcessor;

[0057] 3. Perform H.264 encoding on the frame-by-frame data read from TransformStream through VideoEncoderAPI;

[0058] The solution supports the following conditions: the browser supports VideoEncoder API, the browser's VideoDecoderAPI supports H.264 encoding, and the browser supports MediaStreamTrackProcessor, MediaStreamTrackGenerator, and TransformStream.

[0059] Solution 2: MediaRecoder encoding solution: Use the MediaRecorder API to record MediaStream and automatically encode it to H.264 format;

[0060] Main steps of the program:

[0061] 1. Convert the collected camera video track MediaStreamStrack into a MediaStream video real-time stream;

[0062] 2. Call MediaRecoderAPI and set the recording format to H.264 to start recording the MediaStream video real-time stream;

[0063] 3. During the recording process, the video will be encoded in H.264 and the data of each video frame will be called back in real time;

[0064] The solution supports the following conditions: The browser supports MediaRecoderAPI and H.264 encoding.

[0065] Scheme 3: WASM encoding scheme:

[0066] Use the WebAssembly H.264 encoder to adapt to environments that do not support native encoding solutions;

[0067] Main steps of the program:

[0068] 1. Create a Worker thread;

[0069] 2. In the Workder, dynamically load the WASM H.264 encoder, and notify the main thread after loading is complete;

[0070] 3. The main thread sends the video frame data to the Worker;

[0071] 4. In the Worker, use the WASM H.264 encoder to encode and send the encoded video frame data to the main thread;

[0072] The solution supports the following conditions: Support for WebAssembly;

[0073] 5. Push the encoded H.264 data frame by frame to the cloud phone

[0074] 6. The cloud phone injects the virtual camera: the cloud phone decodes the encoded data and injects it into the cloud phone virtual camera; at this time, the user can see the real-time images collected locally in the cloud phone APP.

[0075] This solution supports low-latency real-time push of the user's local camera image to the application in the cloud phone; adapts to as many different customer operating environments as possible; and selects the encoding and streaming solution with the best performance based on the performance of the customer's device.

[0076] Embodiment 1

[0077] like Figure 1 As shown, this embodiment provides a high-performance and highly compatible cloud phone camera streaming method, comprising the following steps:

[0078] Step 1: The cloud phone sends a camera opening instruction and video acquisition information to the browser, wherein the video acquisition information includes: video width, video height and frame rate;

[0079] Step 2: Call navigator.mediaDevices.getUserMedia to open the local device camera and pass in the video acquisition information;

[0080] Step 3: Obtain video data according to the video acquisition information, obtain the video track MediaStreamTrack through navigator.mediaDevices.getUserMedia, and call back to the browser; perform H.264 encoding on the video track MediaStreamTrack to obtain encoded data;

[0081] Step 4: Push the encoded data frame by frame to the cloud phone;

[0082] Step 5: The cloud phone decodes the encoded data and injects it into the cloud phone virtual camera.

[0083] In this embodiment, preferably, the step 3 is specifically: obtaining video data according to the video acquisition information, obtaining the video track MediaStreamTrack of the video data through navigator.mediaDevices.getUserMedia, and calling back the browser; if the browser supports VideoEncoder API, H.264 encoding, MediaStreamTrackProcessor, MediaStreamTrackGenerat or and TransformStream, the video track MediaStreamTrack is encoded with H.264 using WebCodecs; if the browser supports MediaRecoder API and H.264 encoding, the video track MediaStreamTrack is encoded with H.264 using MediaRecoder; if the browser supports WebAssembly, the video track MediaStreamTrack is encoded with H.264 using WASM to obtain encoded data.

[0084] In this embodiment, preferably, the H.264 encoding of the video track MediaStreamTrack using WebCodecs is specifically: using MediaStreamTrackProcessor to extract the video stream from the video track MediaStreamTrack; creating TransformStream to process the video stream read from MediaStreamTrackProcessor to obtain frame-by-frame data; and performing H.264 encoding on the frame-by-frame data through VideoEncoderAPI.

[0085] In this embodiment, preferably, the H.264 encoding of the video track MediaStreamTrack using MediaRecoder is specifically: converting the video track MediaStreamStrack into a MediaStream video real-time stream; calling the MediaRecoderAPI, and setting the recording format to H.264, and starting to record the MediaStream video real-time stream; during the recording process, performing H.264 encoding on the video real-time stream, and calling back the data of each video frame in real time to obtain the encoded data.

[0086] In this embodiment, preferably, the H.264 encoding of the video track MediaStreamTrack using WASM is specifically as follows: creating a Worker thread; in the Worker thread, dynamically loading the WASM H.264 encoder, and after the loading is completed, notifying the main thread; the main thread sends the video track MediaStreamStrack to the Worker; in the Worker, encoding is performed using the WASM H.264 encoder, and the encoded video frame data is sent to the main thread to obtain the encoded data.

[0087] Based on the same inventive concept, the present application also provides a device corresponding to the method in Example 1, see Example 2 for details.

[0088] Embodiment 2

[0089] like Figure 2 As shown, in this embodiment, a high-performance and highly compatible cloud phone camera streaming device is provided, including:

[0090] In the preparation module, the cloud phone sends a camera opening instruction and video acquisition information to the browser, wherein the video acquisition information includes: video width, video height and frame rate;

[0091] Apply to open the module, call navigator.mediaDevices.getUserMedia to open the local device camera, and pass in the video acquisition information;

[0092] The encoding module obtains video data according to the video acquisition information, obtains the video track MediaStreamTrack through navigator.mediaDevices.getUserMedia, and calls back to the browser; performs H.264 encoding on the video track MediaStreamTrack to obtain encoded data;

[0093] The push module pushes the encoded data frame by frame to the cloud phone;

[0094] Decoding module, the cloud phone decodes the encoded data and injects it into the cloud phone virtual camera.

[0095] In this embodiment, preferably, the encoding module is specifically: obtaining video data according to the video acquisition information, obtaining the video track MediaStreamTrack of the video data through navigator.mediaDevices.getUserMedia, and calling back the browser; if the browser supports VideoEncoder API, H.264 encoding, MediaStreamTrackProcessor, MediaStreamTrackGenerator and TransformStream, the video track MediaStreamTrack is encoded with H.264 using WebCodecs; if the browser supports MediaRecoderAPI and H.264 encoding, the video track MediaStreamTrack is encoded with H.264 using MediaRecoder; if the browser supports WebAssembly, the video track MediaStreamTrack is encoded with H.264 using WASM to obtain encoded data.

[0096] In this embodiment, preferably, the H.264 encoding of the video track MediaStreamTrack using WebCodecs is specifically: using MediaStreamTrackProcessor to extract the video stream from the video track MediaStreamTrack; creating TransformStream to process the video stream read from MediaStreamTrackProcessor to obtain frame-by-frame data; and performing H.264 encoding on the frame-by-frame data through VideoEncoderAPI.

[0097] In this embodiment, preferably, the H.264 encoding of the video track MediaStreamTrack using MediaRecoder is specifically: converting the video track MediaStreamStrack into a MediaStream video real-time stream; calling the MediaRecoderAPI, and setting the recording format to H.264, and starting to record the MediaStream video real-time stream; during the recording process, performing H.264 encoding on the video real-time stream, and calling back the data of each video frame in real time to obtain the encoded data.

[0098] In this embodiment, preferably, the H.264 encoding of the video track MediaStreamTrack using WASM is specifically as follows: creating a Worker thread; in the Worker thread, dynamically loading the WASM H.264 encoder, and after the loading is completed, notifying the main thread; the main thread sends the video track MediaStreamStrack to the Worker; in the Worker, encoding is performed using the WASM H.264 encoder, and the encoded video frame data is sent to the main thread to obtain the encoded data.

[0099] Since the device introduced in the second embodiment of the present invention is a device used to implement the method of the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the device based on the method introduced in the first embodiment of the present invention, so it is not described here in detail. All devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0100] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0101] This embodiment supports pushing the user's local camera image to the application in the cloud phone in real time, realizing the cloud phone's functions such as scanning code, live broadcast, and taking photos.

[0102] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A high-performance and highly compatible cloud phone camera streaming method, characterized in that: The steps include: Step 1: The cloud phone sends a camera opening instruction and video acquisition information to the browser, wherein the video acquisition information includes: video width, video height and frame rate; Step 2: Call navigator.mediaDevices.getUserMedia to open the local device camera and pass in the video acquisition information; Step 3: Obtain video data according to the video acquisition information, obtain the video track MediaStreamTrack through navigator.mediaDevices.getUserMedia, and call back to the browser; perform H.264 encoding on the video track MediaStreamTrack to obtain encoded data; Step 4: Push the encoded data frame by frame to the cloud phone; Step 5: The cloud phone decodes the encoded data and injects it into the cloud phone virtual camera.

2. According to the high-performance and high-compatibility cloud phone camera streaming method of claim 1, it is characterized by: The step 3 specifically includes: obtaining video data according to the video acquisition information, obtaining the video track MediaStreamTrack through navigator.mediaDevices.getUserMedia, and calling back the video data to the browser; if the browser supports VideoEncoder API, H.264 encoding, MediaStreamTrackProcessor, MediaStreamTrackGenerator and TransformStream, the video track MediaStreamTrack is encoded with H.264 using WebCodecs; If the browser supports MediaRecoder API and H.264 encoding, the video track MediaStreamTrack is encoded with H.264 using MediaRecoder; if the browser supports WebAssembly, the video track MediaStreamTrack is encoded with H.264 using WASM to obtain encoded data.

3. According to the high-performance and high-compatibility cloud phone camera streaming method of claim 2, it is characterized by: The specific steps of performing H.264 encoding on the video track MediaStreamTrack using WebCodecs are as follows: extracting a video stream from the video track MediaStreamTrack using MediaStreamTrackProcessor; creating a TransformStream to process the video stream read from the MediaStreamTrackProcessor to obtain frame-by-frame data; and performing H.264 encoding on the frame-by-frame data using the VideoEncoderAPI.

4. According to the high-performance and high-compatibility cloud phone camera streaming method of claim 2, it is characterized by: The specific steps of using MediaRecoder to perform H.264 encoding on the video track MediaStreamTrack are as follows: converting the video track MediaStreamStrack into a MediaStream video real-time stream; calling the MediaRecoderAPI, setting the recording format to H.264, and starting to record the MediaStream video real-time stream; during the recording process, performing H.264 encoding on the video real-time stream, and calling back the data of each video frame in real time to obtain the encoded data.

5. According to a high-performance and high-compatibility cloud phone camera streaming method according to claim 2, it is characterized in that: The method of using WASM to perform H.264 encoding on the video track MediaStreamTrack is specifically as follows: creating a Worker thread; in the Worker thread, dynamically loading the WASM H.264 encoder, and notifying the main thread after the loading is completed; the main thread sends the video track MediaStreamStrack to the Worker; in the Worker, encoding is performed using the WASM H.264 encoder, and the encoded video frame data is sent to the main thread to obtain the encoded data.

6. A high-performance and highly compatible cloud phone camera streaming device, characterized in that: include: In the preparation module, the cloud phone sends a camera opening instruction and video acquisition information to the browser, wherein the video acquisition information includes: video width, video height and frame rate; Apply to open the module, call navigator.mediaDevices.getUserMedia to open the local device camera, and pass in the video acquisition information; The encoding module obtains video data according to the video acquisition information, obtains the video track MediaStreamTrack through navigator.mediaDevices.getUserMedia, and calls back to the browser; performs H.264 encoding on the video track MediaStreamTrack to obtain encoded data; The push module pushes the encoded data frame by frame to the cloud phone; Decoding module, the cloud phone decodes the encoded data and injects it into the cloud phone virtual camera.

7. A high-performance and highly compatible cloud phone camera streaming device according to claim 6, characterized in that: The encoding module specifically: obtains video data according to the video acquisition information, obtains the video track MediaStreamTrack through navigator.mediaDevices.getUserMedia, and calls back to the browser; if the browser supports VideoEncoderAPI, H.264 encoding, MediaStreamTrackProcessor, MediaStreamTrackGenerator and TransformStream, the video track MediaStreamTrack is encoded with H.264 using WebCodecs; If the browser supports MediaRecoder API and H.264 encoding, the video track MediaStreamTrack is encoded with H.264 using MediaRecoder; if the browser supports WebAssembly, the video track MediaStreamTrack is encoded with H.264 using WASM to obtain encoded data.

8. A high-performance and highly compatible cloud phone camera streaming device according to claim 7, characterized in that: The specific steps of performing H.264 encoding on the video track MediaStreamTrack using WebCodecs are as follows: extracting a video stream from the video track MediaStreamTrack using MediaStreamTrackProcessor; creating a TransformStream to process the video stream read from the MediaStreamTrackProcessor to obtain frame-by-frame data; and performing H.264 encoding on the frame-by-frame data using the VideoEncoderAPI.

9. A high-performance and highly compatible cloud phone camera streaming device according to claim 7, characterized in that: The specific steps of using MediaRecoder to perform H.264 encoding on the video track MediaStreamTrack are as follows: converting the video track MediaStreamStrack into a MediaStream video real-time stream; calling the MediaRecoderAPI, setting the recording format to H.264, and starting to record the MediaStream video real-time stream; during the recording process, performing H.264 encoding on the video real-time stream, and calling back the data of each video frame in real time to obtain the encoded data.

10. A high-performance and highly compatible cloud phone camera streaming device according to claim 7, characterized in that: The method of using WASM to perform H.264 encoding on the video track MediaStreamTrack is specifically as follows: creating a Worker thread; in the Worker thread, dynamically loading the WASM H.264 encoder, and notifying the main thread after the loading is completed; the main thread sends the video track MediaStreamStrack to the Worker; in the Worker, encoding is performed using the WASM H.264 encoder, and the encoded video frame data is sent to the main thread to obtain the encoded data.

Images