Browser-based audio and video recording method and device, equipment and storage medium

Abstract

The invention discloses an audio and video recording method, device and equipment based on a browser. The method comprises the steps that an interface provided by a browser is called to be connected with media input equipment; reading video data and audio data; the method comprises the following steps: acquiring a canvas provided by a browser, acquiring a required drawing operation, performing drawing operation processing on the canvas, and creating a new video frame for the processed canvas through a video frame creator provided by the browser; acquiring a scene modification requirement, modifying the digital signal of the audio according to the scene modification requirement, and establishing the modified audio signal into new audio data by using an audio data creator provided by the browser; encoding the new video frame and the new audio data, and adding the encoded data into a video file to obtain a processed video file; and downloading the processed video file to the local. According to the invention, recorded video pictures and videos and audios can be completely controlled, and recoding can be carried out according to a preset coding standard.

Description

Technical Field

[0001] The present invention relates to the technical field of network communications, and more specifically, to a browser-based audio and video recording method, device, equipment and storage medium. Background Art

[0002] In front-end technology, we access media devices such as cameras and microphones through browsers. However, when accessing media devices through browsers for audio and video recording, it is generally impossible to fully control the recorded content, such as adding video filters in real time, adjusting audio effects in real time, etc.

[0003] With the increasingly rich audio and video application scenarios, higher requirements are placed on the technology of recording audio and video through browsers. Therefore, this field urgently needs an audio and video recording method that can fully control the media data stream and get rid of the dependence on the server. Summary of the invention

[0004] In order to overcome at least one of the above-mentioned defects of the prior art, the present invention provides a browser-based audio and video recording method, device, equipment and storage medium. The technical solution adopted by the present invention is as follows.

[0005] In a first aspect, the present invention provides a browser-based audio and video recording method, comprising:

[0006] Call the interface provided by the browser to connect to the media input device;

[0007] Read the video data in the video track and the audio data in the audio track stream of the media data stream sent by the media input device;

[0008] Obtaining a canvas provided by the browser and a required drawing operation, performing a drawing operation on the video data on the canvas according to the required drawing operation, and creating a new video frame for the processed canvas through a video frame creator provided by the browser;

[0009] Performing pulse code modulation on the audio data to obtain an audio digital signal, obtaining a scene modification requirement, modifying the audio digital signal according to the scene modification requirement, and using an audio data creator provided by the browser to create new audio data from the modified audio signal;

[0010] Encode the new video frame and the new audio data, and add the encoded data to the video file to obtain a processed video file;

[0011] Convert the processed video file into a file stream, and download the file obtained by the file stream to the local computer.

[0012] In one implementation, the process of reading video data in a video track and audio data in an audio track stream of a media data stream sent by a media input device includes:

[0013] Creating a video media stream track processor and an audio media stream track processor for the video track and the audio track of the media data stream acquired by the media input device respectively;

[0014] Read the video data in the video track using the video media stream track processor;

[0015] The audio data in the audio track is read using the audio media stream track processor.

[0016] In one implementation, the process of performing pulse code modulation on the audio data to obtain a digital audio signal includes:

[0017] Allocate one or more binary data buffers according to the length and number of channels of the original audio data instance;

[0018] According to the copyTo() method provided by the instance to which the audio data belongs, the data is copied to the binary data buffer to obtain the digital signal of the audio.

[0019] In one implementation, before the process of encoding the new video frame and the new audio data, the process includes: using the WebCodecs API provided by the browser to create an audio and video encoder.

[0020] In one embodiment, the process of adding the encoded data to the video file to obtain the processed video file includes:

[0021] Use the MP4Box.js JavaScript library to create MP4 files and use them as a carrier for writing audio and video data;

[0022] Create an empty MP4 file, and then add basic MP4 format metadata to the empty MP4 file;

[0023] Create corresponding video tracks and audio tracks according to the encoded data;

[0024] Add the encoded video and audio data to the video track and audio track respectively.

[0025] In one implementation, the process of converting the processed video file into a file stream and downloading the file obtained by the file stream to a local computer includes:

[0026] Create a file selector for saving a file through the showSaveFilePicker provided by the browser, and write the file stream of the video file to the handle returned by the file selector.

[0027] In a second aspect, the present invention provides a browser-based audio and video recording device, comprising:

[0028] A calling module is used to call the interface provided by the browser to connect to the media input device;

[0029] A reading module, used for reading video data in a video track and audio data in an audio track stream of a media data stream sent by a media input device;

[0030] A video processing module, used to obtain a canvas provided by the browser, obtain a required drawing operation, perform a drawing operation on the video data on the canvas according to the required drawing operation, and create a new video frame for the processed canvas through a video frame creator provided by the browser;

[0031] An audio processing module, used for performing pulse code modulation on the audio data to obtain an audio digital signal, obtaining a scene modification requirement, modifying the audio digital signal according to the scene modification requirement, and using an audio data creator provided by the browser to create new audio data from the modified audio signal;

[0032] A re-encoding module is used to encode new video frames and new audio data, and add the encoded data to the video file to obtain a processed video file;

[0033] The conversion module is used to convert the processed video file into a file stream and download the file obtained by the file stream to the local computer.

[0034] In one embodiment, it further includes:

[0035] A video media stream track processor, used to read video data in a video track of a media data stream acquired by the media input device;

[0036] The audio media stream track processor is used to read the audio data in the audio track of the media data stream acquired by the media input device.

[0037] In a third aspect, the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned methods when executing the program.

[0038] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method of any of the above-mentioned embodiments.

[0039] In the present invention, the browser parses the media input device data, converts the media data stream obtained by the media input device into video frames and pulse code modulation, and the browser re-encodes the video frames and pulse code modulation. The browser creates a video file and writes the encoded audio and video data, so that the recorded video picture and video audio can be fully controlled and can be re-encoded according to a preset encoding standard. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the overall process of an implementation method of Example 1 of the present invention.

[0041] Figure 2 It is a schematic diagram of the overall structure of an implementation method of the second embodiment of the present invention. DETAILED DESCRIPTION

[0042] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0043] It should be noted that the terms "first\second\..." involved in the embodiments of the present invention are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that the specific order or sequence of "first\second\..." can be interchanged where permitted. It should be understood that the objects distinguished by "first\second\..." can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0044] Embodiment 1

[0045] See also Figure 1 , Figure 1 The flowchart of a method for recording audio and video based on a browser provided in the first embodiment of the present invention includes step S110, step S120, step S130, step S140, step S150 and step S160. It should be noted that step S110, step S120, step S130, step S140, step S150 and step S160 are only reference numerals for clarifying the embodiment and the appendix. Figure 1 The corresponding relationship does not limit the order of the steps in this embodiment.

[0046] Step S110, calling the interface provided by the browser to connect to the media input device;

[0047] Step S120, reading video data in the video track of the media data stream and audio data in the audio track stream sent by the media input device;

[0048] Step S130, obtaining a canvas provided by the browser and a required drawing operation, performing a drawing operation on the video data on the canvas according to the required drawing operation, and creating a new video frame for the processed canvas through a video frame creator provided by the browser;

[0049] Step S140, performing pulse code modulation on the audio data to obtain an audio digital signal, obtaining a scene modification requirement, modifying the audio digital signal according to the scene modification requirement, and using an audio data creator provided by the browser to create new audio data from the modified audio signal;

[0050] Step S150, encoding the new video frame and the new audio data, and adding the encoded data to the video file to obtain a processed video file;

[0051] Step S160, converting the processed video file into a file stream, and downloading the file obtained by the file stream to a local computer.

[0052] This method is applicable to scenarios where a browser is used. In step S110, the interface provided by the browser is called to connect to the media input device, such as a camera, video capture device, microphone, and user desktop. This step S110 is mainly to activate the browser's access to the media device through the navigator.mediaDevices provided by the browser. Through this step, the user will be prompted to use the media input permission. Finally, the media input will generate a media data stream, which usually contains a video track and an audio track.

[0053] After the interface is connected in step S110, step S120 reads the video data in the video track of the media data stream and the audio data in the audio track stream sent by the media input device.

[0054] Step S130 is to perform customized processing on the video data obtained in step S120. This step mainly reads the video data and processes it according to the actual scene, such as adding filters and special effects to the video frames.

[0055] Specifically, each video frame can be drawn on a canvas provided by the browser, and other drawing operations can be performed on the canvas according to actual needs. Finally, a new video frame is created for encoding through a video frame creator provided by the browser on the processed canvas.

[0056] Step S140 is to perform customized processing on the audio data obtained in step S120. This step is mainly to process the audio data according to the actual scene, such as adjusting the audio volume, mixing, etc.

[0057] Specifically, the obtained audio data can be extracted from the pulse code modulation in the audio data to obtain the digital signal of the audio, and then modified according to the actual scene, and then created as new audio data for encoding through the video data creator provided by the browser.

[0058] In one implementation, the process of step S120 includes: step S121, step S122, and step S123.

[0059] Step S121, creating a video media stream track processor and an audio media stream track processor respectively for the video track and the audio track of the media data stream acquired by the media input device;

[0060] Step S122, using the video media stream track processor to read the video data in the video track;

[0061] Step S123: Use the audio media stream track processor to read the audio data in the audio track.

[0062] In this implementation, media stream track processors are created for the video track and audio track of the acquired media data stream respectively. This implementation is mainly to use the browser's MediaStreamTrackProcessor to parse the media data stream, and to construct media stream track processors for the video track and audio track respectively, through which the video frames in the video track and the audio data in the audio track stream can be read.

[0063] In one implementation, the process of performing pulse code modulation on the audio data to obtain a digital audio signal in step S140 includes: step S141 and step S142.

[0064] Step S141, allocating one or more binary data buffers according to the length and number of channels of the original audio data;

[0065] Step S142, copying the data into the binary data buffer according to the copyTo() method provided by the instance to which the audio data belongs, to obtain a digital audio signal.

[0066] This implementation is a specific process of performing pulse code modulation on the audio data to obtain a digital audio signal. One or more binary data buffers are allocated according to the length of the original audio data and the number of channels, and the copyTo() method provided by the instance to which the audio data belongs is used to copy the data to the binary data buffer, and then the digital audio signal can be obtained.

[0067] Step S150 uses a video encoder and an audio encoder to encode the processed video frame and audio data. This step mainly encodes the video frame and audio data, and adds the encoded data to the video file, so that we can obtain a customized video file.

[0068] In one implementation, the process of adding the encoded data to the video file to obtain the processed video file in step S150 includes:

[0069] Step S151, creating an MP4 file through the JavaScript library MP4Box.js and using it as a carrier for writing audio and video data;

[0070] Step S152, creating an empty MP4 file, and then adding basic MP4 format metadata to the empty MP4 file;

[0071] Step S153, creating corresponding video tracks and audio tracks according to the encoded data;

[0072] Step S154, adding the encoded video and audio data to the video track and audio track respectively.

[0073] In this implementation, MP4 files are created through the JavaScript library MP4Box.js and used as a carrier for writing audio and video data. First, an empty MP4 file is created, and then basic MP4 format metadata is added to the MP4 file; then, corresponding video tracks and audio tracks are created based on the encoded data; finally, the encoded video and audio data are added to the video track and audio track respectively.

[0074] In one implementation, before the process of encoding the new video frame and the new audio data in step S150, it includes: using the WebCodecs API provided by the browser to create an audio and video encoder.

[0075] This implementation is mainly for creating a video encoder and an audio encoder that encode the processed audio and video data streams into a file data stream.

[0076] Among them, it is necessary to preset the video encoding standard, such as the codec, frame rate, bit rate, width and height configurations to be used; it is necessary to preset the audio encoding standard, such as the codec, sampling rate, channel, bit rate, etc. to be used. Specifically, the creation of the video encoder mainly uses the WebCodecs API provided by the browser, through which we can complete the encoding and decoding of audio and video in the browser without relying on the server or web page assembly components, etc., and this method can bring a very considerable efficiency improvement for audio and video encoding and decoding compared to the above two methods.

[0077] Step S160 converts the video file obtained in step S150 into a file stream. This step mainly converts the video file into a file stream, and after obtaining the file stream, the file can be downloaded to the local computer.

[0078] In one implementation, the process of step S160 includes: step S161.

[0079] Step S161, creating a file selector for saving a file through showSaveFilePicker provided by the browser, and writing the file stream of the video file to the handle returned by the file selector.

[0080] In this embodiment, a file selector is created through showSaveFilePicker provided by the browser for saving a file, and the file stream of the video file obtained in step S150 is written to the handle returned by the file selector.

[0081] In this method, the browser parses the media input device data, converts the media data stream obtained by the media input device into video frames and pulse code modulation, and the browser re-encodes the video frames and pulse code modulation. The browser creates a video file and writes the encoded audio and video data, so that the recorded video picture and video audio can be fully controlled and can be re-encoded according to the preset encoding standard.

[0082] Embodiment 2

[0083] Corresponding to the method of embodiment 1, as Figure 2 As shown, the present invention also provides a browser-based audio and video recording device, including: a calling module 210, a reading module 220, a video processing module 230, an audio processing module 240, a re-encoding module 250 and a conversion module 260.

[0084] The calling module 210 is used to call the interface provided by the browser to connect to the media input device;

[0085] The reading module 220 is used to read the video data in the video track and the audio data in the audio track stream of the media data stream sent by the media input device;

[0086] The video processing module 230 is used to obtain a canvas provided by the browser, obtain a required drawing operation, perform a drawing operation on the video data on the canvas according to the required drawing operation, and create a new video frame for the processed canvas through a video frame creator provided by the browser;

[0087] The audio processing module 240 is used to perform pulse code modulation on the audio data to obtain an audio digital signal, obtain a scene modification requirement, modify the audio digital signal according to the scene modification requirement, and use an audio data creator provided by the browser to create new audio data from the modified audio signal;

[0088] A re-encoding module 250, for encoding new video frames and new audio data, and adding the encoded data to the video file to obtain a processed video file;

[0089] The conversion module 260 is used to convert the processed video file into a file stream and download the file obtained by the file stream to the local computer.

[0090] In one implementation, the browser-based audio and video recording device further includes: a video media stream track processor and an audio media stream track processor.

[0091] A video media stream track processor, used to read video data in a video track of a media data stream acquired by the media input device;

[0092] The audio media stream track processor is used to read the audio data in the audio track of the media data stream acquired by the media input device.

[0093] In one implementation, the audio processing module performs pulse code modulation on the audio data to obtain a digital audio signal, including:

[0094] Allocate one or more binary data buffers according to the length and number of channels of the original audio data instance;

[0095] According to the copyTo() method provided by the instance to which the audio data belongs, the data is copied to the binary data buffer to obtain the digital signal of the audio.

[0096] In one embodiment, the re-encoding module is further used to create an audio and video encoder using the WebCodecs API provided by the browser.

[0097] In one implementation, the re-encoding module performs the process of adding the encoded data to the video file to obtain the processed video file, including:

[0098] Use the MP4Box.js JavaScript library to create MP4 files and use them as a carrier for writing audio and video data;

[0099] Create an empty MP4 file, and then add basic MP4 format metadata to the empty MP4 file;

[0100] Create corresponding video tracks and audio tracks according to the encoded data;

[0101] Add the encoded video and audio data to the video track and audio track respectively.

[0102] In one embodiment, the conversion module executes the process of converting the processed video file into a file stream and downloading the file obtained by the file stream to a local computer, including:

[0103] Create a file selector for saving a file through the showSaveFilePicker provided by the browser, and write the file stream of the video file to the handle returned by the file selector.

[0104] In this device, the media input device data is parsed based on the browser, the media data stream obtained by the media input device is converted into video frames and pulse code modulation, the browser re-encodes the video frames and pulse code modulation, the browser creates a video file and writes the encoded audio and video data, so that the recorded video picture and video audio can be fully controlled and can be re-encoded according to the preset encoding standard.

[0105] Embodiment 3

[0106] An embodiment of the present invention further provides a storage medium on which computer instructions are stored. When the instructions are executed by a processor, the browser-based audio and video recording method of any of the above embodiments is implemented.

[0107] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, random access memories (RAM), read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0108] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, RAM, ROM, magnetic disks or optical disks.

[0109] Corresponding to the above-mentioned computer storage medium, in one embodiment, a computer device is also provided, which includes a memory, an encoder, and a computer program stored in the memory and executable on the encoder, wherein when the encoder executes the program, any one of the browser-based audio and video recording methods in the above-mentioned embodiments is implemented.

[0110] The above-mentioned computer device parses the data of the media input device based on the browser, converts the media data stream obtained by the media input device into video frames and pulse code modulation, the browser re-encodes the video frames and pulse code modulation, the browser creates a video file and writes the encoded audio and video data, so that the recorded video picture and video audio can be fully controlled and can be re-encoded according to the preset encoding standard.

[0111] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A browser-based audio and video recording method, characterized in that: include: Call the interface provided by the browser to connect to the media input device; Read the video data in the video track and the audio data in the audio track stream of the media data stream sent by the media input device; Obtaining a canvas provided by the browser and a required drawing operation, performing a drawing operation on the video data on the canvas according to the required drawing operation, and creating a new video frame for the processed canvas through a video frame creator provided by the browser; Performing pulse code modulation on the audio data to obtain an audio digital signal, obtaining a scene modification requirement, modifying the audio digital signal according to the scene modification requirement, and using an audio data creator provided by the browser to create new audio data from the modified audio signal; Encode the new video frame and the new audio data, and add the encoded data to the video file to obtain a processed video file; Convert the processed video file into a file stream, and download the file obtained by the file stream to the local computer.

2. The browser-based audio and video recording method according to claim 1, characterized in that: The process of reading video data in a video track and audio data in an audio track stream of a media data stream sent by a media input device includes: Creating a video media stream track processor and an audio media stream track processor for the video track and the audio track of the media data stream acquired by the media input device respectively; Read the video data in the video track using the video media stream track processor; The audio data in the audio track is read using the audio media stream track processor.

3. The browser-based audio and video recording method according to claim 1, characterized in that: The process of performing pulse code modulation on the audio data to obtain a digital audio signal includes: Allocate one or more binary data buffers according to the length and number of channels of the original audio data instance; According to the copyTo() method provided by the instance to which the audio data belongs, the data is copied to the binary data buffer to obtain the digital signal of the audio.

4. The browser-based audio and video recording method according to claim 1, characterized in that: Before the process of encoding the new video frame and the new audio data, the process includes: using the WebCodecs API provided by the browser to create an audio and video encoder.

5. The browser-based audio and video recording method according to claim 1, characterized in that: The process of adding the encoded data to the video file to obtain the processed video file includes: Use the MP4Box.js JavaScript library to create MP4 files and use them as a carrier for writing audio and video data; Create an empty MP4 file, and then add basic MP4 format metadata to the empty MP4 file; Create corresponding video tracks and audio tracks according to the encoded data; Add the encoded video and audio data to the video track and audio track respectively.

6. The browser-based audio and video recording method according to claim 1, characterized in that: The process of converting the processed video file into a file stream and downloading the file obtained by the file stream to a local computer includes: Create a file selector for saving a file through the showSaveFilePicker provided by the browser, and write the file stream of the video file to the handle returned by the file selector.

7. A browser-based audio and video recording device, characterized in that: include: A calling module is used to call the interface provided by the browser to connect to the media input device; A reading module, used for reading video data in a video track and audio data in an audio track stream of a media data stream sent by a media input device; A video processing module, used to obtain a canvas provided by the browser, obtain a required drawing operation, perform a drawing operation on the video data on the canvas according to the required drawing operation, and create a new video frame for the processed canvas through a video frame creator provided by the browser; An audio processing module, used for performing pulse code modulation on the audio data to obtain an audio digital signal, obtaining a scene modification requirement, modifying the audio digital signal according to the scene modification requirement, and using an audio data creator provided by the browser to create new audio data from the modified audio signal; A re-encoding module is used to encode new video frames and new audio data, and add the encoded data to the video file to obtain a processed video file; The conversion module is used to convert the processed video file into a file stream and download the file obtained by the file stream to the local computer.

8. The browser-based audio and video recording device according to claim 7, characterized in that: Also includes: A video media stream track processor, used to read video data in a video track of a media data stream acquired by the media input device; The audio media stream track processor is used to read the audio data in the audio track of the media data stream acquired by the media input device.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images