Browser-based audio and video processing method and device, electronic equipment and medium

Abstract

An audio and video processing method and device based on a browser, an electronic device and a medium are disclosed in the embodiments of the present application. The method comprises: reserving an audio and video area for displaying audio and video information on a browser page; adjusting an audio and video display pane of an audio and video decoding processing unit into the reserved audio and video area; the browser and the audio and video decoding processing unit run on the same device, and the audio and video decoding processing unit does not run on the browser; and audio and video playing is performed through the audio and video display pane adjusted into the audio and video area. According to the scheme of the present application, audio and video data can be decoded through an audio and video decoding processing unit independent of the browser, and the decoded audio and video display pane is adjusted into the audio and video area of the browser for audio and video playing, thereby solving the problems of limited decoding types and limited memory of the browser, without the need to install a decoding plug-in and return decoded data to the browser, and reducing the waste of network resources and the waste of consumption of the memory of the browser.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the technical field of video monitoring, in particular to an audio and video processing method and device based on a browser, electronic equipment and medium. BACKGROUND

[0002] At present, in an audio and video monitoring scene, a user can access an IP address of a front-end device such as a network video recorder (NVR) on a browser to view monitoring audio and video information. However, since mainstream browsers have gradually cancelled the support for ActiveX and NPAPI plug-ins, and the decoding types of the browsers themselves are limited, and the consumption of network resources is also great when the audio and video data decoded by a local decoding unit is transmitted back to the browser for rendering and display. Therefore, how to efficiently play audio and video resources through the browser becomes extremely important. SUMMARY

[0003] The embodiment of the present application provides an audio and video processing method and device based on a browser, and electronic equipment and medium, to realize local audio and video decoding and playing on the browser through the principle of process interaction.

[0004] In a first aspect, the embodiment of the present application provides an audio and video processing method based on a browser, which comprises the following steps:

[0005] Reserving an audio and video area for displaying audio and video information on a browser page;

[0006] Adjusting an audio and video display pane of an audio and video decoding processing unit into the reserved audio and video area; the browser and the audio and video decoding processing unit run on the same device, and the audio and video decoding processing unit does not run on the browser;

[0007] Playing audio and video through the audio and video display pane adjusted into the audio and video area.

[0008] In a second aspect, the embodiment of the present application further provides an audio and video processing device based on a browser, which comprises the following modules:

[0009] An audio and video area reserving module is configured to reserve an audio and video area for displaying audio and video information on a browser page;

[0010] A display pane adapting module is configured to adjust an audio and video display pane of an audio and video decoding processing unit into the reserved audio and video area; the browser and the audio and video decoding processing unit run on the same device, and the audio and video decoding processing unit does not run on the browser;

[0011] The audio and video playing module is configured to play the audio and video through the audio and video display pane adjusted to the audio and video area.

[0012] In a third aspect, an electronic device is provided in the embodiments of the present application, comprising:

[0013] one or more processors;

[0014] a storage device configured to store one or more programs;

[0015] The one or more programs are executed by the one or more processors, so that the one or more processors implement the browser-based audio and video processing method provided in any of the embodiments of the present application.

[0016] In a fourth aspect, a medium having a computer program stored thereon is provided in the embodiments of the present application, and the program is executed by a processor to implement the browser-based audio and video processing method provided in any of the embodiments of the present application.

[0017] The embodiments of the present application provide a browser-based audio and video processing method, which reserves an audio and video area for displaying audio and video information on a browser page, and then adjusts an audio and video display pane of an audio and video decoding processing unit to the reserved audio and video area, wherein the browser and the audio and video decoding processing unit run on the same device, and the audio and video decoding processing unit does not run on the browser, and then the audio and video is played through the audio and video display pane adjusted to the audio and video area. By using the technical solution of the present application, the audio and video data can be decoded by the audio and video decoding processing unit independent of the browser process, and the audio and video is played by adjusting the decoded audio and video display pane to the audio and video area reserved by the browser, effectively solving the problems of limited decoding types of the browser and limited memory of the browser. The audio and video decoding plug-in does not need to be installed in the browser, and the decoded data does not need to be sent to the browser for rendering, thereby greatly reducing the waste of network resources and the waste of consumption of the memory of the browser.

[0018] The above summary of the application is merely a summary of the technical solutions of the present application. In order to enable the technical solutions of the present application to be more clearly understood, the embodiments of the present application can be implemented in accordance with the contents of the specification, and in order to enable the above and other purposes, features and advantages of the present application to be more apparent, the specific embodiments of the present application are described below. BRIEF DESCRIPTION OF DRAWINGS

[0019] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments made with reference to the accompanying drawings. The drawings are merely intended to illustrate the preferred embodiments, and are not considered as limiting the present application. Moreover, the same reference numerals are used to denote the same components throughout the drawings. In the drawings:

[0020] Figure 1 is a flowchart of a browser-based audio and video processing method provided in an embodiment of the present application;

[0021] Figure 2 is a flowchart of another browser-based audio and video processing method provided in an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of data transmission provided in an embodiment of the present application;

[0023] Figure 4 is a structural block diagram of a browser-based audio and video processing apparatus provided in an embodiment of the present application;

[0024] Figure 5 is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The present application will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, but not all the structures.

[0026] Before the example embodiments are discussed in more detail, it should be mentioned that some of the example embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts depict the operations (or steps) as sequential processes, many of the operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The processes can be terminated when their operations are completed, but can also have additional steps not included in the drawings. The processes can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0027] The browser-based audio and video processing method, apparatus, electronic device and medium will be described in detail below through the following embodiments and optional technical solutions of each embodiment.

[0028] Figure 1 is a flowchart of a browser-based audio and video processing method provided in an embodiment of the present application. The embodiment of the present application can be applicable to the case of local audio and video decoding and playing on a browser through the principle of process interaction. The method can be executed by a browser-based audio and video processing apparatus, which can be realized in the form of software and / or hardware and integrated on any electronic device with network communication function. For example, the electronic device can include, but is not limited to, a decoding device, a computer device and a server, etc.

[0029] As Figure 1As shown, the browser-based audio and video processing method provided in the embodiment of the present application can include the following steps S110-S130:

[0030] S110, reserving an audio and video area for displaying audio and video information on a browser page.

[0031] In the present embodiment, in a page requiring audio and video playing, the browser can reserve an audio and video area of a certain size at the audio and video playing position for displaying audio and video information. The position and size of the audio and video area are not fixed and can be adjusted by the browser according to actual playing requirements.

[0032] S120, adjusting an audio and video display pane of an audio and video decoding processing unit into the reserved audio and video area; the browser and the audio and video decoding processing unit run on the same device, and the audio and video decoding processing unit does not run on the browser.

[0033] The audio and video decoding processing unit can be an executable exe file for decoding audio and video streams and rendering and displaying the decoded data. In the present embodiment, the audio and video decoding processing unit can be used to decode and process audio and video data, and play the decoded audio and video in the audio and video display pane of the audio and video decoding processing unit. Then, the audio and video display pane of the audio and video decoding processing unit can be dragged into the audio and video area of the browser according to the position of the audio and video area reserved by the browser page, so as to realize audio and video playing on the browser page. The browser and the audio and video decoding processing unit run on the same device, and the audio and video decoding processing unit does not run on the browser.

[0034] In the present embodiment, when a user accesses the browser, the user can be prompted to download and install the audio and video decoding processing unit to the local browser automatically. It should be noted that the audio and video decoding processing unit does not belong to the browser, and the audio and video decoding processing unit and the browser are two independent running processes. The display position of the audio and video display pane is only adjusted to the position of the audio and video area for display, and the audio and video display pane is displayed on the audio and video area, giving a feeling of immersive display of the content displayed by the audio and video area of the browser.

[0035] S130, playing audio and video through the audio and video display pane adjusted to the audio and video area.

[0036] In the embodiment, the reserved audio and video area information can be sent to the audio and video decoding processing unit through the browser, and then the audio and video display pane of the audio and video decoding processing unit can be dragged to the audio and video display pane of the audio and video area, so that the audio and video playing in the browser is realized. The audio and video area information can include the position information and size information of the audio and video area.

[0037] Specifically, the position information and size information of the reserved audio and video area are sent to the audio and video decoding processing unit through the browser. When the audio and video decoding processing unit receives the message of the browser, the audio and video display pane of the audio and video decoding processing unit can be adjusted according to the position information and size information of the audio and video area, so as to be displayed in the audio and video area reserved in the browser in the reserved size.

[0038] It should be noted that the scheme is to decode the audio and video data through the audio and video decoding processing unit independent of the browser, and to play the decoded audio and video through the audio and video display pane in the audio and video decoding processing unit. The entire decoding process is not run on the browser, and the decoded audio and video data does not need to be transmitted to the browser. Only the process communication is used to adjust the position of the audio and video decoding processing unit to coincide with the audio and video area reserved by the browser, so as to create an effect of playing any format of audio and video through the browser.

[0039] The embodiment of the application provides a browser-based audio and video processing method. According to the scheme, the audio and video data can be decoded through the audio and video decoding processing unit independent of the browser process, and the audio and video playing can be realized by adjusting the decoded audio and video display pane to the audio and video area reserved by the browser. The problems of limited decoding types of the browser and limited memory of the browser are effectively solved. The audio and video decoding plug-in does not need to be installed in the browser, and the decoded data does not need to be sent to the browser for rendering, so as to greatly reduce the waste of network resources and the waste of consumption of the memory of the browser.

[0040] On the basis of the above embodiment, after the audio and video playing through the audio and video display pane adjusted to the audio and video area, the following steps A1-A2 can be further included.

[0041] Step A1, determining pane adjustment information of the audio and video display pane of the audio and video area; the pane adjustment information indicates size adjustment, position adjustment, cropping and start control of the audio and video display pane.

[0042] The pane adjustment information can include size information, position information, cropping information and start adjustment information of the pane, and can be respectively used for size adjustment, position adjustment, cropping and start control of the audio and video display pane.

[0043] In the embodiment, when the user performs a zoom operation on the browser or the area that needs to be displayed by the browser page element is covered by the audio / video decoding processing unit, and when the browser page does not need to perform audio / video playing or the browser page is directly closed, the audio / video display pane can be adaptively adjusted, and the corresponding pane adjustment information is first obtained. The page element can be an object displayed on the browser page, for example, the page element can be a prompt pop-up window.

[0044] It can be understood that when the user performs a zoom operation on the browser, the position and size of the reserved audio / video area on the browser page will change accordingly. When the page element needs to be displayed in the browser audio / video area covered by the audio / video decoding processing unit, the page element can be displayed on the browser by clipping the covered area. In addition, when the browser switches to an interface without audio / video playing or directly closes the page, the audio / video display pane of the audio / video decoding processing unit can be directly closed.

[0045] In step A2, the pane adjustment information is sent to the audio / video decoding processing unit, so that the audio / video decoding processing unit adjusts the audio / video display pane in the audio / video area according to the pane adjustment information.

[0046] In the embodiment, the browser can send the pane adjustment information to the audio / video decoding processing unit, and the audio / video decoding processing unit can adjust the audio / video display pane in the audio / video area according to the pane adjustment information. Specifically, when the user performs a zoom operation on the browser, the browser can resend the position, size and start adjustment information of the audio / video display pane in the audio / video area to the audio / video decoding processing unit, and the audio / video decoding processing unit can start the pane adjustment according to the start adjustment information, and update the position and size of the audio / video display pane in the audio / video area according to the position and size information.

[0047] When the page element needs to be displayed in the area covered by the audio / video decoding processing unit, the browser can obtain the position and size of the area that blocks the page element, and send the position, size and start adjustment information of the blocked area to the audio / video decoding processing unit. Then, the audio / video decoding processing unit can start the pane adjustment according to the start adjustment information, and cut off the area blocked by the pane according to the position and size information of the blocked area in a way of modifying the size of the audio / video display pane. If a mask is needed for the entire page, the entire audio / video display pane needs to be cut off.

[0048] When the browser switches to an interface without audio / video playing or directly closes the page, the browser can send the start adjustment information to the audio / video decoding processing unit to instruct the audio / video decoding processing unit to close the audio / video display pane, so as to prevent the display of other page information.

[0049] In the embodiment, optionally, the window pane adjustment information comprises adjustment information for cropping a local window pane in the audio-video display window pane; the local window pane is determined based on a position and a size of a browser page element blocked by the audio-video display window pane relative to the audio-video display window pane.

[0050] In the embodiment, the local window pane can be determined according to the position and the size of the browser page element blocked by the audio-video display window pane relative to the audio-video display window pane. Specifically, the position and the size of the page element can be obtained by the browser, and the position and the size of the audio-video display window pane in the audio-video area can also be obtained by the browser, and then the position and the size of the area blocked by the page element can be calculated based on the obtained information, so as to determine the local window pane.

[0051] The above optional embodiment has the advantage that the audio-video display window pane in the audio-video area can be flexibly adjusted by the audio-video decoding processing unit in response to the operation of the user on the browser and the display requirement of the browser page element, so as to better meet the audio-video playing requirement.

[0052] On the basis of the above embodiment, optionally, after the audio-video playing is performed on the audio-video display window pane adjusted to the audio-video area, the following steps B1-B2 can be further included.

[0053] Step B1, control the audio-video decoding processing unit to start drawing enabling, and obtain the drawing on the audio-video display window pane.

[0054] In the embodiment, when the user starts the drawing function on the browser page, the browser can send a message to the audio-video decoding processing unit to start the drawing enabling. After the drawing enabling is started, the user can draw corresponding drawing on the audio-video display window pane of the audio-video decoding processing unit by using the mouse. After the drawing is completed, the audio-video decoding processing unit can report the drawing data information to the browser, and the browser can process the drawing data.

[0055] Step B2, send the region position corresponding to the drawing to the target front-end device, so that the target front-end device marks the audio-video stream data in the region position.

[0056] The target front-end device can be a device that sends the audio-video data information to the audio-video decoding processing unit for processing, for example, the target front-end device can be an NVR device, etc. In the embodiment, after the browser processes the drawing data, the processed result can be sent to the target front-end device, and the region position corresponding to the drawing can be obtained by the browser and sent to the target front-end device, so that the target front-end device marks the audio-video stream data in the region position.

[0057] The above optional embodiment has the advantage that the drawing enabling can be started by the audio and video decoding processing unit in response to the user's operation on the browser, so that the user can complete the drawing operation, thereby better meeting the user's drawing needs.

[0058] Figure 2 is a flowchart of another browser-based audio and video processing method provided in an embodiment of the present application. The embodiment of the present application further optimizes the steps of S120 and S130 in the foregoing embodiment on the basis of the above-mentioned embodiment, and can be combined with each optional scheme in one or more of the above-mentioned embodiments. As shown in Figure 2 The browser-based audio and video processing method provided in the embodiment can include the following steps S210-S240:

[0059] S210, reserving an audio and video area for displaying audio and video information on a browser page.

[0060] S220, establishing an instant communication connection between the browser and the audio and video decoding processing unit; wherein the browser and the audio and video decoding processing unit run on the same device, and the audio and video decoding processing unit does not run on the browser; the instant communication connection includes a communication connection established by using a WebSocket protocol.

[0061] The WebSocket protocol can refer to a protocol for full-duplex communication on a single TCP connection. In the embodiment, the instant communication connection between the browser and the audio and video decoding processing unit can be established by using the WebSocket protocol.

[0062] Figure 3 is a schematic diagram of data transmission provided in an embodiment of the present application. Wherein, RTSP (Real Time Streaming Protocol) can refer to a real-time streaming protocol. As shown in Figure 3 The browser and the target front-end device perform data transmission through the HTTP protocol, the browser and the audio and video decoding processing unit establish a connection through the WebSocket protocol for bidirectional communication, and the audio and video decoding processing unit establishes a connection with the target front-end device through the RTSP protocol to obtain the audio and video stream. It should be noted that the information interaction between the target front-end device and the audio and video decoding processing unit, except for receiving the audio and video stream, needs to pass through the browser as an intermediate medium.

[0063] In an optional way of the embodiment, the instant communication connection between the browser and the audio and video decoding processing unit can include the following steps C1-C2:

[0064] Step C1, if the instant communication connection is detected to fail or the number of failures meets a preset condition, it is determined that the audio and video decoding processing unit is in an unawakened state.

[0065] The preset condition can be a preset condition for instant communication connection failure. For example, the preset condition can be set as the number of instant communication connection failures reaching a failure number threshold.

[0066] In this embodiment, when a user accesses a browser interface, the browser can be triggered to establish a connection with the audio and video decoding processing unit through the WebSocket protocol, and whether the audio and video decoding processing unit is awakened can be determined according to whether the connection is successfully established. When the instant communication connection is detected to fail or the number of failures meets a preset condition, it is determined that the audio and video decoding processing unit is in an unawakened state.

[0067] Step C2, if it is determined that the audio and video decoding processing unit is in an unawakened state, the audio and video decoding processing unit is actively awakened, and the browser continues to control the instant communication connection between the browser and the audio and video decoding processing unit.

[0068] In this embodiment, when it is determined that the audio and video decoding processing unit is in an unawakened state, the browser can actively awaken the audio and video decoding processing unit, and continue to establish an instant communication connection between the browser and the audio and video decoding processing unit through the WebSocket protocol.

[0069] The above optional implementation scheme has the advantage that when the audio and video decoding processing unit is in an unawakened state, the browser can actively awaken the audio and video decoding processing unit, thereby realizing the instant communication connection between the browser and the audio and video decoding processing unit.

[0070] S230, after the instant communication connection is established, the position and size information of the audio and video region is sent to the audio and video decoding processing unit, so that the audio and video decoding processing unit adjusts the corresponding audio and video display pane to be configured and displayed in the audio and video region.

[0071] In this embodiment, after the instant communication connection between the browser and the audio and video decoding processing unit is successfully established through the WebSocket protocol, the browser can send the position and size information of the audio and video region to the audio and video decoding processing unit through a WebSocket request, so that the audio and video decoding processing unit can adjust the audio and video display pane to be displayed in the audio and video region according to the position and size information. In addition, the browser can also control the audio and video decoding processing unit through a WebSocket request, such as controlling the display and hiding, positioning, cropping, and drawing functions of the audio and video decoding processing unit.

[0072] S240, playing the audio and video by adjusting the audio and video display pane to the audio and video area.

[0073] The embodiment of the present application provides a browser-based audio and video processing method, and the browser and the audio and video decoding processing unit can be connected through the WebSocket protocol by adopting the scheme, the browser can actively wake up when the audio and video decoding processing unit is in a wake-up state, then the audio and video decoding processing unit independent of the browser process can be used to decode the audio and video data, the decoded audio and video display pane can be adjusted to the audio and video area reserved by the browser to realize audio and video playing, the problems of limited decoding types of the browser and limited memory of the browser are effectively solved, the audio and video decoding plug-in does not need to be installed in the browser, and the decoded data does not need to be sent to the browser, so that the waste of network resources and the waste of consumption of the memory of the browser are greatly reduced.

[0074] In an optional mode of the embodiment, the audio and video playing by adjusting the audio and video display pane to the audio and video area can include the following steps D1-D2.

[0075] S1, sending the target address and the target port to the target front-end device; wherein the target address and the target port include the IP address and the idle port of the target front-end device, which are provided by the audio and video decoding processing unit and are used to receive the audio and video stream.

[0076] In the embodiment, when the user performs the audio and video playing operation through the browser, the browser can first transmit the IP address and the idle port, which are used to receive the audio and video stream, to the target front-end device through the HTTP message, and notify the target front-end device to prepare the audio and video stream. Then the session ID of the audio and video playing sent by the target front-end device can be acquired through the browser, and the session ID and the data required for playing the audio and video can be transmitted to the audio and video decoding processing unit through the WebSocket message, at this time, the audio and video decoding processing unit has started to listen to the idle port transmitted to the target front-end device. Subsequently, the audio and video stream data can be sent to the IP address and the idle port specified by the audio and video decoding processing unit through the target front-end device.

[0077] S2, acquiring the target audio and video stream data sent by the target front-end device through the target address and the target port through the audio and video decoding processing unit, and playing the audio and video display pane in the audio and video area after decoding.

[0078] In the embodiment, the target audio and video stream data sent by the target front-end device at the target address and the target port can be acquired by the audio and video decoding processing unit. After the audio and video stream data is received, the data stream can be decoded, and the decoded audio and video can be played and displayed by the audio and video decoding processing unit. Further, the audio and video display pane of the audio and video decoding processing unit can be placed in the audio and video area of the browser, so as to give the user a feeling that the audio and video is played in the audio and video display pane of the browser.

[0079] Optionally, the target audio and video stream data sent by the target front-end device at the target address and the target port can be acquired by the audio and video decoding processing unit, which can include the following steps: sending the session identification information and the audio and video playing key information of the target audio and video stream data acquired from the target front-end device to the audio and video decoding processing unit; and listening to the target port by the audio and video decoding processing unit, and receiving the target audio and video stream data sent by the target front-end device at the target address and the target port when the stream data is listened to.

[0080] The session identification information can be used to identify the target audio and video stream data, so as to accurately identify the corresponding target audio and video stream data. The key information can include the duration and format of the audio and video playing. In the embodiment, the session identification information and the audio and video playing key information of the target audio and video stream data can be acquired by the target front-end device, and sent to the target address and the target port of the audio and video decoding processing unit. When the audio and video decoding processing unit listens to the target port and finds the stream data, the target audio and video stream data sent by the target front-end device will be received.

[0081] The above optional implementation scheme has the advantage that the target audio and video stream data can be sent to the specified address and port, and the target port can be listened to by the audio and video decoding processing unit, so as to ensure that the target audio and video stream data can be successfully received.

[0082] Figure 4 is a structural block diagram of an audio and video processing device based on a browser provided in an embodiment of the present application. The embodiment of the present application can be applied to the case that the local audio and video decoding and playing are performed on the browser through the principle of process interaction. The device can be realized in the software and / or hardware mode, and integrated on any electronic device with network communication function. As shown in Figure 4 The audio and video processing device based on the browser provided in the embodiment of the present application can include the following contents: an audio and video area reservation module 410, a display pane adaptation module 420 and an audio and video playing module 430. Among them:

[0083] The audio and video area reservation module 410 is used for reserving an audio and video area for displaying audio and video information on the browser page;

[0084] The display pane adaptation module 420 is configured to adjust the audio / video display pane of the audio / video decoding processing unit into the reserved audio / video area; the browser and the audio / video decoding processing unit run on the same device, and the audio / video decoding processing unit does not run on the browser;

[0085] The audio / video playing module 430 is configured to play audio / video through the audio / video display pane adjusted into the audio / video area.

[0086] Optionally, the display pane adaptation module 420 comprises:

[0087] The instant communication connection unit is configured to perform instant communication connection between the browser and the audio / video decoding processing unit; the instant communication connection comprises a communication connection established by using the WebSocket protocol;

[0088] The display pane adaptation unit is configured to, after the instant communication connection is established, send the position and size information of the audio / video area to the audio / video decoding processing unit, so that the audio / video decoding processing unit adjusts the corresponding audio / video display pane into the audio / video area for configuration display.

[0089] Optionally, the instant communication connection unit comprises:

[0090] If it is detected that the instant communication connection fails or the number of failures meets a preset condition, it is determined that the audio / video decoding processing unit is in an unawakened state;

[0091] If it is determined that the audio / video decoding processing unit is in the unawakened state, the audio / video decoding processing unit is awakened actively, and the instant communication connection between the browser and the audio / video decoding processing unit is continued.

[0092] Optionally, the audio / video playing module 430 comprises:

[0093] The target information sending unit is configured to send a target address and a target port to a target front-end device; the target address and the target port comprise an IP address and an idle port provided by the audio / video decoding processing unit, which are used to plan to receive the audio / video stream of the target front-end device;

[0094] The data processing unit is configured to acquire, through the audio / video decoding processing unit, the target audio / video stream data sent by the target front-end device according to the target address and the target port, and play the decoded audio / video stream in the audio / video display pane of the audio / video area.

[0095] Optionally, the data processing unit comprises:

[0096] send session identification information of the target audio and video stream data obtained from the target front-end device and audio and video playing key information to the audio and video decoding processing unit;

[0097] The audio and video decoding processing unit listens to the target port and receives the target audio and video stream data sent by the target front-end device according to the target address and the target port when the stream data is listened to.

[0098] On the basis of the above-mentioned embodiments, the browser-based audio and video processing device further comprises:

[0099] A pane adjustment information determination module is configured to determine pane adjustment information of the audio and video display pane of the audio and video region after audio and video playing is performed through the audio and video display pane adjusted to the audio and video region; the pane adjustment information indicates size adjustment, position adjustment, clipping and start control of the audio and video display pane.

[0100] A display pane adjustment module is configured to send the pane adjustment information to the audio and video decoding processing unit, so that the audio and video decoding processing unit adjusts the audio and video display pane in the audio and video region according to the pane adjustment information.

[0101] On the basis of the above-mentioned embodiments, the pane adjustment information comprises adjustment information for clipping a local pane in the audio and video display pane; the local pane is determined based on the position and size of the browser page element blocked by the audio and video display pane relative to the audio and video display pane.

[0102] On the basis of the above-mentioned embodiments, the browser-based audio and video processing device further comprises:

[0103] A drawing obtaining module is configured to control the audio and video decoding processing unit to start drawing enabling after audio and video playing is performed through the audio and video display pane adjusted to the audio and video region, and to obtain a drawing drawn on the audio and video display pane.

[0104] A data marking module is configured to send a region position corresponding to the drawn drawing to the target front-end device, so that the target front-end device marks the audio and video stream data of the region position.

[0105] The browser-based audio and video processing device provided in the embodiments of the present application can execute the browser-based audio and video processing method provided in any of the embodiments of the present application, has the corresponding functions and beneficial effects of executing the browser-based audio and video processing method, and the detailed process is described in the foregoing embodiments of the browser-based audio and video processing method.

[0106] Figure 5 is a structural schematic diagram of an electronic device provided in an embodiment of the present application. As shown in the structural diagram, the electronic device provided in the embodiment of the present application comprises one or more processors 510 and a storage device 520; the processor 510 in the electronic device can be one or more, Figure 5 Figure 5 The storage device 520 is configured to store one or more programs; the one or more programs are executed by the one or more processors 510, so that the one or more processors 510 implement the browser-based audio and video processing method according to any one of the embodiments of the present application.

[0107] The electronic device can further comprise an input device 530 and an output device 540.

[0108] The processor 510, the storage device 520, the input device 530 and the output device 540 in the electronic device can be connected through a bus or other means, Figure 5 for example, through a bus.

[0109] The storage device 520 in the electronic device can be used to store one or more programs as a kind of computer readable storage medium, the program can be a software program, a computer executable program and a module, such as the program instruction / module corresponding to the browser-based audio and video processing method provided in the embodiment of the present application. The processor 510 executes the software program, instruction and module stored in the storage device 520, thereby performing various functional applications and data processing of the electronic device, that is, implementing the browser-based audio and video processing method in the above method embodiment.

[0110] The storage device 520 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created according to the use of the electronic device, etc. In addition, the storage device 520 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some examples, the storage device 520 can further include a memory remotely arranged with respect to the processor 510, and these remote memories can be connected to the device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0111] The input device 530 can be used to receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device. The output device 540 can include a display device such as a display screen.

[0112] ​And the one or more programs included in the above-mentioned electronic device, when executed by the one or more processors 510, the programs perform the following operations:

[0113] Reserve an audio / video area on a browser page for displaying audio / video information;

[0114] Adjust an audio / video display window of an audio / video decoding processing unit into the reserved audio / video area; the browser and the audio / video decoding processing unit run on the same device, and the audio / video decoding processing unit does not run on the browser;

[0115] Play the audio / video through the audio / video display window adjusted into the audio / video area.

[0116] Of course, those skilled in the art can understand that when the one or more programs included in the above-mentioned electronic device are executed by the one or more processors 510, the programs can also perform the related operations in the browser-based audio / video processing method provided in any embodiment of the present application.

[0117] The present application provides a medium having a computer program stored thereon, which, when executed by a processor, is used to execute a browser-based audio / video processing method, the method comprising:

[0118] Reserve an audio / video area on a browser page for displaying audio / video information;

[0119] Adjust an audio / video display window of an audio / video decoding processing unit into the reserved audio / video area; the browser and the audio / video decoding processing unit run on the same device, and the audio / video decoding processing unit does not run on the browser;

[0120] Play the audio / video through the audio / video display window adjusted into the audio / video area.

[0121] Optionally, the program, when executed by the processor, can also be used to execute the browser-based audio / video processing method provided in any embodiment of the present application.

[0122] The computer readable medium of the embodiments of the present application can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM), a flash memory, an optical fiber, a portable CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus or device.

[0123] The computer readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave in a propagated data signal, in which the computer readable program code is contained. Such propagated data signal can take a variety of forms, including but not limited to electro-magnetic, optical or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium that is not a storage medium and that can communicate, propagate or transport program for use by or in connection with an instruction execution system, apparatus or device.

[0124] The program code contained on the computer readable medium can be transmitted using any suitable medium, including but not limited to wireless, wire line, optical fiber cable, radio frequency (RF), or any suitable combination thereof.

[0125] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0126] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or more embodiments or examples.

[0127] Note that the above only describes the preferred embodiments of the present application and the applied technical principles. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A browser-based audio and video processing method, characterized in that, The method comprises: Reserving an audio and video area for displaying audio and video information on a browser page; Adjusting an audio and video display pane of an audio and video decoding processing unit into the reserved audio and video area; the browser and the audio and video decoding processing unit run on the same device, and the audio and video decoding processing unit does not run on the browser; Playing audio and video through the audio and video display pane adjusted into the audio and video area; The playing of audio and video through the audio and video display pane adjusted into the audio and video area comprises: Sending a target address and a target port to a target front-end device through the browser; wherein the target address and the target port comprise an IP address and a free port provided by the audio and video decoding processing unit for planning to receive audio and video stream of the target front-end device; Obtaining target audio and video stream data sent by the target front-end device according to the target address and the target port through the audio and video decoding processing unit, and playing the decoded target audio and video stream data in the audio and video display pane of the audio and video area; The obtaining of the target audio and video stream data sent by the target front-end device according to the target address and the target port through the audio and video decoding processing unit comprises: Sending session identification information and audio and video playing key information of the target audio and video stream data obtained from the target front-end device to the audio and video decoding processing unit through the browser; Listening to the target port through the audio and video decoding processing unit, and receiving the target audio and video stream data sent by the target front-end device according to the target address and the target port when the stream data is listened to.

2. The method of claim 1, wherein, The adjusting of the audio and video display pane of the audio and video decoding processing unit into the reserved audio and video area comprises: Establishing an instant communication connection between the browser and the audio and video decoding processing unit; the instant communication connection comprises a communication connection established by using a WebSocket protocol; After the instant communication connection is established, sending position and size information of the audio and video area to the audio and video decoding processing unit, so that the audio and video decoding processing unit adjusts a corresponding audio and video display pane into the audio and video area for configuration display.

3. The method of claim 2, wherein, The establishing of the instant communication connection between the browser and the audio and video decoding processing unit comprises: If it is detected that the instant communication connection fails or the number of failures meets a preset condition, it is determined that the audio and video decoding processing unit is in an unawakened state; If it is determined that the audio and video decoding processing unit is in the unawakened state, the audio and video decoding processing unit is awakened actively, and the browser and the audio and video decoding processing unit continue to establish the instant communication connection.

4. The method of claim 1, wherein, After the playing of audio and video through the audio and video display pane adjusted into the audio and video area, the method further comprises: Determining pane adjustment information of the audio and video display pane of the audio and video area; the pane adjustment information indicates size adjustment, position adjustment, cropping and start control of the audio and video display pane; Sending the pane adjustment information to the audio and video decoding processing unit, so that the audio and video decoding processing unit adjusts the audio and video display pane in the audio and video area according to the pane adjustment information.

5. The method of claim 4, wherein, The window pane adjustment information comprises adjustment information for clipping a local window pane in the audio / video display window pane; and the local window pane is determined based on a position and a size of a browser page element blocked by the audio / video display window pane relative to the audio / video display window pane.

6. The method of claim 1, wherein, After audio / video playing is performed through the audio / video display window pane adjusted to the audio / video region, the method further comprises: controlling the audio / video decoding processing unit to start drawing enabling, and acquiring a drawing on the audio / video display window pane; sending a region position corresponding to the drawing to a target front-end device, so that the target front-end device performs directional marking on audio / video stream data of the region position.

7. A browser-based audio-video processing apparatus, characterized by, The apparatus comprises: an audio / video region reserving module configured to reserve an audio / video region for displaying audio / video information on a browser page; a display window pane adapting module configured to adjust an audio / video display window pane of an audio / video decoding processing unit into the reserved audio / video region; the browser and the audio / video decoding processing unit are running on a same device, and the audio / video decoding processing unit is not running on the browser; an audio / video playing module configured to perform audio / video playing through the audio / video display window pane adjusted to the audio / video region; the audio / video playing module comprises: a target information sending unit configured to send a target address and a target port to a target front-end device through the browser; wherein the target address and the target port comprise an IP address and a free port provided by the audio / video decoding processing unit for planning to receive audio / video stream of the target front-end device; a data processing unit configured to acquire target audio / video stream data sent by the target front-end device according to the target address and the target port through the audio / video decoding processing unit, and play the target audio / video stream data decoded in the audio / video display window pane of the audio / video region; the data processing unit is specifically configured to send session identification information and audio / video playing key information of the target audio / video stream data acquired from the target front-end device to the audio / video decoding processing unit through the browser; and listen to the target port through the audio / video decoding processing unit, and receive the target audio / video stream data sent by the target front-end device according to the target address and the target port when the stream data is listened to.

8. An electronic device, comprising: comprise: one or more processors; a storage configured to store one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the browser-based audio / video processing method in any one of claims 1-6.

9. A medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the browser-based audio / video processing method in any one of claims 1-6.

Images