A data recording method and device, computer equipment and storage medium

Abstract

The present disclosure provides a data recording method and device, computer equipment and a storage medium, wherein the method comprises: in response to a media content recording request for a target subprogram running in a target application program, determining an audio and video source matched with the target subprogram; in the case that the audio and video source comprises a front end of the target subprogram, acquiring first audio and video data collected from the front end of the target subprogram in a target data transmission mode matched with attribute information of a client of the target application program; and generating media content recording data corresponding to the target subprogram based on the first audio and video data.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and in particular, to a data recording method and device, a computer device and a storage medium. BACKGROUND

[0002] With the rapid development of Internet technology and mobile terminal device technology, using mobile phones and other terminal smart devices for entertainment has gradually become the mainstream lifestyle of people. In order to meet the actual needs of media content sharing and other actual needs when using terminal devices, there is often a need to record the media content of the application program running in the terminal device.

[0003] In related technologies, when a subprogram is running in an application program, the audio and video data unrelated to the subprogram in the application program will be recorded together when recording the audio and video data related to the subprogram, resulting in recording some audio and video data that the user does not need, and thus failing to meet the actual recording needs of the user. SUMMARY

[0004] The present disclosure provides at least a data recording method and device, a computer device and a storage medium.

[0005] In a first aspect, the present disclosure provides a data recording method, comprising:

[0006] In response to a media content recording request for a target subprogram running in a target application program, determining an audio and video source matched with the target subprogram;

[0007] In a case where the audio and video source includes a front end of the target subprogram, acquiring first audio and video data collected from the front end of the target subprogram in a target data transmission mode matched with attribute information of a client of the target application program;

[0008] Based on the first audio and video data, generating media content recording data corresponding to the target subprogram.

[0009] In a possible implementation, the method further comprises:

[0010] In a case where the audio and video source further includes a client corresponding to the target application program, acquiring second audio and video data collected from the client locally;

[0011] The generating, based on the first audio and video data, of the media content recording data corresponding to the target subprogram comprises:

[0012] Based on the first audio and video data and the second audio and video data, generating the media content recording data corresponding to the target subprogram.

[0013] In a possible implementation, the determining of the audio and video source matched with the target subprogram includes:

[0014] Based on the recording type parameter carried in the target subprogram code and / or the picture rendering parameter used by the target subprogram, the video source matched with the picture rendering attribute information of the target subprogram is determined.

[0015] In a possible implementation, the attribute information of the client includes system version information corresponding to the client.

[0016] The first audio and video data collected from the front end of the target subprogram is obtained in the target data transmission mode matched with the attribute information of the client of the target application program, and includes:

[0017] In a case where it is detected that the system version information corresponding to the client is located within a target version range, an extensible markup language (XML) request carrying the first audio and video data is initiated by the front end, and the first audio and video data is extracted from the XML request based on the client intercepting the XML request; and,

[0018] In a case where it is detected that the system version information corresponding to the client is located outside a target version range, a hypertext transfer protocol secure (HTTPS) request carrying the first audio and video data is initiated by the front end, and the first audio and video data is extracted from the HTTPS request by calling a first target interface to intercept the HTTPS request.

[0019] In a possible implementation, the video data in the first audio and video data is obtained according to the following steps:

[0020] The first picture rendering content of the target subprogram is subjected to texture data collection by calling a second target interface, to obtain target video data corresponding to the first picture rendering content; and the picture rendering code of the target subprogram includes target code injected in advance for indicating that the first picture rendering content can be subjected to texture data collection.

[0021] In a possible implementation, the audio data in the first audio and video data is obtained according to the following steps:

[0022] In response to an audio playback request, a target audio source node for playing audio is created, and an audio data collection node corresponding to the target audio source node is generated; and the audio data collection node is configured to listen to an audio playback event of the target audio source node.

[0023] Based on the audio data collection node, target audio data corresponding to the target audio source node is obtained.

[0024] In a possible implementation, the number of target sound source nodes is multiple;

[0025] After obtaining the target audio data corresponding to the target sound source node based on the audio data collection node, the method further includes:

[0026] segmenting the target audio data corresponding to each of the multiple target sound source nodes to obtain multiple sub-audio data corresponding to each target sound source node;

[0027] mixing the sub-audio data corresponding to the same time segment in the multiple target audio nodes to obtain mixed audio data transmitted from the front end to the client.

[0028] In a possible implementation, the video data in the second audio-video data is obtained according to the following steps:

[0029] calling a third target interface to collect texture data of second screen rendering content of the target application program to obtain target video data corresponding to the second screen rendering content.

[0030] In a second aspect, the embodiments of the present disclosure further provide a data recording device, including:

[0031] A determination module is configured to determine an audio-video source matched with a target subprogram in response to a media content recording request for the target subprogram running in a target application program.

[0032] An acquisition module is configured to acquire first audio-video data collected from a front end of the target subprogram in a target data transmission manner matched with attribute information of a client of the target application program in a case where the audio-video source includes the front end of the target subprogram.

[0033] A generation module is configured to generate media content recording data corresponding to the target subprogram based on the first audio-video data.

[0034] In a possible implementation, the acquisition module is further configured to:

[0035] acquire second audio-video data collected locally from a client corresponding to the target application program in a case where the audio-video source further includes the client;

[0036] The generation module, when generating the media content recording data corresponding to the target subprogram based on the first audio-video data, is configured to:

[0037] generate the media content recording data corresponding to the target subprogram based on the first audio-video data and the second audio-video data.

[0038] In a possible implementation, when determining the audio and video source matching the target subprogram, the determining module is configured to:

[0039] determine the video source matching the picture rendering attribute information of the target subprogram based on the recording type parameter carried in the target subprogram code and / or the picture rendering parameter used by the target subprogram.

[0040] In a possible implementation, the attribute information of the client includes system version information corresponding to the client.

[0041] When obtaining the first audio and video data collected from the front end of the target subprogram in the target data transmission mode matching the attribute information of the client of the target application program, the obtaining module is configured to:

[0042] when it is detected that the system version information corresponding to the client is within a target version range, control the front end to initiate an extensible markup language (XML) request carrying the first audio and video data, and based on the client, intercept the XML request and extract the first audio and video data from the XML request; and

[0043] when it is detected that the system version information corresponding to the client is outside a target version range, control the front end to initiate a hypertext transfer protocol secure (HTTPS) request carrying the first audio and video data, and in the client, intercept the HTTPS request by calling a first target interface and extract the first audio and video data from the HTTPS request.

[0044] In a possible implementation, the obtaining module is configured to obtain the video data in the first audio and video data according to the following steps:

[0045] call a second target interface to perform texture data collection on the first picture rendering content of the target subprogram, to obtain target video data corresponding to the first picture rendering content; wherein the picture rendering code of the target subprogram includes a target code injected in advance and used to indicate that the first picture rendering content can be subjected to texture data collection.

[0046] In a possible implementation, the obtaining module is configured to obtain the audio data in the first audio and video data according to the following steps:

[0047] in response to an audio playback request, create a target audio source node used to play audio, and generate an audio data collection node corresponding to the target audio source node; wherein the audio data collection node is configured to listen to an audio playback event of the target audio source node.

[0048] Based on the audio data collection node, target audio data corresponding to the target sound source node is acquired.

[0049] In a possible implementation, the number of target sound source nodes is multiple.

[0050] After the target audio data corresponding to the target sound source node is acquired based on the audio data collection node, the acquisition module is further configured to:

[0051] The target audio data corresponding to each target sound source node is obtained by performing segmentation processing on the target audio data corresponding to multiple target sound source nodes respectively.

[0052] The sub-audio data corresponding to the same time segment in multiple target audio nodes is mixed to obtain mixed audio data transmitted from the front end to the client.

[0053] In a possible implementation, the acquisition module is configured to acquire the video data in the second audio-video data according to the following steps:

[0054] The third target interface is called to collect texture data of the second screen rendering content of the target application program to obtain target video data corresponding to the second screen rendering content.

[0055] In a third aspect, the embodiments of the present disclosure further provide a computer device, including a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the computer device is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to perform the steps of the first aspect or any possible implementation manner of the first aspect.

[0056] In a fourth aspect, the embodiments of the present disclosure further provide a computer readable storage medium, and the computer readable storage medium stores a computer program, when the computer program is run by the processor, the steps of the first aspect or any possible implementation manner of the first aspect are executed.

[0057] The data recording method and device, computer device and storage medium provided by the embodiments of the present disclosure can, after responding to a media content recording request for a target subprogram running in a target application program, acquire first audio and video data collected from a front end of the target subprogram through a determined audio and video source matched with the target subprogram and a target data transmission mode matched with attribute information of a client of the target application program, and generate media content recording data corresponding to the target subprogram based on the first audio and video data. In this way, by determining the audio and video source matched with the target subprogram, only the audio and video data under the matched audio and video source is recorded, which can avoid recording some unnecessary audio and video data. In the case where the audio and video source includes a front end, the audio and video data of the front end can be transmitted to the client according to the matched target data transmission mode (in the case where the audio and video source is the local client, the audio and video data of the client can be directly collected), so as to realize recording of the audio and video data under different audio and video sources and avoid recording the audio and video data under some unnecessary audio and video sources.

[0058] In order to make the above objectives, characteristics and advantages of the present disclosure more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings needed to be used in the embodiments. The drawings herein are incorporated into the specification and form a part of the specification, which illustrate the embodiments consistent with the present disclosure, and are used to explain the technical solutions of the present disclosure together with the specification. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be considered as a limitation to the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0060] Figure 1 A flowchart of a data recording method provided by an embodiment of the present disclosure is shown;

[0061] Figure 2 A flowchart of determining a video source in a data recording process in a data recording method provided by an embodiment of the present disclosure is shown;

[0062] Figure 3 A schematic diagram of acquiring first audio and video data collected from a front end of the target subprogram in a data recording method provided by an embodiment of the present disclosure is shown;

[0063] Figure 4 A flowchart of acquiring target video data corresponding to first picture rendering content in a data recording method provided by an embodiment of the present disclosure is shown.

[0064] Figure 5 A flow chart of audio data collection and mixing processing in a data recording method provided by an embodiment of the present disclosure is shown;

[0065] Figure 6 A flow chart of processing target video data corresponding to second screen rendering content in a data recording method provided by an embodiment of the present disclosure is shown;

[0066] Figure 7 A flow chart of another data recording method in a data recording method provided by an embodiment of the present disclosure is shown;

[0067] Figure 8 An architectural schematic diagram of a data recording apparatus provided by an embodiment of the present disclosure is shown;

[0068] Figure 9 A structural schematic diagram of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0069] In order to make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The components of the embodiments of the present disclosure described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the claimed present disclosure, but only represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present disclosure.

[0070] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0071] The term "and / or" herein only describes an association relationship, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" herein means any one of multiple or any combination of at least two of multiple, for example, including at least one of A, B and C can mean including any one or more elements selected from the set consisting of A, B and C.

[0072] It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the type, use range, use scenario, etc. of the personal information involved in the present disclosure should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0073] For example, in response to receiving an active request of a user, prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using personal information of the user. Thus, the user can voluntarily choose whether to provide personal information to the software or hardware such as an electronic device, an application program, a server or a storage medium, etc. that performs the operation of the technical solutions of the present disclosure according to the prompt information.

[0074] As an optional but non-limiting implementation manner, in response to receiving an active request of a user, the manner of sending prompt information to the user may, for example, be a pop-up window manner, in which the prompt information can be presented in a text manner. In addition, the pop-up window can also carry selection controls for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0075] It can be understood that the above notification and obtaining of user authorization process is only illustrative and does not limit the implementation manners of the present disclosure, and other manners meeting relevant laws and regulations can also be applied to the implementation manners of the present disclosure.

[0076] It is found through research that when a subprogram runs in an application program, the audio and video data irrelevant to the subprogram in the application program will be recorded together when recording the audio and video data of the subprogram, resulting in recording of some audio and video data that is not needed by the user, and thus the actual recording needs of the user cannot be met.

[0077] For example, taking the iOS system as the operating system and the WKWebview environment as the program running environment, in the related art, the following scheme can be used when recording media content:

[0078] 1. Calling the MediaRecorder interface to record media content

[0079] In this scheme, due to compatibility and system version reasons, there may be incompatibility problems on the iOS system, and the ability to record audio is limited, such as being unable to achieve real-time recording.

[0080] 2. Media content segment restoration scheme based on Document Object Model (DOM)

[0081] In this scheme, only subprograms with a picture rendering manner of DOM can be supported for media content recording, and subprograms using other picture rendering manners are not supported.

[0082] 3. Client recording scheme

[0083] In this scheme, the drawViewHierarchyInRect interface or the takeSnapshotWithConfiguration interface of the application client can be called to realize video data acquisition of the client webview, and then the AVFoundation interface is called to write the video data into a screen recording file to complete screen content recording, but audio data cannot be acquired.

[0084] In addition, since the WKWebview opens hardware acceleration, the drawViewHierarchyInRect interface and the takeSnapshotWithConfiguration interface cannot acquire the screen content rendered by the front end of the subprogram below the iOS15.4 version, and the performance of the interface itself is poor.

[0085] 4. Recording using the ReplayKit recording tool

[0086] In this scheme, UI components outside the webview container may be recorded during recording, such as private messages or push messages in the application, thereby causing user privacy problems, and the storage path of the video file after recording is completed cannot be acquired by the developer below the iOS14 version, and the user can only share the video based on the UI interface provided by the system after recording is completed, and the interaction is limited.

[0087] In addition, using the ReplayKit recording tool for recording still cannot record audio data in the WKWebview running environment.

[0088] Based on the above research, the present disclosure provides a data recording method, device, computer equipment and storage medium. After responding to a media content recording request for a target subprogram running in a target application program, the first audio and video data collected from the front end of the target subprogram can be obtained through a determined audio and video source matched with the target subprogram and a target data transmission mode matched with the attribute information of the client of the target application program, and the media content recording data corresponding to the target subprogram can be generated based on the first audio and video data. In this way, by determining the audio and video source matched with the target subprogram, only the audio and video data under the matched audio and video source can be recorded, which can avoid recording some unnecessary audio and video data. In the case where the audio and video source includes the front end, the audio and video data of the front end can be transmitted to the client according to the matched target data transmission mode (in the case where the audio and video source is the local client, the audio and video data of the client can be directly collected), so as to realize the recording of the audio and video data under different audio and video sources and avoid recording the audio and video data under some unnecessary audio and video sources.

[0089] To facilitate the understanding of the present embodiment, first, a data recording method disclosed by the present embodiment is introduced in detail. The execution subject of the data recording method provided by the present embodiment is generally a computer equipment with certain computing capability, which includes, for example, a terminal equipment or a server or other processing equipment. The terminal equipment can be a smart terminal equipment with display function, for example, a smart phone, a tablet computer, a smart wearable device, etc. In some possible implementation manners, the data recording method can be realized by calling the computer readable instructions stored in the memory by the processor.

[0090] Referring to Figure 1 The flowchart of the data recording method provided by the present embodiment is shown. The method includes S101-S103, wherein:

[0091] S101: In response to a media content recording request for a target subprogram running in a target application program, determine an audio and video source matched with the target subprogram.

[0092] S102: In the case where the audio and video source includes the front end of the target subprogram, acquire first audio and video data collected from the front end of the target subprogram according to a target data transmission mode matched with the attribute information of the client of the target application program.

[0093] S103: Based on the first audio and video data, generate media content recording data corresponding to the target subprogram.

[0094] The following is a detailed introduction to the above steps.

[0095] For S101, the target application program can be an application program deployed on a target operating system, and the target subprogram can be a small program running on the target application program, with a target running environment; for example, the target operating system can be an iOS system, and the target running environment can be a WKWebview environment; for example, the target subprogram can be a game subprogram developed based on HTML5 technology; the audio and video source can include an audio source and / or a video source; for example, the audio source can represent the source of audio data in the media content recording process, and the audio source can include the front end of the target subprogram and / or the client corresponding to the target application program; for example, the audio data can be pulse code modulation (PCM) data; the video source can represent the source of video data in the media content recording process, and the video source can include the front end of the target subprogram and / or the client corresponding to the target application program; for example, the video data can be texture data of picture rendering content.

[0096] In a possible implementation, when determining the audio and video source matched with the target subprogram, the video source matched with the picture rendering attribute information of the target subprogram can be determined based on a recording type parameter carried in the target subprogram code and / or a picture rendering parameter used by the target subprogram.

[0097] Specifically, the recording type parameter can be a recordType parameter, and the parameter value of the recordType parameter can represent the video source when recording the media content of the target subprogram; for example, the parameter value of the recordType parameter can be a client of a target application program, to represent that the video source during recording is the client of the target application program; for example, the picture rendering parameter can be a canvas label used for picture rendering in HTML5, and the picture rendering mode of the target subprogram can be determined according to the type of the canvas label in the execution code corresponding to the target subprogram by analyzing the execution code; for example, the picture rendering mode can include 2d canvas and 3d canvas, and different picture rendering modes can correspond to different video sources.

[0098] In actual application, since the type of the canvas label can only be analyzed to determine the picture rendering mode using the canvas label, but the target subprogram can also not use the canvas label for picture rendering, for example, the DOM can be used for picture rendering.

[0099] Further, in order to ensure that the picture rendering mode of the target subprogram can be accurately determined to obtain more accurate picture rendering attribute information, when the video source matching the picture rendering attribute information of the target subprogram is determined based on the picture rendering attribute information of the target subprogram, the execution code of the front end of the target subprogram can be parsed first, and the picture rendering mode of the target subprogram is determined based on the parsing result and the picture rendering parameter corresponding to the target subprogram. The video source corresponding to the picture rendering mode of the target subprogram is determined based on the mapping relationship between the picture rendering mode and the video source pre-stored, and the determined video source is taken as the video source matching the picture rendering attribute information of the target subprogram.

[0100] The mapping relationship between the picture rendering mode and the video source can include 2d canvas-client, 3d canvas-front end, and DOM-client, which respectively represent that when the 2d canvas is used for picture rendering, the video source includes the client of the target application program, when the 3d canvas is used for picture rendering, the video source includes the front end of the target subprogram, and when the DOM is used for picture rendering, the video source includes the client of the target application program.

[0101] For example, the flowchart for determining the video source during data recording can be as shown in Figure 2 Figure 2 After the user clicks the client button to trigger the screen recording function, the front end can detect the types of all canvas labels. If the 3d canvas is used, the video source is the front end. If the 2d canvas and the 3d canvas are used at the same time, the video source is the front end and the client. If the 2d canvas is used or no canvas is used, the video source is the client. The recordType parameter is passed to the client, and the client combines the recordType parameter and the above judgment result to determine whether the client video acquisition source needs to be supplemented.

[0102] In this way, by combining the picture rendering parameter of the target subprogram and the parsing result of the execution code of the target subprogram, the picture rendering mode of the target subprogram can be accurately determined, so that the video source matching the picture rendering attribute information of the target subprogram can be further determined.

[0103] S102: In the case where the audio and video source includes the front end of the target subprogram, the first audio and video data collected from the front end of the target subprogram is acquired in the target data transmission mode matching the attribute information of the client of the target application program.

[0104] ​Here, the attribute information of the client can include system version information corresponding to the client of the target application, the system version information corresponding to the client being used to represent version information of an operating system of a terminal device on which the target application client is deployed; and the first audio and video data can include audio data and / or video data.

[0105] In a possible implementation, when the first audio and video data collected from the front end of the target subprogram is acquired in a target data transmission mode matching the attribute information of the client of the target application, the following two cases can be included:

[0106] Case 1: The system version information corresponding to the client is within a target version range

[0107] Here, when it is detected that the system version information corresponding to the client is within the target version range, the front end can be controlled to initiate an extensible markup language (XML) request carrying the first audio and video data, and the client can be controlled to intercept the XML request and extract the first audio and video data from the XML request.

[0108] For example, when the target operating system is an iOS system, the system version information corresponding to the client being within the target version range can mean that the iOS version corresponding to the client is greater than or equal to iOS 13, and the first audio and video data can be located in the XML request, and the first audio and video data can be acquired by intercepting and analyzing the XML request.

[0109] Case 2: The system version information corresponding to the client is outside the target version range

[0110] Here, when it is detected that the system version information corresponding to the client is outside the target version range, the front end can be controlled to initiate a hypertext transfer protocol secure (HTTPS) request carrying the first audio and video data, and the client can be controlled to intercept the HTTPS request by calling a first target interface and extract the first audio and video data from the HTTPS request.

[0111] For example, when the target operating system is an iOS system, the system version information corresponding to the client being outside the target version range can mean that the iOS version corresponding to the client is less than iOS 13, and the first audio and video data can be located in the HTTPS request, and the first audio and video data can be acquired by calling a handleURLScheme interface to enable the client to acquire the capability of intercepting the HTTPS request, and then calling a webView:startURLSchemeTask interface (that is, the first target interface) to intercept and analyze the HTTPS request.

[0112] For example, a schematic diagram of acquiring first audio-video data collected from the front end of the target subprogram can be as shown in Figure 3 , Figure 3 In this way, according to whether the iOS system version is greater than or equal to 13, two data transmission modes can be determined. When the iOS system version is greater than or equal to 13, XML request can be used for data transmission, and when the iOS system version is less than 13, HTTPS request can be used for data transmission, and the data transmission function is implemented by calling the handleURLScheme interface. The client can intercept the request through the webView:startURLSchemeTask interface to obtain the first audio-video data carried in the request.

[0113] In a possible implementation, when the video data in the first audio-video data is acquired, a second target interface can be called to collect texture data of the first picture rendering content of the target subprogram, to obtain target video data corresponding to the first picture rendering content. The picture rendering code of the target subprogram contains target code injected in advance for indicating that the first picture rendering content can be subjected to texture data collection.

[0114] Here, the second target interface can be a readPixels interface, which can be used to analyze the first picture rendering content rendered at the front end of the target subprogram, so as to obtain target video data corresponding to the first picture rendering content. The target code can be, for example, "preserveDrawingBuffer: true". By injecting the target code for indicating that the first picture rendering content can be subjected to texture data collection in the corresponding execution code of the front end of the target subprogram in advance before the first picture rendering content is rendered at the front end of the target subprogram, the target video data corresponding to the first picture rendering content can be acquired by calling the second target interface.

[0115] For example, taking the target subprogram as a game subprogram, a flowchart of acquiring target video data corresponding to the first picture rendering content can be as shown in Figure 4 , Figure 4In the game subprogram, after the game subprogram is started, a canvas label used by the game subprogram during running can be automatically found, and the drawArrays, drawElements, and getContext interfaces of webGL are hijacked to force the interface preserveDrawingBuffer to be true, that is, to inject "preserveDrawingBuffer: true". Then, a timer function animate is started to read pixel data according to a preset time interval, so that target video data corresponding to first screen rendering content is obtained, and the target video data is sent to the client.

[0116] In a possible implementation, when the audio data in the first audio and video data is acquired, the following steps A1-A2 can be performed:

[0117] A1: In response to an audio playing request, a target sound source node for playing audio is created, and an audio data acquisition node corresponding to the target sound source node is generated.

[0118] The target sound source node can be a MediaElementAudioSourceNode node for storing audio data, which is created based on a Web Audio standard, and the audio data acquisition node is used to listen to an audio playing event of the target sound source node. The audio data acquisition node can be a ScriptProcessorNode node, for example.

[0119] Specifically, after detecting that the target subprogram front end creates a target sound source node for playing audio in response to an audio playing request, a node code corresponding to a ScriptProcessorNode node can be injected into execution code corresponding to the target subprogram front end, so as to bind the ScriptProcessorNode node to the target sound source node.

[0120] A2: Target audio data corresponding to the target sound source node is acquired based on the audio data acquisition node.

[0121] Specifically, after the audio data acquisition node is bound to the corresponding target sound source node, the audio data acquisition node can acquire audio data of the target sound source node in each preset time period through callback after listening to the audio playing event of the target sound source node. The preset time period can be 3s, for example.

[0122] Further, in the case where the number of target sound source nodes is greater than one, after the target audio data corresponding to the target sound source nodes is acquired, the following steps A3-A4 can be further performed:

[0123] A3: performing a splitting processing on the target audio data corresponding to each of the target sound source nodes, to obtain a plurality of sub-audio data corresponding to each of the target sound source nodes.

[0124] Here, when performing the splitting processing on the target audio data, the splitting processing can be performed according to a preset audio time length splitting interval, which can be 1s for example.

[0125] A4: performing a mixing processing on the sub-audio data corresponding to the same time segment in the plurality of target audio nodes, to obtain mixed audio data transmitted from the front end to the client.

[0126] Here, in order to ensure that the time segments corresponding to each of the sub-audio data participating in the mixing processing are the same, a separate storage array can be used to store each of the sub-audio data when storing the sub-audio data, and each target sound source node corresponds to a storage array, so that when performing the mixing processing, only the first element (i.e. the first audio sub-data) in each storage array needs to be read for mixing processing.

[0127] In addition, in order to ensure that the audio data corresponds to audio content, content judgment can be performed according to the audio data corresponding to the target audio node, and when it is detected that the audio data is all "0" or other data representing no audio content, the currently detected target audio node can be regarded as a silent node and deleted.

[0128] Further, in order to improve the processing efficiency of the audio data, a separate audio processing thread can be set to process the collection and mixing processing of the audio data, so as to avoid occupying the computing resources of the main thread for executing the subprogram picture rendering and other tasks, and reduce the influence of the audio data processing process on the subprogram.

[0129] For example, still taking the target subprogram as a game subprogram, the flowchart of the audio data collection and mixing processing can be as shown in Figure 5 Figure 5 ​In this way, whether audio is currently being played can be determined by detecting whether a new sourceNode node is created, and when it is detected that a new sourceNode node is created, an additional audio data collection node ScriptProcessorNode node is created to listen to the play event of the sourceNode node in real time; when the play event is listened to, the PCM data of the sourceNode node can be called back through the onProcess function; the PCM data is saved and transmitted to a pre-set sub-thread for audio data processing; after receiving the PCM data, the sub-thread can sequentially perform splitting processing, normalization processing, and data format conversion processing on the PCM data, thereby completing the processing of the PCM data, and then the processed audio data is sent to the main thread, and the main thread sends the audio data to the client.

[0130] In a possible implementation, in the case that the audio and video source further includes a client corresponding to the target application, second audio and video data collected locally from the client can also be acquired.

[0131] Here, the second audio and video data can include audio data and / or video data.

[0132] In a possible implementation, when the video data in the second audio and video data is acquired, a third target interface can be called to collect texture data of second screen rendering content of the target application, to obtain target video data corresponding to the second screen rendering content.

[0133] Here, the third target interface can be a takeSnapshotWithConfiguration interface in a webview container of the target application; and the second screen rendering content is screen content rendered by the client of the target application.

[0134] Specifically, by calling the takeSnapshotWithConfiguration interface, initial video data (that is, texture data) in the webview container can be called back; further, after the initial video data is obtained, the initial video data can be processed in a format conversion manner, to convert the initial video data into target video data in a binary format (for example, an NSData format).

[0135] For example, a flowchart for processing target video data corresponding to the second screen rendering content can be as shown in Figure 6 Figure 6 ​In the embodiment, the client can initiate a client video data collection after intercepting a video data acquisition request sent by the front end. The video data (i.e., target video data corresponding to the second picture rendering content) in the webview container can be called back by calling the takeSnapshotWithConfiguration interface, the video data can be bound to a texture data node (texture node) for processing, and the video data can be stored in the cache area of the audio / video recorder.

[0136] In S103, the target subprogram corresponding media content recording data is generated based on the first audio / video data.

[0137] Here, after obtaining the first audio / video data, the obtained first audio / video data can be stored in the cache area of the audio / video recorder of the target application client, so as to generate corresponding audio / video content based on the audio / video generation request and the first audio / video data cached in the cache area of the audio / video recorder.

[0138] In a possible implementation, when the target subprogram corresponding media content recording data is generated based on the first audio / video data, the target subprogram corresponding media content recording data can be generated based on the first audio / video data and the second audio / video data.

[0139] Here, when the target subprogram corresponding media content recording data is generated based on the first audio / video data and the second audio / video data, the target video data in the first audio / video data and the second audio / video data can be transmitted to the audio / video recorder of the target application client. The audio / video recorder can bind the received target video data to a corresponding openGL texture node, so that the mixed shader can perform video mixing processing on the bound target video data according to the RGB value of the pixel point, so as to obtain the target video data after video mixing processing.

[0140] Further, after the target subprogram corresponding media content recording data is generated, the system current time of the application can be used as the write time stamp of the video frame in response to a play request for the video recording result in the media content recording data, the video data in the video cache area of the cache area of the audio / video recorder can be obtained, and the system encoder can be used for video encoding processing to obtain the recording video that can be used for video playing. In addition, the write time stamp of the audio frame can be determined according to the data length and the audio duration in the request in response to a play request for the audio recording result in the media content recording data, and the system encoder can be used for audio encoding processing of the audio data and the write time stamp to obtain the recording audio that can be used for audio playing.

[0141] Below, the data recording method provided by the embodiments of the present disclosure will be introduced in combination with specific embodiments, as shown in the following table: Figure 7 The data recording method provided by the embodiments of the present disclosure can include the following steps:

[0142] Step one, the user clicks the front-end button to trigger screen recording.

[0143] Step two, inform the front end to start screen recording, and the front end decides the video capture method.

[0144] Step three, the client initializes the recorder.

[0145] Step four, create a video encoder and an audio encoder respectively.

[0146] Step five, create an openGL context, a pbuffer, and compile and link a shader program.

[0147] Step six, start the screen recording process.

[0148] In the case of receiving video data, the current system time is taken as the write time stamp and is handed over to the video encoder for encoding; in the case of receiving audio data, the time stamp transmitted by the front end is taken as the write time stamp and is handed over to the audio encoder for encoding.

[0149] Step seven, judge whether the writing fails.

[0150] Here, the writing failure is the recording data writing failure, that is, the recording failure.

[0151] Specifically, if the writing fails, the front end is informed of the error and the screen recording is stopped; if the writing does not fail, the front end or the user can be waited for to trigger the stop of screen recording, and the video file address is returned so as to obtain the recorded video file.

[0152] The specific content of the above steps can refer to the related content in the above text, and will not be repeated here.

[0153] The data recording method provided by the embodiments of the present disclosure can, after responding to a media content recording request for a target subprogram running in a target application program, acquire first audio and video data collected from a front end of the target subprogram by determining an audio and video source matched with the target subprogram and a target data transmission mode matched with attribute information of a client of the target application program, and generate media content recording data corresponding to the target subprogram based on the first audio and video data. In this way, by determining the audio and video source matched with the target subprogram, only audio and video data under the matched audio and video source is recorded, which can avoid recording some unnecessary audio and video data. In the case where the audio and video source includes the front end, the audio and video data of the front end can be transmitted to the client according to the matched target data transmission mode (in the case where the audio and video source is the local client, the audio and video data of the client can be directly collected), so as to realize recording of audio and video data under different audio and video sources and avoid recording audio and video data under some unnecessary audio and video sources.

[0154] Those skilled in the art can understand that, in the above method of the specific implementation, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process, and the specific execution order of each step should be determined according to its function and possible internal logic.

[0155] Based on the same inventive concept, the embodiments of the present disclosure also provide a data recording device corresponding to the data recording method. Since the principle of solving problems of the device in the embodiments of the present disclosure is similar to the above-mentioned data recording method, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described here.

[0156] Referring to Figure 8 FIG. 1 shows an architecture schematic diagram of a data recording device provided by the embodiments of the present disclosure. The device includes a determining module 801, an acquiring module 802, and a generating module 803.

[0157] The determining module 801 is configured to, in response to a media content recording request for a target subprogram running in a target application program, determine an audio and video source matched with the target subprogram.

[0158] The acquiring module 802 is configured to, in the case where the audio and video source includes a front end of the target subprogram, acquire first audio and video data collected from the front end of the target subprogram according to a target data transmission mode matched with attribute information of a client of the target application program.

[0159] The generating module 803 is configured to generate media content recording data corresponding to the target subprogram based on the first audio and video data.

[0160] In a possible implementation, the acquisition module 802 is further configured to:

[0161] In the case where the audio and video source further includes a client corresponding to the target application program, acquire second audio and video data collected locally from the client;

[0162] The generation module 803 is configured to:

[0163] generate the media content recording data corresponding to the target subprogram based on the first audio and video data and the second audio and video data.

[0164] In a possible implementation, the determination module 801 is configured to:

[0165] determine a video source matching the picture rendering attribute information of the target subprogram based on a recording type parameter carried in the target subprogram code and / or a picture rendering parameter used by the target subprogram.

[0166] In a possible implementation, the attribute information of the client includes system version information corresponding to the client.

[0167] The acquisition module 802 is configured to:

[0168] In the case where it is detected that the system version information corresponding to the client is within a target version range, control the front end to initiate an extensible markup language (XML) request carrying the first audio and video data, and based on the client, intercept the XML request and extract the first audio and video data from the XML request; and

[0169] In the case where it is detected that the system version information corresponding to the client is outside a target version range, control the front end to initiate a secure hypertext transfer protocol (HTTPS) request carrying the first audio and video data, and in the client, call a first target interface to intercept the HTTPS request and extract the first audio and video data from the HTTPS request.

[0170] In a possible implementation, the acquisition module 802 is configured to acquire the video data in the first audio and video data according to the following steps:

[0171] The second target interface is called to collect texture data of the first picture rendering content of the target subprogram, to obtain target video data corresponding to the first picture rendering content; wherein the picture rendering code of the target subprogram contains target code injected in advance for indicating that the first picture rendering content can be subjected to texture data collection.

[0172] In a possible implementation, the obtaining module 802 is configured to obtain the audio data in the first audio-video data according to the following steps:

[0173] In response to an audio playing request, a target sound source node for playing audio is created, and an audio data collection node corresponding to the target sound source node is generated; wherein the audio data collection node is configured to listen to an audio playing event of the target sound source node.

[0174] Based on the audio data collection node, target audio data corresponding to the target sound source node is obtained.

[0175] In a possible implementation, the number of target sound source nodes is multiple.

[0176] After obtaining the target audio data corresponding to the target sound source node based on the audio data collection node, the obtaining module 802 is further configured to:

[0177] The target audio data corresponding to each of the multiple target sound source nodes is subjected to splitting processing, to obtain multiple sub-audio data corresponding to each target sound source node.

[0178] The sub-audio data corresponding to the same time segment in the multiple target audio nodes is subjected to mixing processing, to obtain mixed audio data transmitted from the front end to the client.

[0179] In a possible implementation, the obtaining module 802 is configured to obtain the video data in the second audio-video data according to the following steps:

[0180] A third target interface is called to collect texture data of the second picture rendering content of the target application program, to obtain target video data corresponding to the second picture rendering content.

[0181] The data recording apparatus provided by the embodiments of the present disclosure can, in response to a media content recording request for a target subprogram running in a target application program, acquire first audio and video data collected from a front end of the target subprogram through a determined audio and video source matched with the target subprogram and a target data transmission mode matched with attribute information of a client of the target application program, and generate media content recording data corresponding to the target subprogram based on the first audio and video data. In this way, by determining the audio and video source matched with the target subprogram and recording only the audio and video data under the matched audio and video source, recording of some unnecessary audio and video data can be avoided. In the case where the audio and video source includes the front end, the audio and video data of the front end can be transmitted to the client according to the matched target data transmission mode (in the case where the audio and video source is the local client, the audio and video data of the client can be directly collected), so as to realize recording of the audio and video data under different audio and video sources and avoid recording of the audio and video data under some unnecessary audio and video sources.

[0182] The processing flow of each module in the apparatus and the interaction flow between the modules can refer to the related description in the above method embodiments, and will not be described in detail here.

[0183] Based on the same technical concept, the embodiments of the present disclosure also provide a computer device. Referring to FIG. 9, Figure 9 The computer device 900 provided by the embodiments of the present disclosure includes a processor 901, a memory 902, and a bus 903. The memory 902 is used to store execution instructions, including an internal memory 9021 and an external memory 9022. The internal memory 9021 is also called an internal memory, used to temporarily store operation data in the processor 901 and exchange data with the external memory 9022 such as a hard disk. The processor 901 exchanges data with the external memory 9022 through the internal memory 9021. When the computer device 900 is running, the processor 901 and the memory 902 communicate through the bus 903, so that the processor 901 executes the following instructions:

[0184] In response to a media content recording request for a target subprogram running in a target application program, determining an audio and video source matched with the target subprogram;

[0185] In the case where the audio and video source includes a front end of the target subprogram, acquiring first audio and video data collected from the front end of the target subprogram according to a target data transmission mode matched with attribute information of a client of the target application program;

[0186] Generating media content recording data corresponding to the target subprogram based on the first audio and video data.

[0187] In a possible implementation, the instructions of the processor 901 further include the following:

[0188] In a case where the audio and video source further includes a client corresponding to the target application program, the second audio and video data collected locally from the client is acquired.

[0189] The media content recording data corresponding to the target subprogram is generated based on the first audio and video data, including:

[0190] The media content recording data corresponding to the target subprogram is generated based on the first audio and video data and the second audio and video data.

[0191] In a possible implementation, the instructions of the processor 901 include the following: the audio and video source matched with the target subprogram is determined based on the following steps:

[0192] The video source matched with the picture rendering attribute information of the target subprogram is determined based on a recording type parameter carried in the target subprogram code and / or a picture rendering parameter used by the target subprogram.

[0193] In a possible implementation, the instructions of the processor 901 include the following: the attribute information of the client includes system version information corresponding to the client.

[0194] The first audio and video data collected from the front end of the target subprogram is acquired in the target data transmission mode matched with the attribute information of the client of the target application program, including:

[0195] In a case where it is detected that the system version information corresponding to the client is located within a target version range, the front end is controlled to initiate an extensible markup language (XML) request carrying the first audio and video data, and the first audio and video data is extracted from the XML request based on the client intercepting the XML request; and

[0196] In a case where it is detected that the system version information corresponding to the client is located outside a target version range, the front end is controlled to initiate a secure hypertext transfer protocol (HTTPS) request carrying the first audio and video data, and the first audio and video data is extracted from the HTTPS request by calling a first target interface to intercept the HTTPS request in the client.

[0197] In a possible implementation, the instructions of the processor 901 include the following: the video data in the first audio and video data is acquired based on the following steps:

[0198] The second target interface is called to collect texture data of the first picture rendering content of the target subprogram, to obtain target video data corresponding to the first picture rendering content; wherein, the picture rendering code of the target subprogram contains target code injected in advance for indicating that the first picture rendering content can be subjected to texture data collection.

[0199] In a possible implementation, the instructions of the processor 901 acquire the audio data in the first audio-video data according to the following steps:

[0200] In response to an audio playing request, a target sound source node for playing audio is created, and an audio data collection node corresponding to the target sound source node is generated; wherein, the audio data collection node is configured to listen to an audio playing event of the target sound source node.

[0201] The target audio data corresponding to the target sound source node is acquired based on the audio data collection node.

[0202] In a possible implementation, the instructions of the processor 901 include that the number of the target sound source nodes is multiple.

[0203] After the target audio data corresponding to the target sound source node is acquired based on the audio data collection node, the method further includes:

[0204] The target audio data corresponding to each of the multiple target sound source nodes is subjected to splitting processing, to obtain multiple sub-audio data corresponding to each target sound source node.

[0205] The sub-audio data corresponding to the same time segment in the multiple target audio nodes is subjected to mixing processing, to obtain mixed audio data transmitted from the front end to the client.

[0206] In a possible implementation, the instructions of the processor 901 acquire the video data in the second audio-video data according to the following steps:

[0207] The third target interface is called to collect texture data of the second picture rendering content of the target application program, to obtain target video data corresponding to the second picture rendering content.

[0208] The disclosure also provides a computer readable storage medium, which stores a computer program. When the computer program is run by a processor, the steps of the data recording method described in the above method embodiments are executed. The storage medium can be a volatile or non-volatile computer readable storage medium.

[0209] The embodiment of the present disclosure further provides a computer program product carrying program codes, the program codes comprising instructions for executing the steps of the data recording method described in the above method embodiments, which can be specifically referred to the above method embodiments and will not be repeated here.

[0210] The computer program product can be specifically implemented by means of hardware, software or combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) or the like.

[0211] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system and device can refer to the corresponding process in the above method embodiments, which will not be repeated here. In several embodiments provided by the present disclosure, it should be understood that the disclosed system, device and method can be implemented by other means. The above-described device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interface, device or unit, which can be electrical, mechanical or other forms.

[0212] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment.

[0213] In addition, each functional unit in each embodiment of the present disclosure can be integrated into one processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0214] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present disclosure essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present disclosure. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0215] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present disclosure, used to illustrate the technical solutions of the present disclosure, and not to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art who is familiar with the technology in the art can still make modifications or easily think of changes to the technical solutions described in the foregoing embodiments, or make equivalent replacements to some of the technical features; and these modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A data recording method, characterized in that, The method includes: In response to a media content recording request for a target subroutine running in a target application, determine the audio and video sources that match the target subroutine; When the audio and video source includes the front end of the target subprogram, the first audio and video data collected from the front end of the target subprogram is obtained according to the target data transmission method that matches the attribute information of the client of the target application. Based on the first audio and video data, media content recording data corresponding to the target subroutine is generated; The client's attribute information includes the system version information corresponding to the client; The step of acquiring the first audio and video data collected from the front end of the target subroutine according to the target data transmission method matching the attribute information of the client of the target application includes: Upon detecting that the system version information corresponding to the client is within the target version range, the front-end is controlled to initiate an Extensible Markup Language (XML) request carrying the first audio and video data, and the client intercepts the XML request to extract the first audio and video data from the XML request; and, If the system version information corresponding to the client is detected to be outside the target version range, the front end is controlled to initiate a secure hypertext transfer protocol (HTTPS) request carrying the first audio and video data, and the client intercepts the HTTPS request by calling the first target interface and extracts the first audio and video data from the HTTPS request.

2. The method according to claim 1, characterized in that, The method further includes: If the audio and video source also includes a client corresponding to the target application, then acquire the second audio and video data collected locally from the client. The step of generating media content recording data corresponding to the target subroutine based on the first audio and video data includes: Based on the first audio / video data and the second audio / video data, media content recording data corresponding to the target subroutine is generated.

3. The method according to claim 1, characterized in that, The process of determining the audio and video source that matches the target subroutine includes: Based on the recording type parameters carried in the target subroutine code and / or the screen rendering parameters used by the target subroutine, determine the video source that matches the screen rendering attribute information of the target subroutine.

4. The method according to claim 1, characterized in that, The video data in the first audio and video data is obtained according to the following steps: The second target interface is called to collect texture data of the first screen rendering content of the target subroutine to obtain target video data corresponding to the first screen rendering content; wherein, the screen rendering code of the target subroutine contains pre-injected target code to indicate that the first screen rendering content can be subjected to texture data collection.

5. The method according to claim 1, characterized in that, The audio data in the first audio and video data is obtained according to the following steps: In response to an audio playback request, a target audio source node for playing audio is created; and an audio data acquisition node corresponding to the target audio source node is generated; wherein the audio data acquisition node is used to listen for audio playback events of the target audio source node; Based on the audio data acquisition node, the target audio data corresponding to the target audio source node is obtained.

6. The method according to claim 5, characterized in that, The number of target sound source nodes is multiple; After acquiring the target audio data corresponding to the target audio source node based on the audio data acquisition node, the method further includes: The target audio data corresponding to the multiple target audio source nodes are segmented to obtain multiple sub-audio data corresponding to each target audio source node; The sub-audio data corresponding to the same time segment from multiple target audio source nodes are mixed to obtain mixed audio data transmitted from the front end to the client.

7. The method according to claim 2, characterized in that, Obtain the video data from the second audio and video data according to the following steps: The third target interface is invoked to collect texture data of the second screen rendering content of the target application, thereby obtaining target video data corresponding to the second screen rendering content.

8. A data recording device, characterized in that, include: The determination module is used to determine the audio and video sources that match the target subroutine in response to a media content recording request for a target subroutine running in the target application; The acquisition module is used to acquire first audio and video data collected from the front end of the target subprogram according to the target data transmission method that matches the attribute information of the client of the target application when the audio and video source includes the front end of the target subprogram; The generation module is used to generate media content recording data corresponding to the target subroutine based on the first audio and video data; The client's attribute information includes the system version information corresponding to the client; The step of acquiring the first audio and video data collected from the front end of the target subroutine according to the target data transmission method matching the attribute information of the client of the target application includes: If the system version information corresponding to the client is detected to be within the target version range, the front end is controlled to initiate an Extensible Markup Language (XML) request carrying the first audio and video data, and the client intercepts the XML request to extract the first audio and video data from the XML request. as well as, If the system version information corresponding to the client is detected to be outside the target version range, the front end is controlled to initiate a secure hypertext transfer protocol (HTTPS) request carrying the first audio and video data, and the client intercepts the HTTPS request by calling the first target interface and extracts the first audio and video data from the HTTPS request.

9. A computer device, characterized in that, include: The computer device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the data recording method as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the data recording method as described in any one of claims 1 to 7.

Images