Media data processing methods, apparatus, computer equipment and storage media

Abstract

The present application relates to a media data processing method, device, computer equipment, storage medium and computer program product. The method comprises: according to the target call logic matching the operating system for running the Unreal Engine, calling the media data acquisition device to collect data and obtain real scene data; obtaining the media data file; recording the virtual interactive scene presented in the virtual interactive application running on the Unreal Engine to obtain virtual scene data; determining the media data to be processed corresponding to the target business, the media data to be processed includes at least one of the real scene data, the media data file and the virtual scene data; performing corresponding business processing on the media data to be processed and obtaining the business processing result corresponding to the target business. The method can be adapted to different platforms and meet diverse business needs.

Description

Technical Field

[0001] The present application relates to the field of computer processing technology, and in particular to a media data processing method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of technology in the field of game development, the application of Unreal Engine is becoming more and more common. In order to achieve different functions, game developers usually carry out targeted development based on the operating platform of Unreal Engine.

[0003] Currently, all the existing plug-ins in Unreal Engine are targeted at one platform, which means that the use of plug-ins is limited to the operating system and may not work properly on other platforms, such as failing to collect data properly. If you want to obtain data from different sources, game developers must develop and install corresponding plug-ins in a targeted manner to achieve compatibility between the plug-ins and the system, and spend a lot of manpower and material resources on maintenance, which limits the application scenarios of virtual engine plug-ins and cannot meet diverse business needs. Summary of the invention

[0004] Based on this, it is necessary to provide a media data processing method, apparatus, computer equipment, computer-readable storage medium and computer program product to address the above technical issues.

[0005] In one aspect, the present application provides a method for processing media data. The method comprises:

[0006] In the event of a data collection instruction, the media data collection device is called to collect data according to the target call logic matching the operating system for the Unreal Engine to run, so as to obtain real scene data;

[0007] In case of a data read instruction, obtaining a media data file;

[0008] In the event of a data recording instruction, a virtual interactive scene presented in a virtual interactive application running on the Unreal Engine is recorded to obtain virtual scene data;

[0009] Determine the to-be-processed media data corresponding to the target service, wherein the to-be-processed media data includes at least one of the following: the real scene data, the media data file, and the virtual scene data;

[0010] Corresponding business processing is performed on the media data to be processed to obtain a business processing result corresponding to the target business.

[0011] On the other hand, the present application also provides a media data processing device. The device includes:

[0012] The acquisition module is used to call the media data acquisition device to collect data and obtain real scene data according to the target call logic matching the operating system for running the Unreal Engine when a data acquisition instruction occurs;

[0013] A reading module, used for obtaining a media data file when a data reading instruction occurs;

[0014] A recording module, used for recording a virtual interactive scene presented in a virtual interactive application running on the Unreal Engine to obtain virtual scene data when a data recording instruction is issued;

[0015] A determination module, configured to determine the to-be-processed media data corresponding to the target service, wherein the to-be-processed media data includes at least one of the following: the real scene data, the media data file, and the virtual scene data;

[0016] The processing module is used to perform corresponding business processing on the media data to be processed to obtain a business processing result corresponding to the target business.

[0017] On the other hand, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned media data processing method when executing the computer program.

[0018] On the other hand, the present application further provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above-mentioned media data processing method when executed by a processor.

[0019] On the other hand, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned media data processing method are implemented.

[0020] The above-mentioned media data processing method, device, computer equipment, storage medium and computer program product, by calling the media data acquisition device to collect data according to the target calling logic matching the operating system to obtain real scene data, can be adapted to different operating systems and platforms, without the need to develop plug-ins for specific platforms, and have higher applicability; at the same time, in response to data reading instructions, media data files are obtained, and in response to data recording instructions, virtual interactive scenes presented in virtual interactive applications running on the Unreal Engine are recorded to obtain virtual scene data. Therefore, by obtaining media data from different sources and selecting at least one of the various media data as the media data to be processed corresponding to the target business, and performing corresponding business processing on the media data to be processed according to the business needs of the target business, thereby obtaining a business processing result corresponding to the target business, media data from multiple sources can be flexibly processed according to actual business needs, diversified business needs can be met, and the applicable scenarios are more extensive. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is an application environment diagram of a media data processing method in an embodiment;

[0022] Figure 2 is a flow chart of a media data processing method in one embodiment;

[0023] Figure 2A A schematic diagram of obtaining real scene data in one embodiment;

[0024] Figure 3 is a schematic diagram of an overall framework in one embodiment;

[0025] Figure 4A A schematic diagram of a detection service in an embodiment;

[0026] Figure 4B is a schematic diagram of format conversion in one embodiment;

[0027] Figure 4C A schematic diagram of specific steps of detecting a service in an embodiment;

[0028] Figure 5A is a schematic diagram of a video encoding service in one embodiment;

[0029] Figure 5B A schematic diagram of specific steps of a video encoding service in an embodiment;

[0030] Figure 5C A schematic diagram of a module of the PBO principle in one embodiment;

[0031] Figure 5D A schematic diagram of a module of the PBO principle in another embodiment;

[0032] Figure 6 is a schematic diagram of a playback service in an embodiment;

[0033] Figure 7 A schematic diagram of the overall framework of a media data processing method in one embodiment;

[0034] Figure 8 is a structural block diagram of a media data processing device in one embodiment;

[0035] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0037] The media data processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other servers. Among them, the terminal 102 calls the media data acquisition device through the plug-in to collect data to obtain real scene data, obtains media data files through the plug-in, and captures the virtual interactive scene presented in the virtual interactive application through the plug-in to obtain virtual scene data. According to at least one of the real scene data, media data files and virtual scene data, the terminal determines the media data to be processed corresponding to the target business, and performs corresponding business processing on the media data to be processed, and obtains the business processing result corresponding to the target business. Among them, the media data file can be obtained by the terminal 102 downloading and transmitting through the server 104.

[0038] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc.

[0039] Among them, server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0040] In some embodiments, an APP (Application) developed based on the Unreal Engine is loaded on the terminal. When the terminal runs the application, it drives the Unreal Engine to realize picture display and audio playback.

[0041] Among them, Unreal Engine is a game engine, including multiple versions, and the 4th generation version, namely UE4, is currently widely used. Game engine refers to a series of core components of editable game systems or interactive real-time image applications that have been written, so that game developers can use the tools provided by the editable game system to quickly write game programs. Editable game systems include, for example, renderers, physics engines, collision detection systems, sound effects, script engines, computer animations, and network engines. Unreal Engine can run in different operating systems. Operating system (Operating System) refers to a computer program configured in a terminal that manages computer hardware and software resources, including but not limited to Windows operating system, macOS operating system, Linux operating system, iOS operating system, Android operating system, and iPad OS operating system.

[0042] Unreal Engine has a modular system, custom plugins, and source control sets pre-set. Game developers need to develop on the basis of the code framework provided by Unreal Engine. In order to achieve different functions, various services that can be applied are usually provided in the form of plugins. Among them, plugins are a collection of code and data that game developers can enable or disable on a project-by-project basis in the Editor. Plugins can add runtime modifications to built-in engine functions or add functions, create new file types, and extend the functionality of the Editor with new menus, toolbar commands, and sub-modes.

[0043] Since the code framework provided by Unreal Engine can be applied to different platforms (including mobile phones, computers, and consoles, etc.), and different platforms have their own operating systems and operating logic, game developers will develop specifically according to the platform used, which results in the current plug-ins being only applicable to one operating system. For example, the Media Framework plug-in of Unreal Engine can only be applied to Windows systems, and cannot obtain system permissions in the macOS operating system to obtain camera usage permissions.

[0044] For cross-platform game projects, on the one hand, most plug-ins do not support secondary development. For example, Arkit (a plug-in that can be used to create augmented reality applications) provides the ability to obtain the underlying camera stream of the iOS operating system, but it does not allow related plug-ins to be secondary developed and is not applicable to other operating systems. On the other hand, a lot of modifications need to be made to the code logic of these plug-ins, which requires a huge amount of work and time to carry out related development and testing such as plug-in compatibility and plug-in splicing. Furthermore, maintaining the code of different plug-ins on different platforms is prone to errors and has poor maintainability.

[0045] Therefore, the embodiment of the present application provides a media data processing method to solve the above problems. The media data processing method provided in the embodiment of the present application can be integrated in an audio and video suite, and the audio and video suite can be run by a terminal to connect to multiple platforms and perform diversified business functions.

[0046] In some embodiments, the audio and video suite may be a plug-in, and the steps of the above-mentioned media data processing method are integrated into the code of the plug-in. In other embodiments, the audio and video suite may include multiple plug-ins, and each plug-in may perform data transmission and plug-in multiplexing between each other, and each plug-in jointly executes the steps of the above-mentioned media data processing method.

[0047] In one embodiment, Figure 2 As shown, a media data processing method is provided, which can be applied to Figure 1 The terminal or server in the example can also be executed by the terminal and the server in collaboration. Figure 1 The terminal in the example is used to illustrate that the terminal executes the media data processing method through the audio and video suite it runs to realize the cross-platform service function. Specifically, the following steps are included:

[0048] Step S202, when a data collection instruction occurs, the media data collection device is called to collect data according to the target calling logic matching the operating system for the Unreal Engine to run, so as to obtain real scene data.

[0049] The data collection instruction is used to instruct the media data collection device configured in the terminal to collect data of one or more media data. The media data collection device refers to a device provided by the terminal hardware for collecting media data, including but not limited to one or more of a microphone, a camera, etc. The media data includes but is not limited to image data and audio data, etc. The media data can also be video data, which can be composed of one of the image data or the audio data, or composed of both the image data and the audio data.

[0050] The calling logic refers to a series of operation processes to realize the corresponding functions. For example, the calling logic includes a series of operation processes including calling the system interface, applying for system permissions through the system interface, and calling the corresponding hardware device after obtaining the system permissions.

[0051] For the collection of media data, the terminal can initiate a system permission application through the system interface of the audio and video suite, thereby obtaining the use permission of the corresponding media data collection device.

[0052] In the embodiment of the present application, the system interface of the audio and video suite encapsulates the calling logic of the media data acquisition device of various operating systems, thereby being able to adapt to various operating systems. The terminal can invoke the corresponding hardware to collect local media data by running the audio and video suite.

[0053] For the sake of distinction, the media data collected by the media data collection device provided by the terminal hardware is referred to as real scene data.

[0054] Specifically, when a data collection instruction occurs, the terminal responds to the data collection instruction, determines the currently running operating system, and based on the operating system, selects the target calling logic that matches the currently running operating system from multiple sets of calling logics encapsulated in the audio and video suite that correspond to different operating systems, and initiates a call through the system interface of the audio and video suite. The audio and video suite executes the target calling logic, thereby calling the media data collection device provided by the terminal hardware to perform data collection and obtain real scene data.

[0055] Exemplarily, the data acquisition instruction instructs the acquisition of image data and audio data. The terminal responds to the data acquisition instruction, determines that the currently running operating system is the Windows operating system, and initiates system permission acquisition through the system interface of the audio and video suite. The audio and video suite executes the target call logic that matches the Windows operating system, thereby obtaining the system permissions of the Windows operating system, and calling the camera and microphone for data acquisition.

[0056] like Figure 2AAs shown, the terminal invokes a request through the audio and video suite, calls the camera and microphone to obtain the camera image and audio data, and obtains the real scene data, which is subsequently loaded into the audio and video suite to complete the processing of the target business.

[0057] Therefore, the audio and video suite provides calling logic that is compatible with various operating systems, and encapsulates multiple sets of calling logic in the system interface, so that the user is unaware of the underlying operating process when running the audio and video suite, and permission acquisition can be completed by simply calling the system interface, which makes the operation convenient and lowers the threshold for use.

[0058] Step S204: when a data read instruction occurs, obtain the media data file.

[0059] The data read instruction is used to instruct the terminal to read a media data file. A media data file is media data encapsulated in a specific file format. The process of encapsulating media data is usually the process of packaging and merging video compression data and audio compression data into one file. The file format includes but is not limited to one or more of AVI (Audio Video Interleave) format, WMV (Windows Media Video) format, and MPEG (Moving Picture Experts Group) format.

[0060] Specifically, in the case of a data reading instruction, the terminal responds to the data reading instruction and reads a pre-stored media data file from the local storage space, or reads a media data file received in real time and transmitted by other terminals or servers. For example, the media data file can be pre-downloaded by the terminal and stored in the local storage space. For another example, the terminal responds to the data reading instruction and establishes a communication connection with other terminals or servers, and reads the media data file transmitted by other terminals or servers.

[0061] Step S206, when a data recording instruction occurs, the virtual interactive scene presented in the virtual interactive application running based on the Unreal Engine is recorded to obtain virtual scene data.

[0062] The data recording instruction is used to instruct the terminal to record the images and sounds of the currently running virtual interactive application. Virtual interactive applications refer to applications running on the Unreal Engine, such as game programs. During the operation of the virtual interactive application, a virtual interactive scene will be presented. The virtual interactive scene consists of images and sounds, which are used for the operator to watch the plot, control the character to perform various actions, etc.

[0063] The data obtained by recording the virtual interactive scene includes image data and audio data. In order to distinguish it from the real scene data collected by the terminal by calling the media data collection device, the data provided by the virtual interactive application is called virtual scene data.

[0064] Specifically, when a data recording instruction occurs, the terminal responds to the data recording instruction and records the virtual interactive scene presented in the virtual interactive application running in the operating system to obtain virtual scene data. For example, when the terminal runs the virtual interactive application, the screen displays the image and the speaker plays the sound, then the terminal obtains the virtual scene data by collecting the image displayed on the screen and the audio played by the speaker.

[0065] Exemplarily, the terminal can obtain virtual scene data by calling a media acquisition device to record. Alternatively, the terminal can also record by calling an audio and video suite. In some embodiments, the terminal can obtain virtual scene data through a Media Framework plug-in provided by Unreal Engine.

[0066] Step S208: determining the to-be-processed media data corresponding to the target service, where the to-be-processed media data includes at least one of real scene data, media data files, and virtual scene data.

[0067] Among them, according to different business requirements, it is necessary to process media data from different sources to achieve corresponding business functions. Therefore, specifically, before specifically processing the media data, the terminal determines the current target business and determines the media data to be processed corresponding to the target business. The media data to be processed can be a single type of media data in real scene data, media data files, and virtual scene data, or a combination of two of them, or a combination of three types of media data, etc.

[0068] For example, when it is necessary to execute the function of recording a game scene, the media data to be processed includes at least virtual scene data, and on this basis, it can also include virtual scene data and real scene data according to actual needs, or it can include virtual scene data, real scene data and media data files. For another example, when it is necessary to execute the function of playing audio and video in the game, the media data to be processed includes at least media data files, and on this basis, it can also include at least one of the other two sources of media data according to actual needs. For another example, when it is necessary to execute the function of customized data transformation, the required media data to be processed includes at least real scene data, and on this basis, it can also include real scene data and media data files according to actual needs, or it can include media data from all three sources, etc.

[0069] Step S210: performing corresponding service processing on the media data to be processed, and obtaining a service processing result corresponding to the target service.

[0070] Among them, the processing flow for performing specific business processing on the media data to be processed is different according to the target business. Specifically, the terminal runs the audio and video suite to perform corresponding business processing on the media data to be processed according to the determined target business and the pre-set processing flow that matches the target business, thereby obtaining a business processing result corresponding to the target business. Still using the above example to illustrate, for example, for the target business of game scene recording, the business processing result obtained by the terminal performing business processing on the media data to be processed is the encoded recording file.

[0071] It should be noted that the service processing result is the final processing result of the target service, and the final processing result can be used as the input of other target services. For example, the recording file in the above example is subsequently stored, played, and transmitted by the terminal.

[0072] For example, Figure 3 As shown, the terminal loads the acquired media data to be processed into the audio and video suite, and completes the business processing of the media data to be processed by running the audio and video suite according to different target businesses. For example, for a target business for algorithm detection, the terminal converts the format of the media data to be processed by running the audio and video suite, and performs corresponding algorithm detection on the data after format conversion, such as image detection and voice detection, so as to obtain the algorithm detection result as the business processing result, which can be used for other subsequent target businesses and as input, such as as a training sample of a neural network input into the neural network to be trained, etc.

[0073] For another example, for the target service of playing audio and video within a game, the terminal obtains the media data to be processed including the media data file, and converts the media data to be processed into a data format supported by the Unreal Engine by running the audio and video suite, and combines it with the visualization controls provided by the Unreal Engine to make the media data to be processed played in the virtual interactive application running on the terminal, for example, displaying the image data of the media data to be processed in the virtual interactive scene provided by the virtual interactive application, etc.

[0074] For example, for the target business of game scene recording, after obtaining the media data to be processed including virtual scene data and media data files, the terminal performs business processing on the media data to be processed by running the audio and video suite, obtains the encoding status and encodes the video data and audio data in the media data to be processed, thereby obtaining the encoded recording file.

[0075] The above-mentioned media data processing method, device, computer equipment, storage medium and computer program product, by calling the media data acquisition device to collect data according to the target calling logic matching the operating system to obtain real scene data, can be adapted to different operating systems and platforms, without the need to develop plug-ins for specific platforms, and have higher applicability; at the same time, in response to data reading instructions, media data files are obtained, and in response to data recording instructions, virtual interactive scenes presented in virtual interactive applications running on the Unreal Engine are recorded to obtain virtual scene data. Therefore, by obtaining media data from different sources and selecting at least one of the various media data as the media data to be processed corresponding to the target business, and performing corresponding business processing on the media data to be processed according to the business needs of the target business, thereby obtaining a business processing result corresponding to the target business, media data from multiple sources can be flexibly processed according to actual business needs, diversified business needs can be met, and the applicable scenarios are more extensive.

[0076] The audio and video kit may be provided with multiple interfaces for implementing different functions, such as data transmission through interfaces. In order to achieve convenient and fast docking with various platforms and adapt to various operating systems, the audio and video kit is provided with a general data acquisition interface, so that the terminal hardware can be quickly aroused to perform data acquisition by calling the general data acquisition interface. To this end, in some embodiments, according to the target call logic matching the operating system for the Unreal Engine to run, the media data acquisition device is called to perform data acquisition to obtain real scene data, including: determining the general data acquisition interface, the general data acquisition interface encapsulates the call logic adapted to the media data acquisition devices corresponding to multiple different operating systems; determining the target call logic encapsulated in the general data acquisition interface and matching the system type of the operating system; based on the target call logic, the media data acquisition device is called through the general data acquisition interface to perform data acquisition to obtain real scene data.

[0077] Specifically, the data acquisition general interface provided in the audio and video suite encapsulates the call logic of various operating systems it supports. The call logic is used to call the terminal hardware when executed, such as waking up the camera or microphone. The terminal calls the data acquisition general interface of the audio and video suite and selects a set of call logics that matches the operating system from multiple sets of call logics as the target call logic according to the current operating system. The audio and video suite running on the terminal executes the operation process indicated by the target call logic, and calls the media data acquisition device through the data acquisition general interface to collect data, thereby obtaining real scene data.

[0078] In the above embodiment, by encapsulating multiple sets of calling logic in the data acquisition general interface, the specific target calling logic to be used is determined according to the currently configured operating system at runtime, thereby being able to connect to different platforms and adapt to various operating systems, and having wide applicability.

[0079] The task of data collection is mainly to obtain the calling authority of the terminal hardware. In some embodiments, based on the target calling logic, the media data collection device is called to collect data to obtain real scene data, including: based on the target calling logic, initiating a permission acquisition request to the operating system; when the calling authorization is obtained according to the permission acquisition request, calling the media data collection device to collect data; receiving the real scene data collected by the media data collection device.

[0080] Specifically, the terminal executes the target call logic through the audio and video suite, and the audio and video suite initiates a permission acquisition request to the current operating system, which is used to obtain the calling permission of the media data acquisition device configured by the terminal. After the operating system authorizes, the audio and video suite obtains the calling permission and calls the media data acquisition device to collect data.

[0081] Exemplarily, the terminal requests the operating system to obtain permission to use the microphone through the audio and video suite, and calls the microphone to collect audio data after obtaining authorization. Alternatively, the terminal requests the operating system to obtain permission to use the camera through the audio and video suite, and calls the camera to collect image data or video data after obtaining authorization.

[0082] In the above embodiment, by encapsulating multiple sets of call logic in the data acquisition general interface, the specific target call logic to be used is determined according to the currently configured operating system at runtime, thereby being able to connect to different platforms and adapt to various operating systems, and having a wide range of applicability. At the same time, system permissions need to be obtained before calling the media data acquisition device, which has a certain degree of security.

[0083] In order to improve the overall processing efficiency and optimize the performance of the audio and video suite, in some embodiments, acceleration processing is performed during the data loading of the audio and video suite, and the business processing of the audio and video suite for different target businesses (such as customized data processing, decoding, and encoding, etc.), so as to further improve the business processing efficiency of the audio and video suite.

[0084] For example Figure 3 As shown, for example, when the terminal loads the to-be-processed media data into the audio and video suite, the data loading process can be accelerated. To this end, in addition to the general data acquisition interface for adapting to different platforms mentioned in the above embodiment, the audio and video suite is also provided with an acceleration interface, which is used to increase the speed during the data loading process to speed up data transmission.

[0085] Accordingly, in some embodiments, before performing corresponding business processing on the media data to be processed and obtaining a business processing result corresponding to the target business, the above method also includes: determining an acceleration interface that matches the operating system; performing corresponding decoding processing on the media data to be processed through the acceleration interface that matches the operating system, and executing the step of performing corresponding business processing on the media data to be processed based on the decoded media data to be processed.

[0086] The acceleration interface is pre-set with interface usage specifications adapted to different operating systems, so that after receiving the media data to be processed, the audio and video suite does not need to perform further identification or conversion, thereby speeding up the data loading speed. The interface usage specifications include, for example, interface parameters, data formats, and communication protocols used by the operating system.

[0087] Specifically, the terminal calls the acceleration interface of the audio and video suite to load the media data to be processed into the audio and video suite, and the audio and video suite decodes the media data to be processed, thereby performing corresponding business processing based on the decoded media data to be processed to obtain a business processing result. Thus, by accelerating the process of loading the media data to be processed into the audio and video suite, the efficiency of data transmission is improved, thereby improving the processing efficiency of the target business.

[0088] In order to better understand the technical concept of the present invention, the media data processing method provided by the embodiment of the present application is explained below in a plurality of different application scenarios.

[0089] by Figure 4A For example, in some embodiments, the target service includes a detection service, and the detection service is used to perform algorithm detection on the media data to be processed loaded into the audio and video suite. In the detection service, the terminal performs algorithm detection on the media data to be processed by running the audio and video suite, such as image detection and voice detection, so as to obtain the algorithm detection result as the service processing result, and the service processing result can be used for other subsequent target services and as input, for example, as a training sample of a neural network input into the neural network to be trained, etc. In other words, the media data to be processed corresponding to the target service may include at least one of real scene data and media data files.

[0090] Accordingly, corresponding business processing is performed on the media data to be processed to obtain a business processing result corresponding to the target business, including: determining a target format that is suitable for the target detection algorithm; converting the data format of the media data to be processed into the target format; performing image detection on the video data to be processed in the media data to be processed based on the target detection algorithm to obtain a video detection result, and performing audio detection on the audio data to be processed in the media data to be processed to obtain an audio detection result; based on the video detection result and the audio detection result, determining the business processing result corresponding to the target business.

[0091] For real scene data, different operating systems use different encoding methods to obtain image data through cameras: some operating systems use YUV (a color encoding method) format as the encoding method, while some platforms use RGB (a color mode) format as the encoding method. However, the data format used as input for the detection algorithm is often unified. Therefore, it is necessary to convert the format of the input media data into the format supported by the detection algorithm, that is, the target format.

[0092] Specifically, according to the current detection service, the terminal performs data preprocessing on the media data to be processed loaded into the audio and video suite, and first converts the data format of the media data to be processed into a target format suitable for the target detection algorithm. Figure 4B As shown, the terminal performs format conversion on the media data to be processed, thereby converting the media data to be processed from an original format to a target format adapted to the target detection algorithm.

[0093] After the format conversion, the terminal runs the target detection algorithm through the audio and video suite, performs image detection on the video data to be processed in the media data to be processed, and obtains the video detection result, which includes but is not limited to the detection results of the target object image, the background image, etc. Among them, according to the different uses of the algorithm, the relevant detection algorithm of image detection includes but is not limited to one or more of the image recognition algorithm, the target detection algorithm, the threshold segmentation algorithm, and the action detection algorithm. At the same time, the audio data to be processed in the media data to be processed is subjected to audio detection to obtain the audio detection result, and the video detection result includes but is not limited to the human voice, the environmental sound, and the music sound in the extracted audio data. Among them, according to the different uses of the algorithm, the relevant detection algorithm of audio detection includes but is not limited to one or more of the speech emotion recognition algorithm, the audio rhythm detection algorithm, the semantic detection algorithm, and the noise detection algorithm. According to the video detection result and the audio detection result, the terminal can determine the business processing result corresponding to the detection business.

[0094] For example, Figure 4CAs shown, taking the example of the media data to be processed including real scene data and media data files, the terminal separates the audio and video in the real scene data and the media data files respectively to obtain the native audio data and native image data of the real scene data, and obtains the file video data and file audio data in the media data files.

[0095] In this detection service, the audio and video suite preprocesses the data, and the preprocessing includes format conversion of the data so that the format of the data input into the detection algorithm conforms to the format specified by the detection algorithm. Specifically, the audio and video suite converts the input data according to the target format specified by the algorithm protocol to obtain video algorithm data and audio algorithm data. The image detection algorithm then performs algorithm detection on the video algorithm data to obtain a video detection result; the voice detection algorithm performs audio detection on the audio algorithm data to obtain an audio detection result; and thus obtains an algorithm detection result. This algorithm detection result can be used for other subsequent target services.

[0096] In the above embodiment, the native audio and video input is converted into data that matches the audio and video algorithm protocol, and then used as the algorithm input for algorithm detection, which can be further processed later to achieve customized transformation of the data. At the same time, different algorithms can be used for detection according to business needs, and the algorithms can be replaced and updated in real time, which is more flexible to use and can meet different business needs.

[0097] In order to further improve the processing efficiency, Figure 4B As shown, acceleration processing is also performed during the data format conversion process. To this end, in some embodiments, the data format of the to-be-processed media data is converted into a target format, including: starting a child thread, determining the original format of the to-be-processed media data through the child thread, and converting the to-be-processed media data from the original format to the target format through a logical conversion function; after the child thread completes the conversion operation, notifying the main thread that the format conversion has been completed.

[0098] Specifically, the terminal creates a child thread and starts the child thread to perform the format conversion process. The child thread first accesses the media data to be processed to determine the original format of the media data to be processed, and then converts the media data to be processed from the original format to the target format by loading the logical conversion function. For example, for image data, the child thread accesses the media data to be processed and reads it pixel by pixel, and converts the media data to be processed from the original format to the target format through the logical conversion function, thereby outputting pixels in the target format.

[0099] For example, the conversion process can be accelerated by NEON instructions, and a subthread is started when the format conversion is triggered to solve the problem of blocking the main thread during format conversion. Among them, NEON instructions are a 128-bit SIMD (Single Instruction, Multiple Data) extended instruction set suitable for ARM Cortex-A series processors.

[0100] After the child thread completes the conversion operation, the child thread sends a message to the main thread to prompt the main thread that the format conversion has been completed, so that the main thread ends the format conversion process. While the child thread is performing the format conversion, the main thread can execute other logics at the same time to improve efficiency.

[0101] In the above embodiment, by starting a child thread, the child thread independently performs format conversion, and the child thread and the main thread do not interfere with each other, which effectively avoids the problem of media data blocking the main thread during the format conversion process, improves the speed of format conversion, and further improves processing efficiency.

[0102] In addition to the method of accelerating by creating a child thread to perform format conversion, in other embodiments, the data format of the media data to be processed is converted into a target format, including: calling a drawing function that matches the operating system to input the media data to be processed into a graphics processor; performing batch format conversion processing through the graphics processor to convert the data format of the media data to be processed into a target format; and returning the media data to be processed in the target format in the graphics processor to the central processing unit.

[0103] In other words, in this embodiment, the terminal performs format conversion through a graphics processing unit (GPU). Specifically, the terminal calls a graphics interface (API) that matches the current operating system to implement a call to a drawing function, inputs the media data to be processed into the graphics processor, and the graphics processor performs batch format conversion processing to convert the data format of the media data to be processed into a target format. After the conversion is completed, the graphics processor returns the converted media data to be processed in the target format to the central processing unit (CPU). The process of the graphics processor performing batch format conversion processing can be implemented by running a Shader program.

[0104] For example, the terminal calls the graphics interface in the GPU to call the drawing function, and inputs the media data to be processed into the GPU by calling the drawing function. The GPU uses the Shader program to perform batch format conversion on the media data to convert the data format of the media data to be processed into the target format. After the conversion is completed, the CPU reads the GPU texture to realize the return of the data converted by the GPU.

[0105] In the above embodiment, format conversion is performed through graphic processing, which can parallelize calculations in large quantities, and the calculation speed is much higher than the processing speed of the central processing unit. The calculation resources of the central processing unit are not occupied during the conversion process, and the efficiency of format conversion is greatly improved.

[0106] As mentioned above, the business processing results obtained through the target business processing can be used for further processing. In some application scenarios, the business processing results may include drive-related data, which refers to body drive data and expression drive data used to indicate the virtual character in the virtual interaction scene. For example, the drive-related data is a three-dimensional reconstruction result of human skeleton and facial expression. The three-dimensional reconstruction result can be combined with the image data in the collected real scene data to perform expression drive and body movement in the three-dimensional display scene. To this end, in this application scenario, the above method also includes: according to the drive-related data, controlling the virtual character in the virtual interaction scene to move according to the action of the object in the media data to be processed.

[0107] Specifically, the terminal loads the driving-related data into the virtual interactive application according to the driving-related data obtained by performing business processing on the media data to be processed, so as to control the virtual character in the virtual interactive scene displayed in the virtual interactive application according to the driving-related data, and control the virtual character to move according to the action of the object in the media data to be processed. For example, the terminal controls the driven limbs of the virtual character in the virtual interactive scene to move according to the action of the object in the collected real scene data according to the driving-related data, or controls the facial expression of the virtual character to change according to the facial expression of the object in the collected real scene data.

[0108] Therefore, by performing algorithm detection on the media data to be processed and further using the business processing results obtained by the algorithm detection to control the virtual character, the function of interaction between virtual interactive applications and real objects is provided without the need for additional plug-ins. At the same time, it can be connected to different platforms and has good adaptability.

[0109] by Figure 5AFor example, in some embodiments, the target service includes a video encoding service, which is used to encode media data including at least virtual scene data. In the video encoding service, virtual scene data such as pictures and sounds in a virtual interactive application (such as a game) can be loaded into an audio and video suite, and the audio and video suite encodes the virtual scene data to obtain a target encoding file. Alternatively, the virtual scene data can be loaded into the audio and video suite together with the real scene data such as the native camera and microphone captured by the terminal by calling the media data acquisition device and encoded to obtain a target encoding file. Alternatively, the virtual scene data, the real scene data and the media data file can be encoded together to obtain a corresponding target encoding file.

[0110] In other words, the to-be-processed media data corresponding to the target service at least includes virtual scene data, and on this basis may also include one or more of real scene data and media data files.

[0111] Accordingly, corresponding business processing is performed on the media data to be processed to obtain a business processing result corresponding to the target business, including: obtaining a preset encoding state; starting a sub-thread, and encoding the media data to be processed through the sub-thread and according to the encoding state to obtain a target encoding file adapted to the operating system.

[0112] The encoding state is parameter information related to encoding, including but not limited to encoding time, encoding format, bit rate, resolution, etc. The audio and video suite provided in the embodiment of the present application may provide an interface for configuring parameters, and the terminal may flexibly set the encoding state by calling the interface.

[0113] Specifically, the terminal can provide the operator with a visual editing interface, which allows the operator to set the encoding state. Thus, the terminal can obtain the set encoding state according to the input of the editing interface. Then, the terminal performs corresponding encoding processing on the media data to be processed according to the encoding state, thereby obtaining a target encoding file adapted to the local operating system.

[0114] When encoding the media data to be processed, the terminal may encode the media data to be processed into target encoded files in various formats supported by the local operating system based on the file format adapted by the local operating system (such as AVI or mp4).

[0115] In some embodiments, before performing encoding processing, the terminal may start a sub-thread and execute the encoding steps through the sub-thread, which does not affect the progress of the main thread and can improve processing efficiency.

[0116] In the above embodiment, by presetting the encoding state and performing corresponding encoding processing on the media data to be processed in the child thread, fast encoding of various types of data can be achieved in the Unreal Engine.

[0117] For example, Figure 5B As shown, taking the case where the media data to be processed includes real scene data and virtual scene data, the terminal separates the audio and video of the real scene data and the virtual scene data respectively, and obtains the native audio data and native image data of the real scene data, and the game screen data and game audio data of the virtual scene data.

[0118] In this video encoding business, the audio and video suite accelerates the process of encoding data collection. Generally speaking, when collecting screen data when the game is running, the screen data obtained by rendering in the GPU needs to be transferred back to the CPU, and the CPU performs the encoding data collection operation. However, the performance of the GPU to CPU transmission process is severely limited by the graphics card bandwidth. If the system interface is called to read the image data in the GPU, such as calling the ReadPixel (read function) interface on OpenGL (open graphics library) to read the image data in the GPU. However, this method will block the main thread, and the main thread will not be blocked until the complete rendering data is read into the CPU, which is extremely time-consuming.

[0119] To this end, in order to improve processing efficiency, in some embodiments, a virtual interactive scene presented in a virtual interactive application running on the Unreal Engine is recorded to obtain virtual scene data, including: rendering the virtual interactive scene in the virtual interactive application running on the Unreal Engine through a graphics processor to obtain image data; loading the image data into a preset cache area through the graphics processor; asynchronously reading the image data in the preset cache area through the central processing unit, and performing a capture operation based on the image data read by the central processing unit to obtain virtual scene data.

[0120] Specifically, during the operation of the virtual interactive application, the terminal renders the virtual interactive scene in the virtual interactive application based on the Unreal Engine through the graphics processor to obtain image data. For the rendered image data, the terminal loads the image data into a preset buffer area through the graphics processor. The terminal asynchronously reads the image data in the preset buffer area through the central processing unit, thereby completing the acquisition operation and obtaining the virtual scene data.

[0121] Exemplarily, the preset buffer area is, for example, a high-speed storage area set based on PBO (Pixel Buffer Object) technology. Figure 5CAs shown, the glReadPixels (reading function) function can be used to read pixel data from the OpenGL frame buffer and write the data to the PBO. This process can be regarded as a packing process. Then, the glDrawPixels (drawing function) function can be used to read pixel data from the PBO and copy it to the OpenGL frame buffer. This process can be regarded as a decompression process. The same is true for texture objects. Therefore, by using PBO to implement asynchronous DMA (direct memory access) transmission, fast upload and download of pixel data can be achieved, which is more efficient and does not affect the CPU cycle. Figure 5D As shown, the GPU directly loads image texture and other data into PBO through DMA transmission. The CPU can asynchronously read the data in PBO and perform data acquisition to obtain image data. Since the previous step is executed by the GPU, it will not occupy the CPU running cycle.

[0122] In the above embodiment, the rendered image data can be quickly transmitted through asynchronous transmission, which saves the time of image acquisition and improves the processing efficiency.

[0123] by Figure 6 For example, in some embodiments, the target service also includes a playback service. The playback service is used to combine with the UI (User Interface) control in the Unreal Engine, obtain the target video data in the data format supported by the Unreal Engine after preprocessing the to-be-processed media data loaded into the audio and video suite, and play the target video data in the running virtual interactive application.

[0124] In the playback service, the read media data file can be played in the virtual interactive application (for example, music, animation, etc.). Alternatively, the collected real scene data can also be played in the virtual interactive application.

[0125] In other words, the to-be-processed media data corresponding to the target service at least includes virtual scene data, and on this basis may also include one or more of real scene data and media data files.

[0126] Accordingly, corresponding business processing is performed on the media data to be processed to obtain a business processing result corresponding to the target business, including: format conversion of at least one of the real scene data and the media data file to obtain target video data suitable for the Unreal Engine; in the process of playing the virtual interactive scene through the virtual interactive application, the target video data is superimposed for playback.

[0127] Specifically, the terminal can perform format conversion on at least one of the real scene data and the media data file to obtain target video data suitable for the Unreal Engine. That is, based on actual business needs, the terminal can only perform format conversion on the real scene data to obtain target video data suitable for the Unreal Engine, or only perform format conversion on the media data file to obtain target video data suitable for the Unreal Engine, or both perform format conversion on the real scene data and the media data file to obtain target video data suitable for the Unreal Engine.

[0128] Furthermore, in the process of playing the virtual interactive scene corresponding to the virtual scene data through the virtual interactive application, the terminal can overlay the target video data for playback. That is, the target video data is overlaid in the virtual interactive scene for playback. Taking the virtual interactive scene as an example of a game scene, the present application can achieve overlaying the target video data in the game scene for playback.

[0129] In some embodiments, the terminal can convert the format of media data from various sources into a unified data format supported by Unreal Engine, and then play it through interactive controls combined with the Media Framework in Unreal Engine. The audio and video kit mentioned in the embodiments of the present application opens up the native UI controls in Unreal Engine, so that the target video data can be directly displayed on the canvas of the UI controls.

[0130] In the above embodiment, by converting the format of at least one of the real scene data and the media data file, the target video data suitable for the Unreal Engine is obtained, and then the target video can be superimposed in the virtual interactive scene for playback, which broadens the video presentation method and content, greatly enhances the interactivity and interactivity, and can also superimpose the real scene on the virtual scene for fusion display, greatly improving the interactivity.

[0131] In order to further improve the processing speed, the embodiment of the present application also performs accelerated processing during the data preprocessing process. The steps of the accelerated processing can refer to the description of the above embodiment, and will not be repeated here.

[0132] The present application also provides an application scenario, which applies the above-mentioned media data processing method. Specifically, the application of the media data processing method in this application scenario is as follows: the terminal obtains media data from different sources through a plug-in, such as calling the microphone and camera of a computer or mobile phone through a plug-in to collect real scene data, or obtaining media data files through network downloads, or obtaining virtual scene data by recording games through screen recording software, etc. According to actual business needs, the terminal performs business processing on one or more media data through various business processing processes preset in the plug-in, thereby obtaining business processing results, which can be used for subsequent processing and as input, such as playing the obtained media data files in the game, or driving the characters in the game to perform corresponding actions according to the real scene data, and so on.

[0133] Of course, it is not limited to this. The media data processing method provided in this application can also be applied to other application scenarios, such as game live broadcast, VR (Virtual Reality) video or game, and virtual image drawing, etc.

[0134] In a specific embodiment, the terminal determines the current target business and obtains media data from different sources according to the business needs of the target business. When the target business needs to use real scene data, the terminal generates a data collection instruction according to the target business, and runs the audio and video suite according to the data collection instruction. The audio and video suite calls the media data collection device to collect data according to the target call logic matching the operating system for the Unreal Engine to run, and obtains real scene data. When the target business needs to use the media data file, the terminal generates a data reading instruction according to the target business, and reads the media data file according to the data reading instruction. When the target business needs to use the virtual scene data, the terminal generates a data recording instruction according to the target business, and runs the audio and video suite or other plug-ins according to the data recording instruction to record the virtual interactive scene presented in the virtual interactive application running based on the Unreal Engine to obtain virtual scene data. Of course, it is not limited to this. The target business can also use media data from two or more different sources. According to the target business, the terminal determines the to-be-processed media data corresponding to the target business, and performs corresponding business processing on the to-be-processed media data to obtain the business processing result corresponding to the target business. The service processing includes, for example, detection service, video encoding service, and playback service.

[0135] The following is a specific example combined with an actual application scenario. Taking Unreal Engine 4 (UE4) as an example, the overall application framework of the media data processing method provided in the embodiment of the present application can be as follows: Figure 7As shown, for media data from different sources (including real scene data, media data files, and virtual scene data), corresponding audio data and image / video / screen data are obtained respectively, and different target services are processed based on these data, such as the detection service, video encoding service, and playback service in the above embodiment. In addition, in the process of data loading, encoding and decoding and other related operations, deep optimization and acceleration are also carried out to improve processing efficiency and achieve more excellent performance.

[0136] Therefore, the embodiment of the present application provides a cross-platform media data processing method for a virtual engine, which solves the problem of high development cost, high maintenance cost, and high stability risk caused by the current assembly and use of plug-ins for each independent platform. At the same time, the audio and video suite can perform corresponding business processing based on actual business needs, has extremely high scalability, and is easy to carry out secondary development. In addition, the data loading and target business processing processes are deeply accelerated to improve processing efficiency.

[0137] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0138] Based on the same inventive concept, the embodiment of the present application also provides a media data processing device for implementing the media data processing method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more media data processing device embodiments provided below can refer to the limitations of the media data processing method above, and will not be repeated here.

[0139] In one embodiment, Figure 8 As shown, a media data processing device 800 is provided, including: a collection module 801, a reading module 802, a recording module 803, a determination module 804 and a processing module 805, wherein:

[0140] The acquisition module 801 is used to call the media data acquisition device to collect data and obtain real scene data according to the target call logic matching the operating system for the Unreal Engine to run when a data acquisition instruction occurs.

[0141] The reading module 802 is used to obtain the media data file when a data reading instruction occurs.

[0142] The recording module 803 is used to record the virtual interactive scene presented in the virtual interactive application running based on the Unreal Engine to obtain virtual scene data when a data recording instruction occurs.

[0143] The determination module 804 is used to determine the to-be-processed media data corresponding to the target service, where the to-be-processed media data includes at least one of the following: real scene data, media data files, and virtual scene data.

[0144] The processing module 805 is used to perform corresponding business processing on the media data to be processed, and obtain a business processing result corresponding to the target business.

[0145] In some embodiments, the reading module is also used to determine a general data acquisition interface, which encapsulates calling logic that is compatible with media data acquisition devices corresponding to multiple different operating systems; determine the target calling logic encapsulated in the general data acquisition interface and matching the system type of the operating system; based on the target calling logic, call the media data acquisition device through the general data acquisition interface to perform data acquisition to obtain real scene data.

[0146] In some embodiments, the reading module is also used to initiate a permission acquisition request to the operating system based on the target call logic; when the call authorization is obtained according to the permission acquisition request, call the media data acquisition device to perform data acquisition; and receive real scene data collected by the media data acquisition device.

[0147] In some embodiments, the recording module is also used to render a virtual interactive scene in a virtual interactive application running on the Unreal Engine through a graphics processor to obtain image data; load the image data into a preset cache area through the graphics processor; asynchronously read the image data in the preset cache area through the central processing unit, and perform acquisition operations based on the image data read by the central processing unit to obtain virtual scene data.

[0148] In some embodiments, the above-mentioned device also includes an acceleration module, which is used to determine an acceleration interface that matches the operating system; perform corresponding decoding processing on the media data to be processed through the acceleration interface that matches the operating system, and execute corresponding business processing steps on the media data to be processed based on the decoded media data to be processed.

[0149] In some embodiments, the target service includes a detection service, and the media data to be processed corresponding to the target service includes at least one of real scene data and media data files; the processing module is also used to determine a target format suitable for the target detection algorithm; convert the data format of the media data to be processed into the target format; based on the target detection algorithm, image detection is performed on the video data to be processed in the media data to be processed to obtain video detection results, and audio detection is performed on the audio data to be processed in the media data to be processed to obtain audio detection results; based on the video detection results and the audio detection results, the service processing results corresponding to the target service are determined.

[0150] In some embodiments, the processing module is also used to start a child thread, determine the original format of the media data to be processed through the child thread, and convert the media data to be processed from the original format to the target format through a logical conversion function; after the child thread completes the conversion operation, notify the main thread that the format conversion has been completed.

[0151] In some embodiments, the processing module is also used to call a drawing function that matches the operating system to input the media data to be processed into the graphics processor; perform batch format conversion processing through the graphics processor to convert the data format of the media data to be processed into a target format; and return the media data to be processed in the target format in the graphics processor to the central processing unit.

[0152] In some embodiments, the business processing result includes drive-related data, and the above-mentioned device also includes an interaction module, which is used to control the virtual character in the virtual interaction scene according to the drive-related data to move according to the action of the object in the media data to be processed.

[0153] In some embodiments, the target service includes a playback service, and the media data to be processed corresponding to the target service includes virtual scene data, and also includes at least one of real scene data and media data files; the processing module is also used to convert the format of at least one of the real scene data and the media data file to obtain target video data suitable for the Unreal Engine; in the process of playing the virtual interactive scene corresponding to the virtual scene data through the virtual interactive application, the target video data is superimposed for playback.

[0154] In some embodiments, the target service includes a video encoding service, and the media data to be processed corresponding to the target service includes virtual scene data, and at least one of real scene data or media data files; the processing module is also used to obtain a preset encoding state; start a sub-thread, and encode the media data to be processed through the sub-thread and according to the encoding state to obtain a target encoding file adapted to the operating system.

[0155] Each module in the above-mentioned media data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0156] In one embodiment, a computer device is provided, which may be a terminal or a server. Taking the computer device as a terminal as an example, its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a media data processing method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0157] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0158] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0159] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0160] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0161] It should be noted that the media data involved in this application (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0162] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0163] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A media data processing method, characterized in that: The method comprises: In the event of a data collection instruction, by calling the data collection universal interface of the audio and video suite, a target calling logic matching the currently running operating system is selected; the audio and video suite is a plug-in; The audio and video suite executes the target calling logic and initiates a permission acquisition request to the currently running operating system; In the case of obtaining the calling authorization according to the permission acquisition request, calling the media data acquisition device through the data acquisition universal interface to collect data and obtain the real scene data; In case of a data read instruction, obtaining a media data file; In the event of a data recording instruction, a virtual interactive scene presented in a virtual interactive application running on the Unreal Engine is recorded to obtain virtual scene data; Determine the to-be-processed media data corresponding to the target service, wherein the to-be-processed media data includes at least one of the following: the real scene data, the media data file, and the virtual scene data; Corresponding business processing is performed on the media data to be processed to obtain a business processing result corresponding to the target business.

2. The method according to claim 1, characterized in that The method of calling the data acquisition general interface of the audio and video suite and selecting the target calling logic matching the currently running operating system includes: Determine a data acquisition universal interface, wherein the data acquisition universal interface encapsulates call logics adapted to media data acquisition devices corresponding to a plurality of different operating systems; Determine the target calling logic encapsulated in the data acquisition universal interface and matching the system type of the operating system.

3. The method according to claim 1, characterized in that The step of recording a virtual interactive scene presented in a virtual interactive application running on the Unreal Engine to obtain virtual scene data includes: Rendering a virtual interactive scene in a virtual interactive application running on the Unreal Engine through a graphics processor to obtain image data; Loading the image data into a preset buffer area through the graphics processor; The image data in the preset buffer area is asynchronously read by the central processing unit, and a collection operation is performed based on the image data read by the central processing unit to obtain virtual scene data.

4. The method according to claim 1, characterized in that: Before performing corresponding service processing on the to-be-processed media data to obtain a service processing result corresponding to the target service, the method further includes: Determining an acceleration interface that matches the operating system; The media data to be processed is decoded accordingly through an acceleration interface matching the operating system, and based on the decoded media data to be processed, a step of performing corresponding business processing on the media data to be processed is executed.

5. The method according to any one of claims 1 to 4, characterized in that The target service includes a detection service, and the to-be-processed media data corresponding to the target service includes at least one of the real scene data and the media data file; The performing corresponding business processing on the to-be-processed media data to obtain a business processing result corresponding to the target business includes: Determine the target format suitable for the target detection algorithm; Converting the data format of the to-be-processed media data into the target format; Based on the target detection algorithm, image detection is performed on the video data to be processed in the media data to be processed to obtain a video detection result, and audio detection is performed on the audio data to be processed in the media data to be processed to obtain an audio detection result; Based on the video detection result and the audio detection result, a service processing result corresponding to the target service is determined.

6. The method according to claim 5, characterized in that The converting the data format of the to-be-processed media data into the target format includes: Starting a sub-thread, determining the original format of the to-be-processed media data through the sub-thread, and converting the to-be-processed media data from the original format to a target format through a logical conversion function; After the sub-thread completes the conversion operation, the main thread is notified that the format conversion has been completed.

7. The method according to claim 5, characterized in that The converting the data format of the to-be-processed media data into the target format includes: Calling a drawing function that matches the operating system to input the media data to be processed into a graphics processor; Performing batch format conversion processing by the graphics processor to convert the data format of the to-be-processed media data into the target format; The media data to be processed in the target format in the graphics processor is transmitted back to the central processing unit.

8. The method according to claim 5, characterized in that The business processing result includes driving related data, and the method further includes: According to the drive-related data, the virtual character in the virtual interactive scene is controlled to move according to the action of the object in the media data to be processed.

9. The method according to any one of claims 1 to 4, characterized in that The target service includes a playback service, and the to-be-processed media data corresponding to the target service includes the virtual scene data and at least one of the real scene data and the media data file; The performing corresponding business processing on the to-be-processed media data to obtain a business processing result corresponding to the target business includes: Performing format conversion on at least one of the real scene data and the media data file to obtain target video data adapted to the Unreal Engine; In the process of playing the virtual interaction scene corresponding to the virtual scene data through the virtual interaction application, the target video data is superimposed for playback.

10. The method according to any one of claims 1 to 4, characterized in that The target service includes a video encoding service, and the to-be-processed media data corresponding to the target service includes the virtual scene data, or the to-be-processed media data also includes at least one of the real scene data and the media data file; The performing corresponding business processing on the to-be-processed media data to obtain a business processing result corresponding to the target business includes: Get the preset encoding status; A subthread is started, and encoding is performed on the to-be-processed media data through the subthread and according to the encoding state to obtain a target encoding file adapted to the operating system.

11. A media data processing device, characterized in that: The device comprises: The acquisition module is used to select a target call logic that matches the currently running operating system by calling the data acquisition universal interface of the audio and video suite when a data acquisition instruction occurs; the audio and video suite is a plug-in; the audio and video suite executes the target call logic and initiates a permission acquisition request to the currently running operating system; when the call authorization is obtained according to the permission acquisition request, the media data acquisition device is called through the data acquisition universal interface to perform data acquisition to obtain real scene data; A reading module, used for obtaining a media data file when a data reading instruction occurs; A recording module, used for recording a virtual interactive scene presented in a virtual interactive application running on the Unreal Engine to obtain virtual scene data when a data recording instruction is issued; A determination module, configured to determine the to-be-processed media data corresponding to the target service, wherein the to-be-processed media data includes at least one of the following: the real scene data, the media data file, and the virtual scene data; The processing module is used to perform corresponding business processing on the media data to be processed to obtain a business processing result corresponding to the target business.

12. The device according to claim 11, characterized in that The reading module is also used to determine a data acquisition universal interface, wherein the data acquisition universal interface encapsulates call logics adapted to media data acquisition devices corresponding to a plurality of different operating systems respectively; Determine the target calling logic encapsulated in the data acquisition universal interface and matching the system type of the operating system.

13. The device according to claim 11, characterized in that The recording module is also used to render the virtual interactive scene in the virtual interactive application running on the Unreal Engine through a graphics processor to obtain image data; and load the image data into a preset buffer area through the graphics processor; The image data in the preset buffer area is asynchronously read by the central processing unit, and a collection operation is performed based on the image data read by the central processing unit to obtain virtual scene data.

14. The device according to claim 11, characterized in that The media data processing device also includes an acceleration module, which is used to determine an acceleration interface that matches the operating system; perform corresponding decoding processing on the media data to be processed through the acceleration interface that matches the operating system, and execute the steps of performing corresponding business processing on the media data to be processed based on the decoded media data to be processed.

15. The device according to claim 11, characterized in that The target service includes a detection service, and the to-be-processed media data corresponding to the target service includes at least one of the real scene data and the media data file; The processing module is also used to determine a target format suitable for the target detection algorithm; Converting the data format of the media data to be processed into the target format; performing image detection on the video data to be processed in the media data to be processed based on the target detection algorithm to obtain a video detection result, and performing audio detection on the audio data to be processed in the media data to be processed to obtain an audio detection result; Based on the video detection result and the audio detection result, a service processing result corresponding to the target service is determined.

16. The device according to claim 15, characterized in that The processing module is further used to start a sub-thread, determine the original format of the media data to be processed through the sub-thread, and convert the media data to be processed from the original format to a target format through a logical conversion function; After the sub-thread completes the conversion operation, the main thread is notified that the format conversion has been completed.

17. The device according to claim 15, characterized in that The processing module is also used to call a drawing function that matches the operating system, input the media data to be processed into a graphics processor; perform batch format conversion processing through the graphics processor to convert the data format of the media data to be processed into the target format; and return the media data to be processed in the target format in the graphics processor to the central processing unit.

18. The device according to claim 15, characterized in that The business processing result includes drive-related data, and the media data processing device also includes an interaction module, which is used to control the virtual character in the virtual interaction scene according to the drive-related data to move according to the action of the object in the media data to be processed.

19. The device according to claim 11, characterized in that The target service includes a playback service, and the to-be-processed media data corresponding to the target service includes the virtual scene data and at least one of the real scene data and the media data file; The processing module is also used to convert the format of at least one of the real scene data and the media data file to obtain target video data suitable for the Unreal Engine; in the process of playing the virtual interactive scene corresponding to the virtual scene data through the virtual interactive application, the target video data is superimposed for playback.

20. The device according to claim 11, characterized in that The target service includes a video encoding service, and the to-be-processed media data corresponding to the target service includes the virtual scene data, or the to-be-processed media data also includes at least one of the real scene data and the media data file; The processing module is also used to obtain a preset encoding state; A subthread is started, and encoding is performed on the to-be-processed media data through the subthread and according to the encoding state to obtain a target encoding file adapted to the operating system.

21. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

22. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

23. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images