Resource processing method, device, electronic device and storage medium

Abstract

The present application discloses a resource processing method, device, server and storage medium, and relates to the field of data processing technology. The method includes: an electronic device encodes multi-channel audio data, obtains the encoded multi-channel audio data, sends the to-be-processed resources including the encoded multi-channel audio data to a head-mounted device, the head-mounted device receives the to-be-processed resources sent by the electronic device, the to-be-processed resources include the encoded multi-channel audio data, decodes the encoded multi-channel audio data, obtains the decoded multi-channel audio data and the channel position information of the multi-channel audio data, renders the decoded multi-channel audio data according to the channel position information and the posture information of the head-mounted device to obtain the audio data to be played, and plays the audio data to be played, thereby reducing processing delay, improving the situation of freezing, and improving user experience.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and more specifically, to a resource processing method, device, electronic device and storage medium. Background Art

[0002] With the development of science and technology, electronic devices are used more and more widely and have more and more functions. They have become one of the necessities in people's daily life. At present, in many cases, users expect to receive and play resources sent by electronic devices through head-mounted devices, but this method will increase the delay in the process of resource transmission, which may cause lag and reduce the user experience. Summary of the invention

[0003] In view of the above problems, the present application proposes a resource processing method, device, electronic device and storage medium to solve the above problems.

[0004] In a first aspect, an embodiment of the present application provides a resource processing method, which is applied to a head-mounted device, wherein the head-mounted device is used to connect to an electronic device, and the method comprises: receiving resources to be processed sent by the electronic device, wherein the resources to be processed include encoded multi-channel audio data; decoding the encoded multi-channel audio data to obtain decoded multi-channel audio data and channel position information of the multi-channel audio data; rendering the decoded multi-channel audio data according to the channel position information and the posture information of the head-mounted device to obtain audio data to be played; and playing the audio data to be played.

[0005] In a second aspect, an embodiment of the present application provides a resource processing method, which is applied to an electronic device, wherein the electronic device is used to connect to a head-mounted device, and the method includes: encoding multi-channel audio data to obtain encoded multi-channel audio data; sending resources to be processed including the encoded multi-channel audio data to instruct the head-mounted device to decode the encoded multi-channel audio data to obtain decoded channel audio data and channel position information of the multi-channel audio data, and rendering the decoded multi-channel audio data according to the posture information of the head-mounted device and the channel position information to obtain audio data to be played and play it.

[0006] In a third aspect, an embodiment of the present application provides a resource processing device, which is applied to a head-mounted device, and the head-mounted device is used to connect to an electronic device, and the device includes: a resource receiving module to be processed, which is used to receive resources to be processed sent by the electronic device, and the resources to be processed include encoded multi-channel audio data; an audio data decoding module, which is used to decode the encoded multi-channel audio data to obtain decoded multi-channel audio data and channel position information of the multi-channel audio data; an audio data rendering module, which is used to render the decoded multi-channel audio data according to the channel position information and the posture information of the head-mounted device to obtain audio data to be played; and an audio data playing module, which is used to play the audio data to be played.

[0007] In a fourth aspect, an embodiment of the present application provides a resource processing device, which is applied to an electronic device, wherein the electronic device is used to connect to a head-mounted device, and the device includes: an audio data obtaining module, which is used to encode multi-channel audio data and obtain encoded multi-channel audio data; an audio data sending module, which is used to send resources to be processed including the encoded multi-channel audio data to instruct the head-mounted device to decode the encoded multi-channel audio data, obtain decoded channel audio data and channel position information of the multi-channel audio data, and render the decoded multi-channel audio data according to the posture information of the head-mounted device and the channel position information to obtain audio data to be played and play it.

[0008] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor executes the above method.

[0009] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored, and the program code can be called by a processor to execute the above method.

[0010] The resource processing method, device, electronic device and storage medium provided in the embodiments of the present application, the electronic device encodes multi-channel audio data to obtain the encoded multi-channel audio data, sends the to-be-processed resources including the encoded multi-channel audio data to the head-mounted device, the head-mounted device receives the to-be-processed resources sent by the electronic device, the to-be-processed resources including the encoded multi-channel audio data, decodes the encoded multi-channel audio data to obtain the decoded multi-channel audio data and the channel position information of the multi-channel audio data, renders the decoded multi-channel audio data according to the channel position information and the posture information of the head-mounted device to obtain the audio data to be played, and plays the audio data to be played, so as to reduce the processing delay, improve the jamming situation and enhance the user experience by decoding the encoded multi-channel audio data on the head-mounted device, and then rendering and playing the decoded multi-channel audio data. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A schematic diagram showing an application environment of the resource processing method provided in an embodiment of the present application;

[0013] Figure 2 A timing diagram of a resource processing method provided in an embodiment of the present application is shown;

[0014] Figure 3 A schematic diagram showing a flow chart of a resource processing method provided by an embodiment of the present application is shown;

[0015] Figure 4 A schematic diagram showing a flow chart of a resource processing method provided by an embodiment of the present application is shown;

[0016] Figure 5 A schematic diagram showing a flow chart of a resource processing method provided by an embodiment of the present application is shown;

[0017] Figure 6 A schematic diagram showing a flow chart of a resource processing method provided by an embodiment of the present application is shown;

[0018] Figure 7 A schematic diagram showing a flow chart of a resource processing method provided by an embodiment of the present application is shown;

[0019] Figure 8 A module block diagram of a first resource processing device provided in an embodiment of the present application is shown;

[0020] Fig. 9 A module block diagram of a second resource processing device provided in an embodiment of the present application is shown;

[0021] Fig.10 A block diagram of an electronic device for executing a resource processing method according to an embodiment of the present application is shown;

[0022] Fig.11 A storage unit for storing or carrying program codes for implementing the resource processing method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0024] In many cases, users expect to receive and play resources sent by electronic devices through head-mounted devices. Specifically, the resources are rendered by the electronic devices and then transmitted to the head-mounted devices for playback. It is understandable that a lot of data needs to be transmitted in the process of rendering resources, and the amount of data contained in the rendered resources is large, which will increase the delay in resource rendering and transmission, which may cause freezes and reduce user experience.

[0025] In view of the above problems, the inventors have discovered after long-term research and proposed the resource processing method, device, server and storage medium provided in the embodiments of the present application, which can reduce processing delay, improve the situation of freezing, and improve user experience by decoding the encoded multi-channel audio data on the head-mounted device, and then rendering and playing the decoded multi-channel audio data. The specific resource processing method will be described in detail in the subsequent embodiments.

[0026] The following describes an application environment in which the resource processing method provided in the embodiments of the present application can be used.

[0027] See also Figure 1 , Figure 1 A schematic diagram of an application environment of the resource processing method provided in an embodiment of the present application is shown, and the application environment includes: an electronic device 100 and a head-mounted device 200. The electronic device 100 can be connected to the head-mounted device 200 via Bluetooth, WiFi, or ZigBee to achieve data interaction between the electronic device 100 and the head-mounted device 200.

[0028] In some embodiments, the electronic device 100 may include a smart phone, a tablet computer, a wearable electronic device, a smart control panel, a smart home control device, etc., which is not limited here.

[0029] In some embodiments, the head-mounted device 200 may include a virtual reality (VR) head-mounted device, an augmented reality (AR) head-mounted device, a mixed reality (MR), a mixed reality (MR), a head-mounted wireless headset, a bone conduction wireless headset, and a split wireless headset, etc., without limitation herein.

[0030] See also Figure 2 , Figure 2 The timing diagram of the resource processing method provided in the embodiment of the present application is shown. The method is used to decode the encoded multi-channel audio data on the head-mounted device, and then render and play the decoded multi-channel audio data to reduce processing delay, improve the situation of freezing, and improve user experience. In a specific embodiment, the resource processing method is applied to a resource processing system, which includes an electronic device and a head-mounted device, and the head-mounted device is used to connect to the electronic device. Figure 2 The process shown is described in detail, and the resource processing method may specifically include the following steps:

[0031] Step S110: the electronic device encodes the multi-channel audio data to obtain encoded multi-channel audio data.

[0032] In some implementations, the electronic device may be pre-set and store a preset private protocol, which is used to encode the multi-channel audio data, so the electronic device may encode the multi-channel audio data through the preset private protocol to obtain the encoded multi-channel audio data. The preset private protocol may be used to identify the channel position information of the multi-channel audio data, and the encoding rule of the channel position information of the multi-channel audio data may be the encoding rule shown in Table 1.

[0033] Table 1

[0034]

[0035]

[0036] Step S120: the electronic device sends resources to be processed including the encoded multi-channel audio data to the head mounted device.

[0037] In this embodiment, the electronic device can be connected to the head-mounted device, wherein the electronic device can be connected to the head-mounted device in a wireless manner, for example, the electronic device can be connected to the head-mounted device via WiFi.

[0038] In some embodiments, the electronic device may send the to-be-processed resources including the encoded multi-channel audio data to the head-mounted device, and correspondingly, the head-mounted device may receive the to-be-processed resources including the encoded multi-channel audio data sent by the electronic device.

[0039] As a method, the resource to be processed may include only the encoded multi-channel audio data, and the resource to be processed may also include the encoded multi-channel audio data and image data at the same time. In one embodiment of the present application, the electronic device may determine the resource to be processed to be sent according to the type of the head-mounted device. For example, when the type of the head-mounted device is an audio playback type, the resource to be processed may include only the encoded multi-channel audio data. For another example, when the type of the head-mounted device is a video playback type, the resource to be processed may include both the encoded multi-channel audio data and image data.

[0040] Step S130: the head mounted device receives resources to be processed sent by the electronic device, where the resources to be processed include encoded multi-channel audio data.

[0041] In this embodiment, the head-mounted device can receive resources to be processed including encoded multi-channel audio data sent by the electronic device, and the head-mounted device can receive resources to be processed including encoded multi-channel audio data and image data sent by the electronic device.

[0042] Step S140: the head mounted device decodes the encoded multi-channel audio data to obtain the decoded multi-channel audio data and channel position information of the multi-channel audio data.

[0043] In some embodiments, the head mounted device decodes the encoded multi-channel audio data through a preset private protocol to obtain the decoded multi-channel audio data and the channel position information of the multi-channel audio data. For example, the encoded multi-channel audio data is 25103, and the channel position information of the decoded multi-channel audio data corresponds to the encoding rule in Table 1, and the channel position information of the decoded multi-channel audio data is the left front channel, the right channel, the center channel, the low frequency channel, and the right front channel.

[0044] Step S150: the head mounted device renders the decoded multi-channel audio data according to the channel position information and the posture information of the head mounted device to obtain audio data to be played.

[0045] In this embodiment, the head-mounted device renders the decoded multi-channel audio data according to the channel position information and the posture information of the head-mounted device to obtain the audio data to be played, wherein the posture information represents a view matrix, that is, when a user views an object in the real world through the head-mounted device (augmented reality head-mounted device), the corresponding views are different when viewed from different directions or angles.

[0046] In some embodiments, the degree of rendering of the decoded multi-channel audio data can be different depending on the type of head-mounted device. For example, when the head-mounted device is a VR head-mounted device, the degree of rendering is enhanced; when the head-mounted device is a split wireless headset, the degree of rendering is reduced.

[0047] Step S160: the head mounted device plays the audio data to be played.

[0048] In some real-time modes, the audio data played by the head mounted device may be audio and video data of the real world, or audio and video superimposed between virtual objects and the real world, which is not limited here.

[0049] In this embodiment, the head mounted device can play the audio data to be played that has been rendered, and the head mounted device can also play the audio and video data to be played that has been rendered.

[0050] The resource processing method provided in the embodiment of the present application is that an electronic device encodes multi-channel audio data to obtain encoded multi-channel audio data, sends resources to be processed including the encoded multi-channel audio data to a head-mounted device, the head-mounted device receives the resources to be processed sent by the electronic device, the resources to be processed include the encoded multi-channel audio data, decodes the encoded multi-channel audio data to obtain decoded multi-channel audio data and channel position information of the multi-channel audio data, renders the decoded multi-channel audio data according to the channel position information and the posture information of the head-mounted device to obtain audio data to be played, and plays the audio data to be played, thereby decoding the encoded multi-channel audio data on the head-mounted device, and then rendering and playing the decoded multi-channel audio data to reduce processing delays, improve freezes, and enhance user experience.

[0051] See also Figure 3 , Figure 3A flow chart of a resource processing method provided by an embodiment of the present application is shown. The method is used to decode the encoded multi-channel audio data on a head-mounted device, and then render and play the decoded multi-channel audio data, so as to reduce processing delays, improve the situation of freezing, and enhance user experience. In a specific embodiment, the encoded multi-channel audio data is obtained by the electronic device encoding the multi-channel audio data based on a preset private protocol, and the preset private protocol is used to identify the channel position information of the multi-channel audio data. In a specific embodiment, the resource processing method is applied to Figure 8 The resource processing device 500 and the server 100 ( Fig.10 ). The following will take the head-mounted device as an example to illustrate the specific process of this embodiment. In this embodiment, the head-mounted device is used to connect to the electronic device. Figure 3 The process shown is described in detail, and the resource processing method may specifically include the following steps:

[0052] Step S210: receiving resources to be processed sent by the electronic device, wherein the resources to be processed include encoded multi-channel audio data.

[0053] Step S220: Decode the encoded multi-channel audio data to obtain decoded multi-channel audio data and channel position information of the multi-channel audio data.

[0054] Step S230: Rendering the decoded multi-channel audio data according to the channel position information and the posture information of the head mounted device to obtain audio data to be played.

[0055] Step S240: Play the audio data to be played.

[0056] For the detailed description of step S210 to step S240 , please refer to step S130 to step S160 , which will not be repeated here.

[0057] In a resource processing method provided by an embodiment of the present application, a head-mounted device receives resources to be processed sent by an electronic device, the resources to be processed include encoded multi-channel audio data, the encoded multi-channel audio data is decoded to obtain the decoded multi-channel audio data and channel position information of the multi-channel audio data, the decoded multi-channel audio data is rendered according to the channel position information and the posture information of the head-mounted device to obtain the audio data to be played, and the audio data to be played is played, thereby decoding the encoded multi-channel audio data on the head-mounted device, and then rendering and playing the decoded multi-channel audio data, so as to reduce processing delays, improve freeze conditions, and enhance user experience.

[0058] See also Figure 4 , Figure 4 The schematic diagram of the process of resource processing method provided by an embodiment of the present application is shown. The method is applied to an electronic device. In this embodiment, the electronic device is used to connect to a head-mounted device. Figure 4 The process shown is described in detail, and the resource processing method may specifically include the following steps:

[0059] Step S310: Encode the multi-channel audio data to obtain encoded multi-channel audio data.

[0060] Step S320: Sending resources to be processed including the encoded multi-channel audio data to the head-mounted device to instruct the head-mounted device to decode the encoded multi-channel audio data, obtain the decoded channel audio data and the channel position information of the multi-channel audio data, and render the decoded multi-channel audio data according to the posture information of the head-mounted device and the channel position information to obtain the audio data to be played and play it.

[0061] For the detailed description of step S310 to step S320, please refer to step S110 to step S160, which will not be repeated here.

[0062] In a resource processing method provided by an embodiment of the present application, an electronic device encodes multi-channel audio data to obtain the encoded multi-channel audio data, and then sends the to-be-processed resources including the encoded multi-channel audio data to the head-mounted device, thereby rendering and playing the multi-channel audio data on the head-mounted device to reduce processing delays, improve freeze conditions, and enhance user experience.

[0063] See also Figure 5 , Figure 5 A schematic diagram of a resource processing method provided in another embodiment of the present application is shown. The method is applied to an electronic device, and the electronic device is used to connect to a head-mounted device. In this embodiment, the electronic device is connected to the head-mounted device through at least one communication method. Figure 5 The process shown is described in detail, and the resource processing method may specifically include the following steps:

[0064] Step S410: Encode the multi-channel audio data to obtain encoded multi-channel audio data.

[0065] For the detailed description of step S410, please refer to step S110, which will not be repeated here.

[0066] Step S420: Obtain the data volume corresponding to the resource to be processed.

[0067] In some embodiments, the electronic device can obtain the amount of data corresponding to the resources to be processed, wherein the electronic device can obtain the amount of data of the resources to be processed by burying points on the resources to be processed, and the electronic device can also obtain the amount of data of the resources to be processed by monitoring the resources to be processed. The specific method of obtaining the amount of data is not limited here.

[0068] Step S430: Determine a first communication mode from the at least one communication mode according to the data volume corresponding to the to-be-processed resources.

[0069] In some embodiments, the electronic device may pre-set and store a data volume threshold, determine the size of the data volume corresponding to the resource to be processed according to the data volume threshold, and determine the first communication mode from at least one communication mode according to the size relationship between the data volume corresponding to the resource to be processed and the data volume threshold. For example, when the data volume corresponding to the resource to be processed is greater than the data volume threshold, WiFi may be determined as the first communication mode; when the data volume corresponding to the resource to be processed is less than the data volume threshold, Bluetooth may be determined as the first communication mode.

[0070] In some embodiments, the electronic device can determine the default first communication method for connecting to different types of head-mounted devices based on the communication method selected more frequently by the user when connecting the head-mounted devices of that type. For example, if the user selects Bluetooth as the communication method for connecting to a split wireless headset multiple times, then Bluetooth will be the default first communication method for the split wireless headset.

[0071] Step S440: Sending resources to be processed including the encoded multi-channel audio data to the head mounted device through the first communication mode.

[0072] In this embodiment, the electronic device can be connected to the head-mounted device via a first communication method, and send resources to be processed including encoded multi-channel audio data to the head-mounted device via the first communication method. For example, the first communication method is Bluetooth, and the electronic device sends resources to be processed including encoded multi-channel audio data to the head-mounted device via Bluetooth.

[0073] Step S450: Detecting the network congestion corresponding to the first communication mode.

[0074] In some embodiments, the electronic device detects the network congestion corresponding to the first communication method connected to the head-mounted device. The electronic device itself is provided with an application for detecting the network congestion level. The network congestion level can be detected by the application for detecting the network congestion level. The electronic device can also detect the network congestion corresponding to the first communication method through the running status of other applications.

[0075] Step S460: When it is detected that the network congestion corresponding to the first communication mode is greater than the congestion threshold, a second communication mode is determined from the at least one communication mode, wherein the network congestion corresponding to the second communication mode is less than the congestion threshold.

[0076] In this embodiment, the electronic device may pre-set and store a congestion threshold, and when it is detected that the network congestion corresponding to the first communication mode is greater than the congestion threshold, the second communication mode is determined from at least one communication mode. For example, the first communication mode is Bluetooth, and the amount of data that can be transmitted by Bluetooth is relatively small. When a larger amount of data is transmitted, the network congestion increases. When it is detected that the network congestion corresponding to Bluetooth is greater than the congestion threshold, WiFi whose network congestion is less than the congestion threshold is determined from at least one communication mode as the second communication mode.

[0077] In some implementations, the electronic device may prompt the user that the playback fluency of the first communication method is poor, provide the playback fluency of multiple communication methods and prompt the user to select one of the communication methods, and determine the second communication method based on the user's selection.

[0078] Step S470: Sending resources to be processed including the encoded multi-channel audio data to the head mounted device through the second communication mode.

[0079] In this embodiment, the electronic device can be connected to the head-mounted device via a second communication method, and send resources to be processed including encoded multi-channel audio data to the head-mounted device via the second communication method. For example, the second communication method is WiFi, and the electronic device can send resources to be processed including encoded multi-channel audio data to the head-mounted device via WiFi.

[0080] Step S480: instruct the head-mounted device to decode the encoded multi-channel audio data, obtain the decoded channel audio data and the channel position information of the multi-channel audio data, and render the decoded multi-channel audio data according to the posture information of the head-mounted device and the channel position information to obtain the audio data to be played and play it.

[0081] For the detailed description of step S480, please refer to steps S150 to S160, which will not be repeated here.

[0082] Compared with the resource processing method provided in one embodiment of the present application, Figure 4The resource processing method shown in this embodiment can also determine the first communication mode between the head-mounted device and the electronic device according to the data volume, send the resources to be processed to the head-mounted device through the first communication mode, detect the network congestion of the first communication mode, and when the network congestion corresponding to the first communication mode is greater than the congestion threshold, switch to the second communication mode with smaller network congestion, and send the resources to be processed to the head-mounted device through the first communication mode, so that by determining the communication mode according to the network congestion and switching to the communication mode with smaller network congestion, the speed of the head-mounted device receiving the resources to be processed is increased, the playback is smoother, and the user experience is improved.

[0083] See also Figure 6 , Figure 6 A schematic diagram of a resource processing method provided by another embodiment of the present application is shown. The method is applied to an electronic device, and the electronic device is used to connect to a head-mounted device. Figure 6 The process shown is described in detail, and the resource processing method may specifically include the following steps:

[0084] Step S510: receiving posture information sent by the head mounted device.

[0085] In this embodiment, the electronic device receives the posture information sent by the head mounted device.

[0086] Step S520: Based on the channel position information of the multi-channel audio data and the posture information, the multi-channel audio data is rendered to obtain a to-be-played resource including the rendered multi-channel audio data.

[0087] For the detailed description of step S520, please refer to step S150, which will not be repeated here.

[0088] Step S530: sending the resource to be played to the head mounted device, and obtaining parameter information corresponding to the resource to be played.

[0089] In some implementations, the resource to be played is sent to the head mounted device via a determined communication method, and parameter information corresponding to the resource to be played is obtained.

[0090] In some implementations, the electronic device may obtain parameter information corresponding to the resource to be played by detecting a process of sending the resource to be played to the head mounted device.

[0091] In some implementations, the electronic device may also obtain parameter information corresponding to the resource to be played through an application program interface where the resource to be played is located.

[0092] Step S540: When the parameter information satisfies a preset parameter condition, the multi-channel audio data is encoded to obtain encoded multi-channel audio data.

[0093] In this embodiment, when the parameter information meets the preset parameter conditions, it means that the resources to be played are large and the delay in sending the resources to be played is large. In this way, the multi-channel audio data is encoded through a preset private protocol, and after the encoded multi-channel audio data is obtained, it is sent to the head-mounted device for decoding and rendering. The parameter information may include delay duration and packet loss rate, etc., which are not limited here.

[0094] In some embodiments, the parameter information includes a delay duration, and the preset parameter condition includes a preset delay duration. Therefore, after obtaining the delay duration corresponding to the resource to be played, the delay duration can be compared with the preset delay duration to determine whether the delay duration is greater than the preset delay duration. When the delay duration corresponding to the resource to be played is greater than the preset delay duration, the multi-channel audio data is encoded to obtain the encoded multi-channel audio data. When the delay duration corresponding to the resource to be played is not greater than the preset delay duration, the posture information sent by the head-mounted device is received, and the multi-channel audio data is rendered according to the channel position information of the multi-channel audio data and the received posture information, and the resource to be played of the rendered multi-channel audio data is obtained, and the resource to be played is sent to the head-mounted device.

[0095] In some embodiments, the parameter information includes a packet loss rate, and the preset parameter condition includes a preset packet loss rate. Therefore, after obtaining the packet loss rate corresponding to the resource to be played, the packet loss rate can be compared with the preset delayed packet loss rate to determine whether the packet loss rate is greater than the preset packet loss rate. When the packet loss rate corresponding to the resource to be played is greater than the preset packet loss rate, the multi-channel audio data is encoded to obtain the encoded multi-channel audio data. When the packet loss rate corresponding to the resource to be played is not greater than the preset packet loss rate, the posture information sent by the head-mounted device is received, and the multi-channel audio data is rendered according to the channel position information of the multi-channel audio data and the received posture information, and the resource to be played of the rendered multi-channel audio data is obtained, and the resource to be played is sent to the head-mounted device.

[0096] Step S550: Sending resources to be processed including the encoded multi-channel audio data to the head-mounted device to instruct the head-mounted device to decode the encoded multi-channel audio data, obtain the decoded channel audio data and the channel position information of the multi-channel audio data, and render the decoded multi-channel audio data according to the posture information of the head-mounted device and the channel position information to obtain the audio data to be played and play it.

[0097] For the detailed description of step S550, please refer to steps S120 to S160, which will not be repeated here.

[0098] Compared with the resource processing method provided in one embodiment of the present application, Figure 4 The resource processing method shown in the figure can also render the multi-channel audio data according to the channel position information and posture information of the received multi-channel audio data, and when the parameter information meets the preset parameter conditions, encode the multi-channel audio data to obtain the encoded multi-channel audio data, and send the unprocessed resources including the encoded multi-channel audio data to the head-mounted device, so that when the parameter information meets the preset parameter conditions, the head-mounted device can decode the encoded multi-channel audio data, and then render and play the decoded multi-channel audio data, so as to reduce the resources to be processed, increase the speed of sending the resources to be processed to the head-mounted device, improve the situation of freeze, and improve the user experience.

[0099] See also Figure 7 , Figure 7 A schematic diagram of a resource processing method provided by an embodiment of the present application is shown. The method is applied to an electronic device, and the electronic device is used to connect to a head-mounted device. Figure 7 The process shown is described in detail, and the resource processing method may specifically include the following steps:

[0100] Step S610: Acquire the audio type of the multi-channel audio data.

[0101] In this embodiment, the electronic device can obtain the audio type of the multi-channel audio data.

[0102] In some implementations, the audio data type may include audio data types in MP4, MAV, WMA, AC3, etc. formats, which are not limited here.

[0103] In some embodiments, the audio type may also include audio with high real-time requirements, such as call audio, video call audio, competitive game audio, etc. The audio type may also include audio with high real-time requirements, such as surround sound audio, etc. and stand-alone game audio.

[0104] Step S620: When the audio type meets the preset audio type, the multi-channel audio data is encoded to obtain the encoded multi-channel audio data.

[0105] In this embodiment, the electronic device can be pre-set and store a preset audio type. When the audio type meets the preset audio type, the multi-channel audio data is encoded through a preset private protocol to obtain the encoded multi-channel audio data; when the audio type does not meet the preset audio type, the posture information sent by the head-mounted device is received, the multi-channel audio data is rendered according to the channel position information of the multi-channel audio data and the received posture information, the rendered multi-channel audio data is obtained. The resources to be played are sent to the head-mounted device. For example, the preset audio type is audio with high real-time requirements. When the audio type is a fighting game audio, the preset audio type is met, and the multi-channel audio data is encoded through a preset private protocol to obtain the encoded multi-channel audio data.

[0106] Step S630: Sending resources to be processed including the encoded multi-channel audio data to the head-mounted device to instruct the head-mounted device to decode the encoded multi-channel audio data, obtain the decoded channel audio data and the channel position information of the multi-channel audio data, and render the decoded multi-channel audio data according to the posture information of the head-mounted device and the channel position information to obtain the audio data to be played and play it.

[0107] For the detailed description of step S630, please refer to steps S120 to S160, which will not be repeated here.

[0108] Compared with the resource processing method provided in one embodiment of the present application, Figure 4 The resource processing method shown in the embodiment can also obtain the audio type of multi-channel audio data, determine whether the audio type meets the preset audio type, and when the audio type meets the preset audio type, encode the multi-channel audio data to obtain the encoded multi-channel audio data, and send the to-be-processed resources including the encoded multi-channel audio data to the head-mounted device, so as to decode the encoded multi-channel audio data on the head-mounted device, and then render and play the decoded multi-channel audio data, so as to reduce the processing delay for the audio data of the preset audio type, improve the situation of freeze, and improve the user experience.

[0109] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0110] See also Figure 8 , Figure 8 The resource processing device 300 is applied to the head mounted device mentioned above, and the head mounted device is used to connect with the electronic device. Fig.10 The resource processing device 300 includes: a to-be-processed resource receiving module 310, an audio data decoding module 320, an audio data rendering module 330 and an audio data playing module 340, wherein:

[0111] The to-be-processed resource receiving module 310 is configured to receive the to-be-processed resource sent by the electronic device, wherein the to-be-processed resource includes encoded multi-channel audio data.

[0112] The audio data decoding module 320 is used to decode the encoded multi-channel audio data to obtain decoded multi-channel audio data and channel position information of the multi-channel audio data.

[0113] Furthermore, the audio data decoding module 320 includes: a channel position information acquisition submodule.

[0114] The channel position information acquisition submodule is used to decode the encoded multi-channel audio data based on the preset private protocol to obtain the decoded multi-channel audio data and the channel position information of the multi-channel audio data.

[0115] The audio data rendering module 330 is used to render the decoded multi-channel audio data according to the channel position information and the posture information of the head mounted device to obtain the audio data to be played.

[0116] The audio data playing module 340 is used to play the audio data to be played.

[0117] See also Fig. 9 , Fig. 9 The resource processing device 400 is applied to the above electronic device, which is used to connect with the head-mounted device. Fig.10 The resource processing device 400 includes an audio data obtaining module 410 and an audio data sending module 420, wherein:

[0118] The audio data obtaining module 410 is used to encode the multi-channel audio data to obtain encoded multi-channel audio data.

[0119] Furthermore, the audio data acquisition module 410 includes: a posture information receiving submodule, a multi-channel audio data rendering submodule, a parameter information acquisition submodule and a preset parameter condition judgment submodule, wherein:

[0120] The posture information receiving submodule is used to receive the posture information sent by the head mounted device.

[0121] The multi-channel audio data rendering submodule is used to render the multi-channel audio data based on the channel position information of the multi-channel audio data and the posture information, and obtain a to-be-played resource including the rendered multi-channel audio data.

[0122] The parameter information acquisition submodule is used to send the resource to be played to the head mounted device and obtain parameter information corresponding to the resource to be played.

[0123] The preset parameter condition judgment submodule is used to encode the multi-channel audio data to obtain the encoded multi-channel audio data when the parameter information meets the preset parameter condition.

[0124] Furthermore, the preset parameter condition judgment submodule includes: a delay duration judgment unit and a packet loss rate judgment unit, wherein:

[0125] The delay duration determination unit is used to encode the multi-channel audio data to obtain the encoded multi-channel audio data when the delay duration is greater than a preset delay duration.

[0126] The packet loss rate judgment unit is used to encode the multi-channel audio data to obtain the encoded multi-channel audio data when the packet loss rate is greater than a preset packet loss rate.

[0127] Furthermore, the audio data acquisition module 410 includes: an audio type acquisition submodule and an audio type determination submodule, wherein:

[0128] The audio type acquisition submodule is used to acquire the audio type of the multi-channel audio data.

[0129] The audio type determination submodule is used to encode the multi-channel audio data to obtain the encoded multi-channel audio data when the audio type meets the preset audio type.

[0130] The audio data sending module 420 is used to send the resources to be processed including the encoded multi-channel audio data to the head-mounted device, so as to instruct the head-mounted device to decode the encoded multi-channel audio data, obtain the decoded channel audio data and the channel position information of the multi-channel audio data, and render the decoded multi-channel audio data according to the posture information of the head-mounted device and the channel position information to obtain the audio data to be played and play it.

[0131] Further, the electronic device is connected to the head mounted device via at least one communication mode, and the audio data sending module 420 includes: a data stream acquisition submodule, a first communication mode determination module, a first audio data sending submodule, a network congestion detection submodule, a second communication mode determination submodule, and a second audio data sending submodule, wherein:

[0132] The data flow acquisition submodule is used to obtain the data volume corresponding to the resource to be processed.

[0133] The first communication mode determining module is used to determine a first communication mode from the at least one communication mode according to the data volume corresponding to the resource to be processed.

[0134] The first audio data sending submodule is used to send the to-be-processed resources including the encoded multi-channel audio data to the head mounted device through the first communication mode.

[0135] The network congestion detection submodule is used to detect the network congestion corresponding to the first communication mode.

[0136] The second communication mode determination submodule is used to determine a second communication mode from the at least one communication mode when it is detected that the network congestion corresponding to the first communication mode is greater than the congestion threshold, wherein the network congestion corresponding to the second communication mode is less than the congestion threshold.

[0137] The second audio data sending submodule is used to send the to-be-processed resources including the encoded multi-channel audio data to the head mounted device through the second communication mode.

[0138] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0139] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0140] See also Fig.10 , which shows a structural block diagram of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may be an electronic device capable of running applications, such as a smart phone, a tablet computer, an e-book, etc. The electronic device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, a touch screen 130, and one or more applications, wherein one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0141] Among them, the processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 110 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0142] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the terminal 100 during use.

[0143] The touch screen 130 is used to display information input by the user, information provided to the user, and various graphical user interfaces of the electronic device 100. These graphical user interfaces can be composed of graphics, text, icons, numbers, videos, and any combination thereof. In one example, the touch screen 130 can be a liquid crystal display (LCD) or an organic light-emitting diode (OLED), which is not limited here.

[0144] See also Fig.11, which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 500 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0145] The computer-readable storage medium 500 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer-readable storage medium 500 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 500 has storage space for program code 510 that performs any method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code 510 can be compressed, for example, in an appropriate form.

[0146] In summary, the resource processing method, device, electronic device and storage medium provided in the embodiments of the present application, the electronic device encodes multi-channel audio data to obtain encoded multi-channel audio data, sends resources to be processed including the encoded multi-channel audio data to a head-mounted device, the head-mounted device receives the resources to be processed sent by the electronic device, the resources to be processed include the encoded multi-channel audio data, decodes the encoded multi-channel audio data to obtain decoded multi-channel audio data and channel position information of the multi-channel audio data, renders the decoded multi-channel audio data according to the channel position information and the posture information of the head-mounted device to obtain audio data to be played, and plays the audio data to be played, so as to reduce processing delays, improve the situation of freezes, and enhance user experience by decoding the encoded multi-channel audio data on the head-mounted device, and then rendering and playing the decoded multi-channel audio data.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A resource processing method, characterized in that: Applied to a head mounted device, the head mounted device is used to connect to an electronic device, the method comprising: Receive resources to be processed sent by the electronic device, where the resources to be processed include encoded multi-channel audio data, wherein the electronic device receives posture information sent by the head-mounted device, renders the multi-channel audio data based on channel position information and the posture information of the multi-channel audio data, obtains resources to be played including the rendered multi-channel audio data, sends the resources to be played to the head-mounted device, and obtains parameter information corresponding to the resources to be played, and when the parameter information meets a preset parameter condition, encodes the multi-channel audio data to obtain the encoded multi-channel audio data, wherein the parameter information includes a delay duration and / or a packet loss rate; Decoding the encoded multi-channel audio data to obtain decoded multi-channel audio data and channel position information of the multi-channel audio data; Rendering the decoded multi-channel audio data to obtain audio data to be played according to the channel position information and the posture information of the head mounted device; Play the audio data to be played.

2. The method according to claim 1, characterized in that The encoded multi-channel audio data is obtained by encoding the multi-channel audio data by the electronic device based on a preset private protocol, wherein the preset private protocol is used to identify channel position information of the multi-channel audio data, and the decoding of the encoded multi-channel audio data to obtain the decoded multi-channel audio data and the channel position information of the multi-channel audio data includes: The encoded multi-channel audio data is decoded based on the preset private protocol to obtain the decoded multi-channel audio data and channel position information of the multi-channel audio data.

3. A resource processing method, characterized in that: Applied to an electronic device, the electronic device is used to connect with a head mounted device, the method comprising: Encoding the multi-channel audio data to obtain encoded multi-channel audio data; Sending to the head mounted device a to-be-processed resource including the encoded multi-channel audio data, so as to instruct the head mounted device to decode the encoded multi-channel audio data, obtain the decoded channel audio data and the channel position information of the multi-channel audio data, and render the decoded multi-channel audio data according to the posture information of the head mounted device and the channel position information to obtain the audio data to be played and play the audio data; The step of encoding the multi-channel audio data to obtain the encoded multi-channel audio data includes: Receiving posture information sent by the head mounted device; Based on the channel position information and the posture information of the multi-channel audio data, the multi-channel audio data is rendered to obtain a to-be-played resource including the rendered multi-channel audio data; Sending the resource to be played to the head mounted device, and acquiring parameter information corresponding to the resource to be played, wherein the parameter information includes delay duration and / or packet loss rate; When the parameter information meets the preset parameter conditions, the multi-channel audio data is encoded to obtain encoded multi-channel audio data.

4. The method according to claim 3, characterized in that The electronic device is connected to the head mounted device through at least one communication mode, and the electronic device sends the to-be-processed resource including the encoded multi-channel audio data to the head mounted device, including: Obtaining the amount of data corresponding to the resource to be processed; Determining a first communication mode from the at least one communication mode according to the amount of data corresponding to the resource to be processed; The resource to be processed including the encoded multi-channel audio data is sent to the head mounted device through the first communication mode.

5. The method according to claim 4, characterized in that After sending the to-be-processed resource including the encoded multi-channel audio data to the head mounted device through the first communication mode, the method further includes: detecting a network congestion degree corresponding to the first communication mode; When it is detected that the network congestion corresponding to the first communication mode is greater than a congestion threshold, determining a second communication mode from the at least one communication mode, wherein the network congestion corresponding to the second communication mode is less than the congestion threshold; The resource to be processed including the encoded multi-channel audio data is sent to the head mounted device through the second communication mode.

6. The method according to claim 5, characterized in that The parameter information includes a delay time, and when the parameter information satisfies a preset parameter condition, encoding the multi-channel audio data to obtain the encoded multi-channel audio data includes: When the delay time is greater than the preset delay time, the multi-channel audio data is encoded to obtain encoded multi-channel audio data.

7. The method according to claim 5, characterized in that The parameter information includes a packet loss rate, and when the parameter information satisfies a preset parameter condition, encoding the multi-channel audio data to obtain the encoded multi-channel audio data includes: When the packet loss rate is greater than a preset packet loss rate, the multi-channel audio data is encoded to obtain encoded multi-channel audio data.

8. The method according to any one of claims 3 to 5, characterized in that: The step of encoding the multi-channel audio data to obtain the encoded multi-channel audio data comprises: Obtaining the audio type of the multi-channel audio data; When the audio type satisfies a preset audio type, the multi-channel audio data is encoded to obtain the encoded multi-channel audio data.

9. A resource processing device, characterized in that: Applied to a head-mounted device, the head-mounted device is used to connect to an electronic device, and the device includes: A to-be-processed resource receiving module, used for receiving the to-be-processed resource sent by the electronic device, the to-be-processed resource comprising the encoded multi-channel audio data, wherein the electronic device receives the posture information sent by the head-mounted device, renders the multi-channel audio data based on the channel position information of the multi-channel audio data and the posture information, obtains the to-be-played resource comprising the rendered multi-channel audio data, sends the to-be-played resource to the head-mounted device, and obtains parameter information corresponding to the to-be-played resource, and when the parameter information meets the preset parameter condition, encodes the multi-channel audio data to obtain the encoded multi-channel audio data, wherein the parameter information comprises a delay duration and / or a packet loss rate; An audio data decoding module, used to decode the encoded multi-channel audio data to obtain decoded multi-channel audio data and channel position information of the multi-channel audio data; An audio data rendering module, configured to render the decoded multi-channel audio data according to the channel position information and the posture information of the head mounted device to obtain audio data to be played; The audio data playing module is used to play the audio data to be played.

10. A resource processing device, characterized in that: Applied to an electronic device, the electronic device is used to connect with a head-mounted device, and the device includes: An audio data acquisition module, used to encode the multi-channel audio data to obtain the encoded multi-channel audio data; an audio data sending module, configured to send resources to be processed including the encoded multi-channel audio data to the head mounted device, so as to instruct the head mounted device to decode the encoded multi-channel audio data, obtain the decoded channel audio data and the channel position information of the multi-channel audio data, and render the decoded multi-channel audio data according to the posture information of the head mounted device and the channel position information to obtain the audio data to be played and play the audio data; Wherein, the audio data acquisition module includes: A posture information receiving submodule, used to receive the posture information sent by the head mounted device; A multi-channel audio data rendering submodule, used to render the multi-channel audio data based on the channel position information of the multi-channel audio data and the posture information, and obtain a to-be-played resource including the rendered multi-channel audio data; A parameter information acquisition submodule, used for sending the resource to be played to the head mounted device, and acquiring parameter information corresponding to the resource to be played, wherein the parameter information includes a delay duration and / or a packet loss rate; The preset parameter condition judgment submodule is used to encode the multi-channel audio data to obtain the encoded multi-channel audio data when the parameter information meets the preset parameter condition.

11. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is coupled to the processor, and the memory stores instructions. When the instructions are executed by the processor, the processor performs the method according to any one of claims 1 to 2 or any one of claims 3 to 8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 2 or any one of claims 3 to 8.

Images