A rendering method and related device
By acquiring a single-object audio track from a multimedia file, determining the sound source location based on reference information, and performing spatial rendering, the problem of insufficient stereo sound effects for single-sound-object devices in existing technologies is solved, thereby enhancing the sense of stereo space and improving audio creation capabilities.
Patent Information
- Application Number
- CN202110477321.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-04-29
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-04-29
AI Technical Summary
Existing audio and video playback devices struggle to achieve realistic stereo sound effects when rendering single-sound objects in an audio track, especially when processing dual-channel/multi-channel audio and video data, where the stereo spatial sense of single-sound objects is lacking.
By acquiring single-object audio tracks from multimedia files, the sound source location of the sound-producing object is determined based on reference information, and spatial rendering is performed to enhance the sense of three-dimensional space. This includes considerations of sensors, user interaction, and playback device type, and techniques such as head-related transfer functions are used to process audio and video data.
It enhances the stereo spatial sense of single-sound objects in multimedia files, providing users with an immersive stereo sound experience, and improves audio editing and creation capabilities, making it suitable for music and film production.
Smart Images

Figure CN115278350B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of audio applications, and in particular to a rendering method and related equipment. Background Art
[0002] As audio and video playback technology continues to mature, people have higher and higher requirements on the playback effects of audio and video playback equipment.
[0003] At present, in order to enable users to experience realistic stereo surround sound effects when playing audio and video, audio and video playback devices can use processing technologies such as head related transfer function (HRTF) to process the audio and video data to be played.
[0004] However, a large amount of audio and video data on the Internet (such as music, film and television works, etc.) are all dual-channel / multi-channel audio tracks. How to perform spatial rendering on single-sounding objects in the audio track is an urgent problem to be solved. Summary of the invention
[0005] The embodiment of the present application provides a rendering method, which can enhance the stereoscopic spatial sense of a first single object audio track corresponding to a first sound-emitting object in a multimedia file, and provide users with an immersive stereo sound effect.
[0006] In a first aspect, an embodiment of the present application provides a rendering method, which can be applied to scenes such as music and film and television production. The method can be executed by a rendering device, or by a component of the rendering device (such as a processor, a chip, or a chip system, etc.). The method includes: obtaining a first single-object audio track based on a multimedia file, the first single-object audio track corresponding to a first sound object; determining a first sound source position of the first sound object based on reference information, the reference information including reference position information and / or media information of the multimedia file, the reference position information being used to indicate the first sound source position; performing spatial rendering on the first single-object audio track based on the first sound source position to obtain a rendered first single-object audio track.
[0007] In the embodiment of the present application, a first single object audio track is obtained based on a multimedia file, and the first single object audio track corresponds to a first sound object; a first sound source position of the first sound object is determined based on reference information, and the first single object audio track is spatially rendered based on the first sound source position to obtain a rendered first single object audio track. The stereoscopic spatial sense of the first single object audio track corresponding to the first sound object in the multimedia file can be enhanced, providing an immersive stereo sound effect for the user.
[0008] Optionally, in a possible implementation of the first aspect, the media information in the above steps includes: text to be displayed in the multimedia file, images to be displayed in the multimedia file, music features of music to be played in the multimedia file, and at least one of the sound source types corresponding to the first sound object.
[0009] In this possible implementation, if the media information includes music features, the rendering device can set the orientation and dynamics of the extracted specific sound object according to the music features of the music, so that the sound track corresponding to the sound object is more natural in the 3D rendering, and the artistry is better reflected. If the media information includes text, images, etc., the 3D immersive feeling is rendered in headphones or external speakers, so that the sound follows the picture, allowing users to obtain the best sound experience. In addition, if the media information includes video, tracking the sound object in the video and rendering the sound track corresponding to the sound object in the entire video can also be used in professional mixing post-production to improve the work efficiency of the mixer.
[0010] Optionally, in a possible implementation manner of the first aspect, the reference position information in the above steps includes first position information of the sensor or second position information selected by a user.
[0011] In this possible implementation, if the reference position information includes the first position information of the sensor, the user can perform real-time or later dynamic rendering of the selected sound object through the orientation or position provided by the sensor. Such control can give the sound object a specific spatial orientation and movement, enabling the user to interact with the audio and provide the user with a new experience. If the reference position information includes the second position information selected by the user, the user can control the selected sound object by dragging the interface and perform real-time or later dynamic rendering, giving it a specific spatial orientation and movement, enabling the user to interact with the audio and provide the user with a new experience. In addition, the sound and image of the sound object can be edited without the user having any sensors.
[0012] Optionally, in a possible implementation of the first aspect, the above steps also include: determining a type of playback device, the playback device is used to play a target audio track, the target audio track is obtained based on the rendered first single object audio track; spatially rendering the first single object audio track based on the first sound source position, including: spatially rendering the first single object audio track based on the first sound source position and the type of playback device.
[0013] In this possible implementation, when performing spatial rendering on the audio track, the type of playback device is considered. Different playback device types may correspond to different spatial rendering formulas, so that the spatial effect of the first single-object audio track played by the playback device after rendering is more realistic and accurate.
[0014] Optionally, in a possible implementation of the first aspect, the reference information in the above steps includes media information, and when the media information includes an image and the image includes a first sound object, determining the first sound source position of the first sound object based on the reference information includes: determining third position information of the first sound object in the image, the third position information including the two-dimensional coordinates and depth of the first sound object in the image; and obtaining the first sound source position based on the third position information.
[0015] In this possible implementation, the multi-modality of audio, video and images is combined, the coordinates of the sound object and the single-object audio track are extracted, and the 3D immersion is rendered in the headphones or external speaker environment, so that the sound moves with the picture, so that users can get the best sound experience. In addition, the technology of tracking and rendering the object audio in the entire video after selecting the sound object can also be applied to professional mixing post-production to improve the work efficiency of the mixer. By separating the single-object audio track of the audio in the video, and by analyzing and tracking the sound object in the video image, the motion information of the sound object is obtained, and the selected sound object is dynamically rendered in real time or in the later stage. The video picture is matched with the direction of the audio source to improve the user experience.
[0016] Optionally, in a possible implementation of the first aspect, the reference information in the above steps includes media information. When the media information includes music features of music to be played in a multimedia file, determining the first sound source position of the first sound object based on the reference information includes: determining the first sound source position based on an association relationship and the music features, wherein the association relationship is used to represent the association between the music features and the first sound source position.
[0017] In this possible implementation, the position and dynamics of the extracted specific sound object are set according to the musical characteristics of the music, so that the 3D rendering is more natural and the artistry is better reflected.
[0018] Optionally, in a possible implementation of the first aspect, the media information in the above steps includes media information. When the media information includes text that needs to be displayed in a multimedia file and the text contains position text related to the position, determining the first sound source position of the first sound object based on the reference information includes: identifying the position text; determining the first sound source position based on the position text.
[0019] In this possible implementation, by identifying location text related to the location, a 3D immersive feeling is rendered in a headphone or external speaker environment, and a sense of space corresponding to the location text is achieved, allowing users to obtain the best sound experience.
[0020] Optionally, in a possible implementation of the first aspect, the reference information in the above steps includes reference position information. When the reference position information includes first position information, before determining the first sound source position of the first sound emitting object based on the reference information, the method also includes: obtaining the first position information, the first position information including a first attitude angle of the sensor and a distance between the sensor and the playback device; determining the first sound source position of the first sound emitting object based on the reference information, including: converting the first position information into the first sound source position.
[0021] In this possible implementation, the user can use the orientation (i.e., the first attitude angle) provided by the sensor to dynamically render the selected sound object in real time or later. At this time, the sensor is like a laser pen, and the direction of the laser is the location of the sound source. This control can give the sound object a specific spatial orientation and movement, enabling users to interact with audio and provide users with a new experience.
[0022] Optionally, in a possible implementation of the first aspect, the reference information in the above steps includes reference position information. When the reference position information includes first position information, before determining the first sound source position of the first sound emitting object based on the reference information, the method also includes: acquiring the first position information, the first position information including the second attitude angle of the sensor and the acceleration of the sensor; determining the first sound source position of the first sound emitting object based on the reference information, including: converting the first position information into the first sound source position.
[0023] In this possible implementation, the user can use the actual position information of the sensor as the sound source position to control the sound object and perform real-time or post-dynamic rendering. The movement trajectory of the sound object can be simply and completely controlled by the user, greatly increasing the flexibility of editing.
[0024] Optionally, in a possible implementation of the first aspect, the reference information in the above steps includes reference position information. When the reference position information includes second position information, before determining the first sound source position of the first sound emitting object based on the reference information, the method also includes: providing a spherical view for the user to select, the center of the spherical view being the user's position, and the radius of the spherical view being the distance between the user's position and the playback device; obtaining the second position information selected by the user in the spherical view; determining the first sound source position of the first sound emitting object based on the reference information, including: converting the second position information into the first sound source position.
[0025] In this possible implementation, the user can select the second position information through the ball view (such as clicking, dragging, sliding, etc.) to control the selected sound object and perform real-time or post-dynamic rendering, giving it a specific spatial orientation and movement, so that the user can interact with the audio and provide the user with a new experience. In addition, the sound and image of the sound object can be edited without the user having any sensors.
[0026] Optionally, in a possible implementation of the first aspect, the above step: obtaining a first single object audio track based on a multimedia file includes: separating the first single object audio track from an original audio track in the multimedia file, the original audio track being obtained by synthesizing at least the first single object audio track and a second single object audio track, and the second single object audio track corresponds to the second sound object.
[0027] In this possible implementation, when the original audio track is composed of at least the first single object audio track and the second single object audio track, by separating the first single object audio track, a spatial rendering can be created for a specific sound object in the audio track, thereby enhancing the user's audio editing ability, and can be applied to object production of music and film and television works, thereby increasing the user's controllability and playability of music.
[0028] Optionally, in a possible implementation manner of the first aspect, the step of separating the first single object audio track from the original audio track in the multimedia file includes separating the first single object audio track from the original audio track by using a trained separation network.
[0029] In this possible implementation, when the original audio track is composed of at least the first single object audio track and the second single object audio track, the first single object audio track is separated by the separation network, and the specific sound object in the original audio track can be spatially rendered, thereby enhancing the user's audio editing ability and being applicable to the object production of music and film and television works, thereby increasing the user's controllability and playability of music.
[0030] Optionally, in a possible implementation of the first aspect, the trained separation network in the above steps is obtained by training the separation network with training data as the input of the separation network and with the value of the loss function being less than a first threshold as the goal, the training data includes a training audio track, the training audio track is synthesized from at least an initial third single-object audio track and an initial fourth single-object audio track, the initial third single-object audio track corresponds to the third sound object, the initial fourth single-object audio track corresponds to the fourth sound object, the third sound object and the first sound object are of the same type, and the second sound object and the fourth sound object are of the same type, the output of the separation network includes the separated third single-object audio track; the loss function is used to indicate the difference between the separated third single-object audio track and the initial third single-object audio track.
[0031] In this possible implementation, the separation network is trained with the goal of reducing the value of the loss function, that is, the difference between the third single object audio track output by the separation network and the initial third single object audio track is continuously reduced, so that the single object audio track separated by the separation network is more accurate.
[0032] Optionally, in a possible implementation manner of the first aspect, the above step of spatially rendering the first single object audio track based on the first sound source position and the type of the playback device includes: if the playback device is a headphone, obtaining the rendered first single object audio track by the following formula;
[0033]
[0034] in, is the first single-object audio track after rendering, S is the sound object of the multimedia file and includes the first sound object, i indicates the left channel or the right channel, a s (t) is the adjustment coefficient of the first sound object at time t, h i,s (t) is the head-related transfer function HRTF filter coefficient of the left channel or right channel corresponding to the first sound source at time t. The HRTF filter coefficient is related to the position of the first sound source. s (t) is the first single object audio track at time t, and τ is the integral term.
[0035] In this possible implementation, when the playback device is headphones, a technical problem of how to obtain the rendered first single-object audio track is solved.
[0036] Optionally, in a possible implementation manner of the first aspect, the above step of performing spatial rendering on the first single object audio track based on the first sound source position and the type of the playback device includes: if the playback device is N external playback devices, obtaining the rendered first single object audio track by the following formula;
[0037] in,
[0038]
[0039] in,
[0040] in, is the first single-object audio track after rendering, i indicates the i-th channel in the multi-channel, S is the sound object of the multimedia file and includes the first sound object, a s (t) is the adjustment coefficient of the first sound object at time t, g s (t) represents the translation coefficient of the first sound object at time t, o s (t) is the first single object track at time t, λ iThe azimuth angle obtained by the calibrator to calibrate the i-th external device, Φ i The tilt angle obtained by the calibrator to calibrate the i-th external device, r i is the distance between the i-th external speaker and the calibrator, N is a positive integer, i is a positive integer and i≤N, and the position of the first sound source is within the tetrahedron formed by the N external speakers.
[0041] In this possible implementation, when the playback device is an external playback device, a technical problem of how to obtain the rendered first single-object audio track is solved.
[0042] Optionally, in a possible implementation manner of the first aspect, the above steps also include: obtaining a target audio track based on the rendered first single-object audio track, the original audio track in the multimedia file, and the type of the playback device; and sending the target audio track to the playback device, which is used to play the target audio track.
[0043] In this possible implementation, the target audio track can be obtained, which facilitates the storage of the rendered audio track for subsequent playback and reduces repeated rendering operations.
[0044] Optionally, in a possible implementation manner of the first aspect, the step of obtaining the target audio track based on the rendered first single object audio track, the original audio track in the multimedia file, and the type of the playback device includes: if the type of the playback device is headphones, obtaining the target audio track by the following formula:
[0045]
[0046] Among them, i indicates the left channel or the right channel, is the target track at time t, X i (t) is the original audio track at time t, is the first single object track that has not been rendered at time t, is the first single object track after rendering, a s (t) is the adjustment coefficient of the first sound object at time t, h i,s (t) is the head-related transfer function HRTF filter coefficient of the left or right channel corresponding to the first sound source at time t. The HRTF filter coefficient is related to the position of the first sound source. s (t) is the first single-object audio track at time t, τ is the integral term, S1 is the sound object that needs to be replaced in the original audio track. If the first sound object is to replace the sound object in the original audio track, S1 is an empty set; S2 is the sound object added to the target audio track compared to the original audio track. If the first sound object is a sound object in the copied original audio track, S2 is an empty set; S1 and / or S2 are sound objects in the multimedia file and include the first sound object.
[0047] In this possible implementation, when the playback device is headphones, the technical problem of how to obtain the target audio track is solved, which facilitates the storage of the rendered audio track for subsequent playback and reduces repeated rendering operations.
[0048] Optionally, in a possible implementation of the first aspect, the step of obtaining the target audio track based on the rendered first single object audio track, the original audio track in the multimedia file, and the type of the playback device includes: if the type of the playback device is N external playback devices, obtaining the target audio track by the following formula:
[0049] in,
[0050]
[0051]
[0052] Where i indicates the i-th channel in the multi-channel, is the target track at time t, X i (t) is the original audio track at time t, is the first single audio track that has not been rendered at time t, is the first single object track after rendering, a s (t) is the adjustment coefficient of the first sound object at time t, g s (t) represents the translation coefficient of the first sound object at time t, g i,s (t) represents g s The i-th row in (t), o s (t) is the first single-object audio track at time t, S1 is the sound object that needs to be replaced in the original audio track. If the first sound object replaces the sound object in the original audio track, S1 is an empty set; S2 is the sound object added to the target audio track compared to the original audio track. If the first sound object is a copy of the sound object in the original audio track, S2 is an empty set; S1 and / or S2 are the sound objects of the multimedia file and include the first sound object, λ i The azimuth angle obtained by the calibrator to calibrate the i-th external device, Φ i The tilt angle obtained by the calibrator to calibrate the i-th external device, r i is the distance between the i-th external speaker and the calibrator, N is a positive integer, i is a positive integer and i≤N, and the position of the first sound source is within the tetrahedron formed by the N external speakers.
[0053] In this possible implementation, when the playback device is an external playback device, the technical problem of how to obtain the target audio track is solved, so as to facilitate the storage of the rendered audio track for subsequent playback and reduce repeated rendering operations.
[0054] Optionally, in a possible implementation manner of the first aspect, the music features in the above steps include: at least one of music structure, music emotion and singing mode.
[0055] Optionally, in a possible implementation of the first aspect, the above steps also include: separating a second single object audio track from the multimedia file; determining a second sound source position of a second sound object based on reference information, and spatially rendering the second single object audio track based on the second sound source position to obtain a rendered second single object audio track.
[0056] In this possible implementation, at least two single-object audio tracks can be separated from a multimedia file and corresponding spatial rendering can be performed, thereby enhancing the user's ability to edit specific sound objects in the audio, and can be applied to object production of music and film and television works, thereby increasing the user's controllability and playability of music.
[0057] The second aspect of the embodiment of the present application provides a rendering method, which can be executed by a rendering device, can be applied to scenes such as music, film and television production, and can also be executed by a component of the rendering device (such as a processor, chip, or chip system, etc.). The method includes: obtaining a multimedia file; obtaining a first single-object audio track based on the multimedia file, the first single-object audio track corresponding to the first sound object; displaying a user interface, the user interface including a rendering mode option; responding to a user's first operation in the user interface, determining an automatic rendering mode or an interactive rendering mode from the rendering mode options; when the automatic rendering mode is determined, obtaining the rendered first single-object audio track based on a preset mode; or when the interactive rendering mode is determined, responding to the user's second operation to obtain reference position information; determining the first sound source position of the first sound object based on the reference position information; rendering the first single-object audio track based on the first sound source position to obtain the rendered first single-object audio track.
[0058] In the embodiment of the present application, the rendering device determines the automatic rendering mode or the interactive rendering mode from the rendering mode options according to the first operation of the user. On the one hand, the rendering device can automatically obtain the rendered first single object audio track based on the first operation of the user. On the other hand, through the interaction between the rendering device and the user, the spatial rendering of the audio track corresponding to the first sound object in the multimedia file can be realized, providing the user with an immersive stereo sound effect.
[0059] Optionally, in a possible implementation method of the second aspect, the preset method in the above steps includes: obtaining media information of a multimedia file; determining a first sound source position of a first sound object based on the media information; and rendering a first single object audio track based on the first sound source position to obtain a rendered first single object audio track.
[0060] In this possible implementation,
[0061] Optionally, in a possible implementation of the second aspect, the media information in the above steps includes: text to be displayed in the multimedia file, images to be displayed in the multimedia file, music features of music to be played in the multimedia file, and at least one of the sound source types corresponding to the first sound object.
[0062] In this possible implementation, the rendering device determines the multimedia file to be processed through interaction with the user, thereby increasing the user's controllability and playability of the music in the multimedia file.
[0063] Optionally, in a possible implementation manner of the second aspect, the reference position information in the above steps includes first position information of the sensor or second position information selected by a user.
[0064] In this possible implementation, when spatially rendering the audio track, the type of playback device is determined by the user's operation. Different playback device types can correspond to different spatial rendering formulas, so that the spatial effect of the rendered audio track played by the playback device later is more realistic and accurate.
[0065] Optionally, in a possible implementation of the second aspect, the above steps: when the media information includes an image and the image includes a first sound emitting object, determining the first sound source position of the first sound emitting object based on the media information, includes: presenting the image; determining third position information of the first sound emitting object in the image, the third position information including the two-dimensional coordinates and depth of the first sound emitting object in the image; and obtaining the first sound source position based on the third position information.
[0066] In this possible implementation, the rendering device can automatically present an image and determine the sound-emitting object in the image, obtain the third position information of the sound-emitting object, and then obtain the first sound source position. In this way, the rendering device can automatically identify multimedia files. When the multimedia file includes an image, and the image includes the first sound-emitting object, the rendering device can automatically obtain the rendered first single-object audio track. After automatically extracting the coordinates of the sound-emitting object and the single-object audio track, the 3D immersive feeling is rendered in the environment of headphones or external speakers, so that the sound truly moves with the picture, allowing users to obtain the best sound experience.
[0067] Optionally, in a possible implementation manner of the second aspect, the above step of: determining third position information of the first sound emitting object in the image includes: determining the third position information of the first sound emitting object in response to a third operation of the user on the image.
[0068] In this possible implementation, the user can select the first sound object from multiple sound objects in the presented image, that is, can select the first single object audio track corresponding to the rendered first sound object. The coordinates of the sound object and the single object audio track are extracted according to the user's operation, and the 3D immersive feeling is rendered in the headphones or external speaker environment, so that the sound moves with the picture, allowing the user to obtain the best sound experience.
[0069] Optionally, in a possible implementation of the second aspect, when the above-mentioned media information includes music features of music to be played in a multimedia file, determining the first sound source position of the first sound object based on the media information includes: identifying the music features; determining the first sound source position based on an association relationship and the music features, wherein the association relationship is used to represent the association between the music features and the first sound source position.
[0070] In this possible implementation, the position and dynamics of the extracted specific sound object are set according to the musical characteristics of the music, so that the 3D rendering is more natural and the artistry is better reflected.
[0071] Optionally, in a possible implementation of the second aspect, when the above-mentioned media information includes text that needs to be displayed in a multimedia file and the text contains position text related to the position, determining the first sound source position of the first sound object based on the media information includes: identifying the position text; determining the first sound source position based on the position text.
[0072] In this possible implementation, by identifying location text related to the location, a 3D immersive feeling is rendered in a headphone or external speaker environment, and a sense of space corresponding to the location text is achieved, allowing users to obtain the best sound experience.
[0073] Optionally, in a possible implementation of the second aspect, when the above-mentioned reference position information includes first position information, obtaining the reference position information in response to a second operation of the user includes: obtaining first position information in response to a second operation of the user on the sensor, the first position information including a first attitude angle of the sensor and a distance between the sensor and the playback device; determining a first sound source position of a first sound-emitting object based on the reference position information includes: converting the first position information into a first sound source position.
[0074] In this possible implementation, the user can use the orientation (i.e., the first attitude angle) provided by the sensor to dynamically render the selected sound object in real time or later. At this time, the sensor is like a laser pen, and the direction of the laser is the location of the sound source. This control can give the sound object a specific spatial orientation and movement, enabling users to interact with audio and provide users with a new experience.
[0075] Optionally, in a possible implementation of the second aspect, when the above-mentioned reference position information includes first position information, obtaining the reference position information in response to a second operation of the user includes: obtaining first position information in response to a second operation of the user on the sensor, the first position information including a second attitude angle of the sensor and an acceleration of the sensor; determining a first sound source position of a first sound-emitting object based on the reference position information includes: converting the first position information into a first sound source position.
[0076] In this possible implementation, the user can use the actual position information of the sensor as the sound source position to control the sound object and perform real-time or post-dynamic rendering. The movement trajectory of the sound object can be simply and completely controlled by the user, greatly increasing the flexibility of editing.
[0077] Optionally, in a possible implementation of the second aspect, when the above-mentioned reference position information includes second position information, obtaining the reference position information in response to the user's second operation includes: presenting a spherical view, the center of the spherical view is the user's position, and the radius of the spherical view is the distance between the user's position and the playback device; in response to the user's second operation, determining the second position information in the spherical view; determining the first sound source position of the first sound-emitting object based on the reference position information, including: converting the second position information into the first sound source position.
[0078] In this possible implementation, the user can select the second position information through the ball view (such as clicking, dragging, sliding, etc.) to control the selected sound object and perform real-time or post-dynamic rendering, giving it a specific spatial orientation and movement, so that the user can interact with the audio and provide the user with a new experience. In addition, the sound and image of the sound object can be edited without the user having any sensors.
[0079] Optionally, in a possible implementation manner of the second aspect, the above step of: obtaining the multimedia file includes: in response to a fourth operation of the user, determining the multimedia file from at least one stored multimedia file.
[0080] In this possible implementation method, based on the user's selection, a multimedia file can be determined from at least one stored multimedia file, and then the first single object audio track corresponding to the first sound object in the multimedia file selected by the user can be rendered and produced, thereby improving the user experience.
[0081] Optionally, in a possible implementation of the second aspect, the above-mentioned user interface also includes a playback device type option; the method also includes: in response to the user's fifth operation, determining the type of playback device from the playback device type options; rendering the first single object audio track based on the first sound source position to obtain the rendered first single object audio track, including: rendering the first single object audio track based on the first sound source position and type to obtain the rendered first single object audio track.
[0082] In this possible implementation, according to the type of playback device used by the user, a rendering method suitable for the playback device being used by the user is selected, thereby improving the rendering effect of the playback device and making the 3D rendering more natural.
[0083] Optionally, in a possible implementation of the second aspect, the above step: obtaining a first single object audio track based on a multimedia file includes: separating the first single object audio track from an original audio track in the multimedia file, the original audio track being obtained by synthesizing at least the first single object audio track and a second single object audio track, and the second single object audio track corresponds to the second sound object.
[0084] In this possible implementation, when the original audio track is composed of at least the first single object audio track and the second single object audio track, by separating the first single object audio track, a spatial rendering can be created for a specific sound object in the audio track, thereby enhancing the user's audio editing ability, and can be applied to object production of music and film and television works, thereby increasing the user's controllability and playability of music.
[0085] In this possible implementation, the first single-object audio track can be separated from the multimedia file to render the single-object audio track corresponding to a specific sound-emitting object in the multimedia file, thereby improving the user's audio creation and user experience.
[0086] In this possible implementation, the user can dynamically render the selected sound object in real time or later through the orientation or position provided by the sensor. This control can give the sound object a specific spatial orientation and movement, enabling the user to interact with the audio and provide a new experience for the user.
[0087] In this possible implementation, the user can use the orientation (i.e., the first attitude angle) provided by the sensor to dynamically render the selected sound object in real time or later. At this time, the sensor is like a laser pen, and the direction of the laser is the location of the sound source. This control can give the sound object a specific spatial orientation and movement, enabling users to interact with audio and provide users with a new experience.
[0088] In this possible implementation, the actual position information of the sensor is used as the sound source position to control the sound object and perform real-time or post-dynamic rendering. The movement trajectory of the sound object can be simply and completely controlled by the user, greatly increasing the flexibility of editing.
[0089] In this possible implementation, the user can control the selected sound object by dragging the interface and perform real-time or post-production dynamic rendering, giving it a specific spatial orientation and movement, so that the user can interact with the audio and provide the user with a new experience. In addition, the sound and image of the sound object can be edited without the user having any sensors.
[0090] In this possible implementation, the rendering device can set the orientation and dynamics of the extracted specific sound object according to the musical characteristics of the music, so that the sound track corresponding to the sound object is more natural in the 3D rendering and the artistry is better reflected.
[0091] In this possible implementation, 3D immersion is rendered in a headphone or external speaker environment, so that the sound truly moves with the picture, allowing users to obtain the best sound experience.
[0092] In this possible implementation, after the rendering device determines the sound-emitting object, it can automatically track the sound-emitting object in the video and render the audio track corresponding to the sound-emitting object in the entire video. It can also be used in professional mixing post-production to improve the work efficiency of the mixer.
[0093] In this possible implementation, the rendering device can determine the sound object in the image according to the fourth operation of the user, track the sound object in the image, and render the audio track corresponding to the sound object. Such control can give the sound object a specific spatial orientation and movement, enable the user to interact with the audio and provide the user with a new experience.
[0094] Optionally, in a possible implementation manner of the second aspect, the music features in the above steps include: at least one of music structure, music emotion and singing mode.
[0095] Optionally, in a possible implementation of the second aspect, the above steps also include: separating a second single-object audio track from the original audio track; determining a second sound source position of a second sound object based on reference information, and spatially rendering the second single-object audio track based on the second sound source position to obtain a rendered second single-object audio track.
[0096] In this possible implementation, at least two single-object audio tracks can be separated from the original audio track, and corresponding spatial rendering can be performed, thereby enhancing the user's ability to edit specific sound objects in the audio, and can be applied to object production of music and film and television works, thereby increasing the user's controllability and playability of music.
[0097] A third aspect of the present application provides a rendering device, which can be applied to scenes such as music and film and television production, and the rendering device includes:
[0098] An acquisition unit, configured to acquire a first single object audio track based on the multimedia file, wherein the first single object audio track corresponds to the first sound object;
[0099] a determining unit, configured to determine a first sound source position of a first sound emitting object based on reference information, wherein the reference information includes reference position information and / or media information of a multimedia file, and the reference position information is used to indicate a first sound source position;
[0100] The rendering unit is used to perform spatial rendering on the first single object audio track based on the first sound source position to obtain a rendered first single object audio track.
[0101] Optionally, in a possible implementation of the third aspect, the above-mentioned media information includes: text that needs to be displayed in the multimedia file, images that need to be displayed in the multimedia file, music features of music that need to be played in the multimedia file, and at least one of the sound source types corresponding to the first sound object.
[0102] Optionally, in a possible implementation manner of the third aspect, the reference position information includes first position information of the sensor or second position information selected by a user.
[0103] Optionally, in a possible implementation of the third aspect, the above-mentioned determination unit is also used to determine the type of playback device, the playback device is used to play the target audio track, and the target audio track is obtained based on the rendered first single object audio track; the rendering unit is specifically used to spatially render the first single object audio track based on the first sound source position and the type of playback device.
[0104] Optionally, in a possible implementation of the third aspect, the above-mentioned reference information includes media information. When the media information includes an image and the image includes a first sound object, a determination unit is specifically used to determine third position information of the first sound object in the image, and the third position information includes the two-dimensional coordinates and depth of the first sound object in the image; the determination unit is specifically used to obtain the first sound source position based on the third position information.
[0105] Optionally, in a possible implementation of the third aspect, the above-mentioned reference information includes media information. When the media information includes music features of music to be played in a multimedia file, the determination unit is specifically used to determine the position of the first sound source based on the association relationship and the music features, and the association relationship is used to represent the association between the music features and the position of the first sound source.
[0106] Optionally, in a possible implementation of the third aspect, the above-mentioned media information includes media information. When the media information includes text that needs to be displayed in a multimedia file and the text contains location text related to the location, a determination unit is specifically used to identify the location text; the determination unit is specifically used to determine the position of the first sound source based on the location text.
[0107] Optionally, in a possible implementation of the third aspect, the above-mentioned reference information includes reference position information. When the reference position information includes first position information, the acquisition unit is also used to obtain the first position information, and the first position information includes a first attitude angle of the sensor and a distance between the sensor and the playback device; the determination unit is specifically used to convert the first position information into a first sound source position.
[0108] Optionally, in a possible implementation of the third aspect, the above-mentioned reference information includes reference position information. When the reference position information includes first position information, the acquisition unit is also used to acquire the first position information, and the first position information includes a second attitude angle of the sensor and an acceleration of the sensor; the determination unit is specifically used to convert the first position information into a first sound source position.
[0109] Optionally, in a possible implementation of the third aspect, the above-mentioned reference information includes reference position information. When the reference position information includes second position information, the rendering device also includes: a providing unit, used to provide a spherical view for the user to select, the center of the spherical view is the user's position, and the radius of the spherical view is the distance between the user's position and the playback device; an acquiring unit, also used to acquire the second position information selected by the user in the spherical view; and a determining unit, specifically used to convert the second position information into the first sound source position.
[0110] Optionally, in a possible implementation of the third aspect, the above-mentioned acquisition unit is specifically used to separate a first single-object audio track from an original audio track in a multimedia file, the original audio track is obtained by synthesizing at least the first single-object audio track and the second single-object audio track, and the second single-object audio track corresponds to the second sound object.
[0111] Optionally, in a possible implementation manner of the third aspect, the acquisition unit is specifically configured to separate the first single-object audio track from the original audio track by using a trained separation network.
[0112] Optionally, in a possible implementation of the third aspect, the above-mentioned trained separation network is obtained by training the separation network with training data as the input of the separation network and with the value of the loss function being less than a first threshold as the goal, the training data includes a training audio track, the training audio track is synthesized from at least an initial third single-object audio track and an initial fourth single-object audio track, the initial third single-object audio track corresponds to the third sound object, the initial fourth single-object audio track corresponds to the fourth sound object, the third sound object and the first sound object are of the same type, and the second sound object and the fourth sound object are of the same type, the output of the separation network includes the separated third single-object audio track; the loss function is used to indicate the difference between the separated third single-object audio track and the initial third single-object audio track.
[0113] Optionally, in a possible implementation manner of the third aspect, if the playback device is a headset, the acquisition unit is specifically configured to acquire the rendered first single-object audio track by using the following formula;
[0114]
[0115] in, is the first single-object audio track after rendering, S is the sound object of the multimedia file and includes the first sound object, i indicates the left channel or the right channel, a s (t) is the adjustment coefficient of the first sound object at time t, h i,s (t) is the head-related transfer function HRTF filter coefficient of the left channel or right channel corresponding to the first sound source at time t. The HRTF filter coefficient is related to the position of the first sound source. s (t) is the first single object audio track at time t, and τ is the integral term.
[0116] Optionally, in a possible implementation manner of the third aspect, if the above-mentioned playback device is N external playback devices, the acquisition unit is specifically configured to acquire the rendered first single object audio track through the following formula;
[0117] in,
[0118]
[0119] in,
[0120] in, is the first single-object audio track after rendering, i indicates the i-th channel in the multi-channel, S is the sound object of the multimedia file and includes the first sound object, a s (t) is the adjustment coefficient of the first sound object at time t, g s (t) represents the translation coefficient of the first sound object at time t, os (t) is the first single object track at time t, λ i The azimuth angle obtained by the calibrator to calibrate the i-th external device, Φ i The tilt angle obtained by the calibrator to calibrate the i-th external device, r i is the distance between the i-th external speaker and the calibrator, N is a positive integer, i is a positive integer and i≤N, and the position of the first sound source is within the tetrahedron formed by the N external speakers.
[0121] Optionally, in a possible implementation of the third aspect, the above-mentioned acquisition unit is also used to acquire the target audio track based on the rendered first single-object audio track and the original audio track in the multimedia file; the rendering device also includes: a sending unit, used to send the target audio track to the playback device, and the playback device is used to play the target audio track.
[0122] Optionally, in a possible implementation manner of the third aspect, if the above-mentioned playback device is a headset, the acquisition unit is specifically configured to acquire the target audio track by using the following formula:
[0123]
[0124] Among them, i indicates the left channel or the right channel, is the target track at time t, X i (t) is the original audio track at time t, is the first single object track that has not been rendered at time t, is the first single object track after rendering, a s (t) is the adjustment coefficient of the first sound object at time t, h i,s (t) is the head-related transfer function HRTF filter coefficient of the left channel or right channel corresponding to the first sound source at time t. The HRTF filter coefficient is related to the position of the first sound source. s (t) is the first single-object audio track at time t, τ is the integral term, S1 is the sound object that needs to be replaced in the original audio track. If the first sound object replaces the sound object in the original audio track, S1 is an empty set; S2 is the sound object added to the target audio track compared to the original audio track. If the first sound object is a copy of the sound object in the original audio track, S2 is an empty set; S1 and / or S2 are sound objects in the multimedia file and include the first sound object.
[0125] Optionally, in a possible implementation manner of the third aspect, if the above-mentioned playback device is N external playback devices; the acquisition unit is specifically configured to acquire the target audio track by using the following formula:
[0126] in,
[0127]
[0128]
[0129] Where i indicates the i-th channel in the multi-channel, is the target track at time t, X i (t) is the original audio track at time t, is the first single audio track that has not been rendered at time t, is the first single object track after rendering, a s (t) is the adjustment coefficient of the first sound object at time t, g s (t) represents the translation coefficient of the first sound object at time t, g i,s (t) represents g s The i-th row in (t), o s (t) is the first single-object audio track at time t, S1 is the sound object that needs to be replaced in the original audio track. If the first sound object replaces the sound object in the original audio track, S1 is an empty set; S2 is the sound object added to the target audio track compared to the original audio track. If the first sound object is a copy of the sound object in the original audio track, S2 is an empty set; S1 and / or S2 are the sound objects of the multimedia file and include the first sound object, λ i The azimuth angle obtained by the calibrator to calibrate the i-th external device, Φ i The tilt angle obtained by the calibrator to calibrate the i-th external device, r i is the distance between the i-th external speaker and the calibrator, N is a positive integer, i is a positive integer and i≤N, and the position of the first sound source is within the tetrahedron formed by the N external speakers.
[0130] A fourth aspect of the present application provides a rendering device, which can be applied to scenes such as music and film and television production, and includes:
[0131] An acquisition unit, used for acquiring multimedia files;
[0132] The acquisition unit is further used to acquire a first single object audio track based on the multimedia file, where the first single object audio track corresponds to the first sound object;
[0133] A display unit, used for displaying a user interface, wherein the user interface includes a rendering mode option;
[0134] a determining unit, configured to respond to a first operation of the user on the user interface and determine an automatic rendering mode or an interactive rendering mode from rendering mode options;
[0135] The acquiring unit is further configured to acquire the rendered first single object audio track based on a preset method when the determining unit determines that the rendering method is automatic; or
[0136] The acquisition unit is also used to obtain reference position information in response to a second operation of the user when the determination unit determines the interactive rendering mode; determine the first sound source position of the first sound-emitting object based on the reference position information; and render the first single-object audio track based on the first sound source position to obtain the rendered first single-object audio track.
[0137] Optionally, in a possible implementation method of the fourth aspect, the above-mentioned preset method includes: an acquisition unit, which is also used to acquire media information of a multimedia file; a determination unit, which is also used to determine a first sound source position of a first sound object based on the media information; and an acquisition unit, which is also used to render a first single-object audio track based on the first sound source position to obtain a rendered first single-object audio track.
[0138] Optionally, in a possible implementation of the fourth aspect, the above-mentioned media information includes: text that needs to be displayed in the multimedia file, images that need to be displayed in the multimedia file, music features of music that need to be played in the multimedia file, and at least one of the sound source types corresponding to the first sound object.
[0139] Optionally, in a possible implementation manner of the fourth aspect, the reference position information includes first position information of the sensor or second position information selected by a user.
[0140] Optionally, in a possible implementation of the fourth aspect, when the above-mentioned media information includes an image and the image includes a first sound object, a determination unit is specifically used to present the image; the determination unit is specifically used to determine third position information of the first sound object in the image, the third position information includes the two-dimensional coordinates and depth of the first sound object in the image; the determination unit is specifically used to obtain the first sound source position based on the third position information.
[0141] Optionally, in a possible implementation manner of the fourth aspect, the above-mentioned determination unit is specifically used to determine third position information of the first sound-emitting object in response to a third operation of the user on the image.
[0142] Optionally, in a possible implementation manner of the fourth aspect, when the above-mentioned media information includes music features of music to be played in the multimedia file, the determination unit is specifically used to identify the music features;
[0143] The determination unit is specifically used to determine the first sound source position based on the association relationship and the music feature, and the association relationship is used to represent the association between the music feature and the first sound source position.
[0144] Optionally, in a possible implementation of the fourth aspect, when the above-mentioned media information includes text that needs to be displayed in a multimedia file and the text contains location text related to the location, the determination unit is specifically used to identify the location text; the determination unit is specifically used to determine the first sound source position based on the location text.
[0145] Optionally, in a possible implementation of the fourth aspect, when the above-mentioned reference position information includes first position information, the determination unit is specifically used to respond to the user's second operation on the sensor to obtain the first position information, the first position information including a first attitude angle of the sensor and a distance between the sensor and the playback device; the determination unit is specifically used to convert the first position information into a first sound source position.
[0146] Optionally, in a possible implementation of the fourth aspect, when the above-mentioned reference position information includes first position information, the determination unit is specifically used to respond to the user's second operation on the sensor to obtain the first position information, the first position information including the second attitude angle of the sensor and the acceleration of the sensor; the determination unit is specifically used to convert the first position information into a first sound source position.
[0147] Optionally, in a possible implementation of the fourth aspect, when the above-mentioned reference position information includes second position information, the determination unit is specifically used to present a spherical view, the center of the spherical view is the user's position, and the radius of the spherical view is the distance between the user's position and the playback device; the determination unit is specifically used to respond to the user's second operation and determine the second position information in the spherical view; the determination unit is specifically used to convert the second position information into the first sound source position.
[0148] Optionally, in a possible implementation manner of the fourth aspect, the acquisition unit is specifically configured to respond to a fourth operation of the user and determine a multimedia file from at least one stored multimedia file.
[0149] Optionally, in a possible implementation of the fourth aspect, the above-mentioned user interface also includes a playback device type option; a determination unit is also used to respond to the user's fifth operation and determine the type of playback device from the playback device type options; and an acquisition unit is specifically used to render the first single-object audio track based on the first sound source position and type to obtain the rendered first single-object audio track.
[0150] Optionally, in a possible implementation of the fourth aspect, the above-mentioned acquisition unit is specifically used to separate a first single-object audio track from an original audio track in a multimedia file, the original audio track is obtained by synthesizing at least the first single-object audio track and the second single-object audio track, and the second single-object audio track corresponds to the second sound object.
[0151] Optionally, in a possible implementation of the fourth aspect, the above-mentioned music features include: at least one of music structure, music emotion and singing mode.
[0152] Optionally, in a possible implementation of the fourth aspect, the above-mentioned acquisition unit is also used to separate a second single object audio track from the multimedia file; determine a second sound source position of a second sound object, and spatially render the second single object audio track based on the second sound source position to obtain a rendered second single object audio track.
[0153] A fifth aspect of the present application provides a rendering device, which executes the method in the aforementioned first aspect or any possible implementation of the first aspect, or executes the method in the aforementioned second aspect or any possible implementation of the second aspect.
[0154] In a sixth aspect, the present application provides a rendering device, including: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the rendering device implements the method in the above-mentioned first aspect or any possible implementation of the first aspect, or the rendering device implements the above-mentioned second aspect or any possible implementation of the second aspect.
[0155] The seventh aspect of the present application provides a computer-readable medium having a computer program or instruction stored thereon. When the computer program or instruction is executed on a computer, the computer executes the method of the first aspect or any possible implementation of the first aspect, or the computer executes the method of the second aspect or any possible implementation of the second aspect.
[0156] An eighth aspect of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method in the aforementioned first aspect or any possible implementation of the first aspect, the second aspect or any possible implementation of the second aspect.
[0157] Among them, the technical effects brought about by the third, fifth, sixth, seventh, and eighth aspects or any possible implementation methods thereof can refer to the technical effects brought about by the first aspect or different possible implementation methods of the first aspect, and will not be repeated here.
[0158] Among them, the technical effects brought about by the fourth, fifth, sixth, seventh, and eighth aspects or any possible implementation methods thereof can refer to the technical effects brought about by the second aspect or different possible implementation methods of the second aspect, and will not be repeated here.
[0159] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages: obtaining a first single object audio track based on a multimedia file, the first single object audio track corresponding to a first sound object; determining a first sound source position of the first sound object based on reference information, and spatially rendering the first single object audio track based on the first sound source position to obtain a rendered first single object audio track. The stereoscopic sense of the first single object audio track corresponding to the first sound object in the multimedia file can be enhanced, providing users with an immersive stereo sound effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0160] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0161] Figure 1 A schematic diagram of the structure of a system architecture provided for this application;
[0162] Figure 2 A schematic diagram of a convolutional neural network structure provided for this application;
[0163] Figure 3 Another convolutional neural network structure diagram provided for this application;
[0164] Figure 4 A schematic diagram of a chip hardware structure provided for this application;
[0165] Figure 5 A schematic flow chart of a separation network training method provided in the present application;
[0166] Figure 6 A schematic diagram of the structure of a separation network provided in this application;
[0167] Figure 7 A schematic diagram of the structure of another separation network provided in this application;
[0168] Figure 8 A structural diagram of another system architecture provided for this application;
[0169] Fig. 9 A schematic diagram of an application scenario provided for this application;
[0170] Fig.10 A flowchart of the rendering method provided by this application;
[0171] Fig.11A flowchart of a playback device calibration method provided by the present application;
[0172] Figure 12-Figure 17 Several schematic diagrams of the display interface of the rendering device provided for this application;
[0173] Fig.18 Schematic diagram of the mobile phone orientation provided for this application;
[0174] Fig.19 Another schematic diagram of the display interface of the rendering device provided by this application;
[0175] Fig. 20 A schematic diagram of using a mobile phone to determine the location of a sound source provided by this application;
[0176] Figure 21-Figure 47 Several other schematic diagrams of the display interface of the rendering device provided for this application;
[0177] Fig.48 A schematic diagram of the structure of the external speaker system provided by this application in a spherical coordinate system;
[0178] Figure 49-Figure 50 Several schematic diagrams of shared rendering rules between users provided for this application;
[0179] Figure 51-Figure 53 Several other schematic diagrams of the display interface of the rendering device provided for this application;
[0180] Fig.54 A schematic diagram of user interaction in the game scenario of the Sound Hunter provided in this application;
[0181] Figure 55-Figure 57 Several schematic diagrams of user interaction in multi-person interaction scenarios provided by this application;
[0182] Figure 58-Figure 61 Several structural schematic diagrams of rendering devices provided for this application;
[0183] Fig.62 A schematic diagram of the structure of the sensor device provided in this application. DETAILED DESCRIPTION
[0184] The embodiment of the present application provides a rendering method, which can enhance the stereoscopic spatial sense of a first single object audio track corresponding to a first sound-emitting object in a multimedia file, and provide users with an immersive stereo sound effect.
[0185] The technical solutions in the embodiments of the present invention will be described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0186] To facilitate understanding, the relevant terms and concepts mainly involved in the embodiments of the present application are first introduced below.
[0187] 1. Neural network.
[0188] A neural network can be composed of neural units. A neural unit can be a s The output of the operation unit with the intercept 1 as input can be:
[0189]
[0190] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For X s The weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.
[0191] 2. Deep neural network.
[0192] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. According to the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Of course, a deep neural network may not include hidden layers, which is not limited here.
[0193] The work of each layer in a deep neural network can be expressed mathematically as To describe: From a physical level, the work of each layer in a deep neural network can be understood as completing the transformation from input space to output space (i.e., from row space to column space of a matrix) through five operations on the input space (a set of input vectors). These five operations include: 1. Dimension increase / reduction; 2. Enlargement / reduction; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are represented by Completed, the operation of 4 is done by Completed, operation 5 is implemented by α(). The word "space" is used here because the classified object is not a single thing, but a class of things, and space refers to the set of all individuals of this class of things. Among them, W is a weight vector, and each value in the vector represents the weight value of a neuron in this layer of the neural network. The vector W determines the spatial transformation from the input space to the output space described above, that is, the weight W of each layer controls how to transform the space. The purpose of training a deep neural network is to eventually obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by many layers of vectors W). Therefore, the training process of a neural network is essentially to learn how to control spatial transformation, and more specifically to learn the weight matrix.
[0194] 3. Convolutional neural network.
[0195] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as convolving the same trainable filter with an input image or convolution feature plane (feature map). Convolution layer refers to the neuron layer in convolutional neural network that performs convolution processing on the input signal. In the convolution layer of convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolution layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The implicit principle is that the statistical information of a part of the image is the same as that of other parts. This means that the image information learned in a part can also be used in another part. Therefore, for all positions on the image, the same image information obtained by learning can be used. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.
[0196] The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of sharing weights is to reduce the connection between the layers of the convolutional neural network, while reducing the risk of overfitting. The separation network, recognition network, detection network, depth estimation network and other networks in the embodiments of the present application can all be CNNs.
[0197] 4. Recurrent Neural Network (RNN).
[0198] In the traditional neural network model, the layers are fully connected, and the nodes between each layer are disconnected. However, this ordinary neural network cannot solve many problems. For example, predicting the next word in a sentence, because the previous and next words in a sentence are not independent, the previous word is generally needed. Recurrent neural network (RNN) refers to a sequence in which the current output is also related to the previous output. The specific manifestation is that the network will remember the previous information, save it in the internal state of the network, and apply it to the calculation of the current output.
[0199] 5. Loss function.
[0200] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict lower, and keep adjusting until the neural network can predict the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.
[0201] 6. Head-related transfer function.
[0202] Head related transfer function (HRTF): The sound waves emitted by the sound source are scattered by the head, auricle, trunk, etc. and then reach the ears. The physical process can be regarded as a linear time-invariant sound filter system, and its characteristics can be described by HRTF, that is, HRTF describes the transmission process of sound waves from the sound source to the ears. A more vivid explanation is: if the audio signal emitted by the sound source is X, and the audio signal corresponding to X after being transmitted to the predetermined position is Y, then X*Z=Y (X convolution Z equals Y), where Z is the HRTF.
[0203] 7. Audio track.
[0204] An audio track is a track for recording audio data. Each audio track has one or more attribute parameters, including audio format, bit rate, dubbing language, sound effects, number of channels, volume, etc. When the audio data is multi-track, two different audio tracks have at least one different attribute parameter, or at least one attribute parameter in two different audio tracks has a different value. An audio track can be a single track or multiple tracks (or mixed tracks). Among them, a single track can correspond to one or more sound objects, and a multi-track includes at least two single tracks. Generally, a single object track corresponds to a sound object.
[0205] 8. Short-time Fourier transform.
[0206] The core idea of short-time Fourier transform (STFT) is "windowing", that is, decomposing the entire time domain process into countless small processes of equal length, each small process is approximately stable, and then performing fast Fourier transform (FFT) on each small process.
[0207] The following introduces the system architecture provided by the embodiments of the present application.
[0208] See attached Figure 1, an embodiment of the present invention provides a system architecture 100. As shown in the system architecture 100, a data acquisition device 160 is used to collect training data. In the embodiment of the present application, the training data includes: a multimedia file, the multimedia file includes an original audio track, and the original audio track corresponds to at least one sound object. The training data is stored in a database 130, and the training device 120 obtains a target model / rule 101 based on the training data maintained in the database 130. The following will describe in more detail how the training device 120 obtains the target model / rule 101 based on the training data using the first embodiment. The target model / rule 101 can be used to implement the rendering method provided in the embodiment of the present application, wherein the target model / rule 101 has multiple situations. In one situation of the target model / rule 101 (when the target model / rule 101 is the first model), the multimedia file is input into the target model / rule 101 to obtain the first single-object audio track corresponding to the first sound object. In another case of the target model / rule 101 (when the target model / rule 101 is the second model), the multimedia file is input into the target model / rule 101 after relevant preprocessing, and the first single object audio track corresponding to the first sound-emitting object can be obtained. The target model / rule 101 in the embodiment of the present application may specifically include a separation network, and may further include a recognition network, a detection network, a depth estimation network, etc., which are not specifically limited here. In the embodiment provided in the present application, the separation network is obtained by training training data. It should be noted that in actual applications, the training data maintained in the database 130 does not necessarily come from the collection of the data acquisition device 160, but may also be received and obtained from other devices. It should also be noted that the training device 120 does not necessarily train the target model / rule 101 completely based on the training data maintained by the database 130, and it is also possible to obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.
[0209] The target model / rule 101 obtained through training by the training device 120 can be applied to different systems or devices, such as Figure 1 The execution device 110 shown in the figure may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR), a vehicle terminal, etc., or a server or a cloud. Figure 1In the embodiment, the execution device 110 is configured with an I / O interface 112 for data interaction with an external device. The user can input data to the I / O interface 112 through the client device 140. The input data in the embodiment of the present application may include: multimedia files, which can be input by the user or uploaded by the user through an audio device, and of course can also come from a database, which is not limited here.
[0210] The preprocessing module 113 is used to preprocess the multimedia file received by the I / O interface 112. In the embodiment of the present application, the preprocessing module 113 can be used to perform short-time Fourier transform processing on the audio track in the multimedia file to obtain a time-frequency spectrum.
[0211] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.
[0212] Finally, the I / O interface 112 returns the processing result, such as the first single-object audio track corresponding to the first sound-emitting object obtained as described above, to the client device 140 for providing to the user.
[0213] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different goals or different tasks. The corresponding target models / rules 101 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.
[0214] In the attached Figure 1 In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 130.
[0215] It is worth noting that Figure 1 This is only a schematic diagram of a system architecture provided by an embodiment of the present invention. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. Figure 1 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.
[0216] like Figure 1 As shown, the target model / rule 101 is obtained through training according to the training device 120. The target model / rule 101 may be a separation network in the embodiment of the present application. Specifically, in the network provided in the embodiment of the present application, the separation network may be a convolutional neural network or a recurrent neural network.
[0217] Since CNN is a very common neural network, Figure 2 The focus is on a detailed introduction to the structure of CNN. As mentioned in the previous basic concept introduction, a convolutional neural network is a deep neural network with a convolutional structure and a deep learning architecture. A deep learning architecture refers to multiple levels of learning at different levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network in which each neuron can respond to the image input into it.
[0218] like Figure 2 As shown, the convolutional neural network (CNN) 100 may include an input layer 110 , a convolutional layer / pooling layer 120 , wherein the pooling layer is optional, and a neural network layer 130 .
[0219] Convolutional layer / pooling layer 120:
[0220] Convolutional Layer:
[0221] like Figure 2 The convolution layer / pooling layer 120 shown may include layers 121-126 as shown in the examples. In one implementation, layer 121 is a convolution layer, layer 122 is a pooling layer, layer 123 is a convolution layer, layer 124 is a pooling layer, layer 125 is a convolution layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolution layers, layer 123 is a pooling layer, layers 124 and 125 are convolution layers, and layer 126 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.
[0222] Taking the convolution layer 121 as an example, the convolution layer 121 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator can essentially be a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is usually processed one pixel after another (or two pixels after two pixels... depending on the value of the step length stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is not used, but multiple weight matrices of the same dimension are applied. The output of each weight matrix is stacked to form the depth dimension of the convolved image. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract image edge information, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to blur unnecessary noise in the image... The multiple weight matrices have the same dimensions, and the feature maps extracted by the multiple weight matrices with the same dimensions are also of the same dimension. The extracted feature maps with the same dimensions are then merged to form the output of the convolution operation.
[0223] The weight values in these weight matrices need to be acquired through a lot of training in practical applications. The weight matrices formed by the weight values acquired through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.
[0224] When the convolutional neural network 100 has multiple convolutional layers, the initial convolutional layer (for example, 121) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by the later convolutional layers (for example, 126) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.
[0225] Pooling layer:
[0226] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer, such as Figure 2The layers 121-126 illustrated in 120 may be a convolution layer followed by a pooling layer, or may be multiple convolution layers followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and / or a maximum pooling operator for sampling the input image to obtain an image of smaller size. The average pooling operator may calculate the pixel values in the image within a specific range to generate an average value. The maximum pooling operator may take the pixel with the largest value in the range as the result of the maximum pooling within a specific range. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.
[0227] Neural network layer 130:
[0228] After being processed by the convolution layer / pooling layer 120, the convolution neural network 100 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 120 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 100 needs to use the neural network layer 130 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 130 may include multiple hidden layers (such as Figure 2 131, 132 to 13n) and the output layer 140 shown, the parameters contained in the multi-layer hidden layers can be pre-trained and acquired according to relevant training data of a specific task type. For example, the task type may include multi-track separation, image recognition, image classification, image super-resolution reconstruction, etc.
[0229] After the multiple hidden layers in the neural network layer 130, that is, the last layer of the entire convolutional neural network 100 is the output layer 140, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 100 (such as Figure 2 The propagation from 110 to 140 is forward propagation), and the reverse propagation (such as Figure 2 The propagation from 140 to 110 is called back propagation) and then the weight values and biases of the aforementioned layers will begin to be updated to reduce the loss of the convolutional neural network 100 and the error between the result output by the convolutional neural network 100 through the output layer and the ideal result.
[0230] It should be noted that if Figure 2The convolutional neural network 100 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models, for example, Figure 3 The multiple convolutional layers / pooling layers shown are operated in parallel, and the features extracted respectively are input to the full neural network layer 130 for processing.
[0231] The following introduces a chip hardware structure provided by an embodiment of the present application.
[0232] Figure 4 A chip hardware structure provided in an embodiment of the present invention includes a neural network processor 40. The chip can be arranged as follows: Figure 1 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 1 The training device 120 shown in FIG. 1 is used to complete the training work of the training device 120 and output the target model / rule 101. Figure 2 The algorithms of each layer in the convolutional neural network shown in the figure can be used in Figure 4 is implemented in the chip shown.
[0233] The neural network processor 40 can be a neural network processor (neural-network processing unit, NPU), a tensor processing unit (tensor processing unit, TPU), or a graphics processing unit (graphics processing unit, GPU) and any other processor suitable for large-scale XOR operation processing. Take NPU as an example: the neural network processor NPU40 is mounted on the main central processing unit (central processing unit, CPU) (host CPU) as a coprocessor, and the main CPU assigns tasks. The core part of the NPU is the operation circuit 403, and the controller 404 controls the operation circuit 403 to extract data from the memory (weight memory or input memory) and perform operations.
[0234] In some implementations, the operation circuit 403 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 403 is a two-dimensional systolic array. The operation circuit 403 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 403 is a general-purpose matrix processor.
[0235] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 402 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 401 and performs matrix operation with matrix B, and the partial result or final result of the matrix is obtained and stored in the accumulator 408.
[0236] The vector calculation unit 407 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 407 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.
[0237] In some implementations, the vector calculation unit 407 can store the processed output vector to the unified buffer 406. For example, the vector calculation unit 407 can apply a nonlinear function to the output of the operation circuit 403, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 407 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 403, such as for use in a subsequent layer in a neural network.
[0238] The unified memory 406 is used to store input data and output data.
[0239] The weight data is directly transferred from the external memory to the input memory 401 and / or the unified memory 406 through the direct memory access controller 405 (DMAC), the weight data in the external memory is stored in the weight memory 402, and the data in the unified memory 506 is stored in the external memory.
[0240] The bus interface unit (BIU) 410 is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 409 through the bus.
[0241] An instruction fetch buffer 409 connected to the controller 404 is used to store instructions used by the controller 404 .
[0242] The controller 404 is used to call the instructions cached in the memory 409 to control the working process of the computing accelerator.
[0243] Generally, the unified memory 406, the input memory 401, the weight memory 402 and the instruction fetch memory 409 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.
[0244] in, Figure 2 or Figure 3 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 403 or the vector calculation unit 407.
[0245] The following is a detailed introduction to the training method and rendering method of the separation network of the embodiment of the present application in conjunction with the accompanying drawings.
[0246] First, combine Figure 5 The training method of the separation network in the embodiment of the present application is introduced in detail. Figure 5 The method shown can be executed by a training device for a separate network, which can be a cloud service device or a terminal device, for example, a computer, a server, or other device with sufficient computing power to execute the training method for a separate network, or a system composed of a cloud service device and a terminal device. Figure 1 Training equipment in 120, Figure 4 The neural network processor 40 in is executed.
[0247] Optionally, the training method can be processed by the CPU, or by the CPU and GPU together, or other processors suitable for neural network calculations can be used without using a GPU, and this application does not impose any restrictions.
[0248] It is understandable that, if the number of sound objects corresponding to the original audio track in the multimedia file in the embodiment of the present application is multiple, the separation network can be used to separate the original audio track to obtain at least one single object audio track. Of course, if the original audio track in the multimedia file corresponds to only one sound object, then the original audio track is a single object audio track and does not need to be separated using a separation network.
[0249] The training method may include step 501 and step 502. Step 501 and step 502 are described in detail below.
[0250] Step 501: Obtain training data.
[0251] The training data in the embodiment of the present application is obtained by synthesizing at least the initial third single-object audio track and the initial fourth single-object audio track, and can also be understood as the training data including multiple audio tracks synthesized from single-object audio tracks corresponding to at least two sound objects. The initial third single-object audio track corresponds to the third sound object, and the initial fourth single-object audio track corresponds to the fourth sound object. In addition, the training data can also include images that match the original audio track, wherein the training data can also be a multimedia file, which includes the above-mentioned multiple audio tracks. In addition to the audio track, the multimedia file can also include a video track or a text track (or a barrage track), etc., which is not specifically limited here.
[0252] The audio tracks (original audio tracks, first single object audio tracks, etc.) in the embodiments of the present application may include vocal tracks, instrument tracks (for example, drum tracks, piano tracks, trumpet tracks, etc.), airplane sounds, and other audio tracks generated by sound-emitting objects (or called sound-emitting objects). The specific sound-emitting objects corresponding to the audio tracks are not limited here.
[0253] In the embodiment of the present application, the training data can be obtained by directly recording the sound of the sound-emitting object, or by the user inputting audio information and video information, or by receiving information sent by an acquisition device. In actual applications, there are other ways to obtain training data, and the specific method of obtaining training data is not limited here.
[0254] Step 502: Using the training data as the input of the separation network, training the separation network with the goal of making the value of the loss function less than a first threshold, and obtaining a trained separation network.
[0255] The separation network in the embodiments of the present application may be referred to as a separation neural network, a separation model, or a separation neural network model, etc., which is not specifically limited here.
[0256] The loss function is used to indicate the difference between the separated third single object audio track and the initial third single object audio track.
[0257] In this case, the separation network is trained with the goal of reducing the value of the loss function, that is, the difference between the third single-object audio track output by the separation network and the initial third single-object audio track is continuously reduced. The training process can be understood as a separation task. The loss function can be understood as the loss function corresponding to the separation task. Among them, the output of the separation network (at least one single-object audio track) is a single-object audio track corresponding to at least one sound object in the input (audio track). The third sound object belongs to the same type as the first sound object, and the second sound object belongs to the same type as the fourth sound object. For example: the first sound object and the third sound object are both human voices, but the first sound object can be user A, and the second sound object can be user B. In other words, the third single-object audio track and the first single-object audio track are audio tracks corresponding to sounds emitted by different people. The third sound object and the first sound object in the embodiment of the present application can be two sound objects of the same type, or one sound object of the same type, which is not specifically limited here.
[0258] Optionally, the training data input to the separation network includes original audio tracks corresponding to at least two sound objects. The separation network can output a single object audio track corresponding to one of the at least two sound objects, or can output single object audio tracks corresponding to each of the at least two sound objects.
[0259] Exemplarily, the multimedia file includes a track corresponding to human voice, a track corresponding to piano, and a track corresponding to vehicle sound. After the multimedia file is separated through the separation network, one single object track (e.g., a single object track corresponding to human voice), two single object tracks (e.g., a single object track corresponding to human voice and a single object track corresponding to vehicle sound) or three single object tracks can be obtained.
[0260] In one possible implementation, the separation network is as follows Figure 6 As shown, the separation network includes a one-dimensional convolution and a residual structure. Among them, adding a residual structure can improve the efficiency of gradient transfer. Of course, the separation network can also include activation, pooling, etc. The specific structure of the separation network is not limited here. Figure 6 The separation network shown takes the signal source (i.e., the signal corresponding to the audio track in the multimedia file) as input, transforms it through multiple convolutions and deconvolutions, and outputs the object signal (a single audio track corresponding to a sound object). In addition, the temporal correlation can be improved by adding a recurrent neural network module, and the connection between high- and low-dimensional features can be improved by connecting different output layers.
[0261] In another possible implementation, the separation network is as follows Figure 7As shown, before the signal source is input into the separation network, the signal source can be preprocessed first, for example, the signal source can be subjected to STFT mapping to obtain a time-frequency spectrum. The amplitude spectrum in the time-frequency spectrum is transformed by two-dimensional convolution and deconvolution to obtain a masked spectrum (the screened spectrum), and the masked spectrum is combined with the amplitude spectrum to obtain a target amplitude spectrum. The target amplitude spectrum is then multiplied by the phase spectrum to obtain the target time-frequency spectrum. The target time-frequency spectrum is mapped by inverse short-time Fourier transform (iSTFT) to obtain the object signal (a single audio track corresponding to a sounding object). Among them, the connection between high- and low-dimensional features can also be improved by connecting different output layers, the efficiency of gradient transfer can be improved by adding a residual structure, and the time series correlation can be improved by adding a recurrent neural network module.
[0262] in, Figure 6 The input can also be understood as a one-dimensional time domain signal. Figure 7 The input is a two-dimensional time-frequency spectrum signal.
[0263] The above two separation models are just examples. In practical applications, there are other possible structures. The input of the separation model can be a time domain signal, the output can be a time domain signal, the input of the separation model can be a time-frequency domain signal, the output can be a time-frequency domain signal, etc. The structure, input or output of the separation model is not specifically limited here.
[0264] Optionally, before inputting the separation network, the multiple audio tracks in the multimedia file can also be identified through the recognition network, and the multiple audio tracks including the number of tracks and the object category (for example: vocals, drums, etc.) can be identified, which can reduce the training time of the separation network. Of course, the separation network can also include a recognition subnetwork for identifying multiple audio tracks, which is not specifically limited here. Among them, the input of the recognition network can be a time domain signal, and the output is a category probability. It is equivalent to inputting the time domain signal into the recognition network, obtaining the probability that the object is a certain category, and selecting the category whose probability exceeds the threshold as the classification category. The object here can also be understood as a sound object.
[0265] For example, the input of the above recognition network is a multimedia file synthesized from the audio corresponding to vehicle A and vehicle B. The multimedia file is input into the recognition network, and the recognition network can output the category as vehicle. Of course, if the training data is comprehensive enough, the recognition network can also identify the specific type of vehicle, which is equivalent to further fine-grained recognition. The recognition network is set according to actual needs and is not specifically limited here.
[0266] It should be noted that the training process may also use other training methods instead of the aforementioned training methods, which is not limited here.
[0267] See also Figure 8, another system architecture provided by the present application. The system architecture includes an input module, a function module, a database module and an output module. Each module is described in detail below:
[0268] 1. Input module.
[0269] The input module includes a database option submodule, a sensor information acquisition submodule, a user interface input submodule and a file input submodule. The above four submodules can also be understood as four input methods.
[0270] The database option submodule is used to perform spatial rendering according to the rendering method stored in the database selected by the user.
[0271] The sensor information acquisition submodule is used to specify the spatial position of a specific sound-emitting object based on a sensor (which can be a sensor in a rendering device or another sensor device, and the specific details are not limited here). In this way, the user can select the position of a specific sound-emitting object.
[0272] The user interface input submodule is used to determine the spatial position of a specific sound-emitting object in response to the user's operation on the user interface. Optionally, the user can control the spatial position of the specific sound-emitting object by clicking, dragging, etc.
[0273] The file input submodule is used to track a specific sound object based on image information or text information (eg, lyrics, subtitles, etc.), and then determine the spatial position of the specific sound object based on the position of the tracked specific sound object.
[0274] 2. Functional module.
[0275] The functional modules include a signal transmission submodule, an object recognition submodule, a calibration submodule, an object tracking submodule, an orientation calculation submodule, an object separation submodule and a rendering submodule.
[0276] The signal transmission submodule is used to receive and send information. Specifically, it can receive input information from the input module and output feedback information to other modules. For example, the feedback information includes information such as position change information of a specific sound object and separated single object audio tracks. Of course, the signal transmission submodule can also be used to feed back the identified object information to the user through the user interface (UI), etc., which is not limited here.
[0277] The object recognition submodule is used to recognize all the object information of the multi-track information sent by the input module received by the signal transmission submodule. The object here refers to the sound object (or sound object), such as human voice, drum sound, airplane sound, etc. Optionally, the object recognition submodule can be the aforementioned Figure 5The illustrated embodiments describe an identification network or an identification sub-network in a separate network.
[0278] The calibration submodule is used to calibrate the initial state of the playback device. For example, when the playback device is a headset, the calibration submodule is used to calibrate the headset. When the playback device is an external speaker, the calibration submodule is used to calibrate the external speaker. For headset calibration: the default sensor ( Fig. 9 The relationship between the sensor device and the playback device will be explained. The initial state is directly in front of the headset, and the calibration is performed based on the direct front. You can also obtain the actual position of the user visiting the sensor to ensure that the direct front of the sound image is directly in front of the headset. For external speaker calibration: first obtain the coordinate position of each external speaker (you can obtain it interactively based on the sensor of the user terminal, and then Fig. 9 There will be corresponding descriptions). After the calibration submodule calibrates the playback device, the calibrated playback device information is transmitted to the database module through the signal transmission submodule.
[0279] The object tracking submodule is used to track the motion trajectory of a specific sound-emitting object. The specific sound-emitting object can be a sound-emitting object in text or image displayed in a multimodal file (for example: audio information and video information corresponding to the audio information, audio information and text information corresponding to the audio information, etc.). Optionally, the object tracking submodule can also be used to render the motion trajectory on the audio side. In addition, the object tracking submodule can also include a target recognition network and a depth estimation network, the target recognition network is used to identify the specific sound-emitting object to be tracked, and the depth estimation network is used to obtain the relative coordinates of the specific sound-emitting object in the image (described in detail in subsequent embodiments), so that the object tracking submodule renders the orientation and motion trajectory of the audio corresponding to the specific sound-emitting object according to the relative coordinates.
[0280] The position calculation submodule is used to convert the information obtained by the input module (for example, sensor information, input information of the UI interface, file information, etc.) into position information (also known as the sound source position). There are corresponding conversion methods for different information, and the specific process of conversion will be described in detail in the subsequent embodiments.
[0281] The object separation submodule is used to separate at least one single object track from a multimedia file (or multimedia information) or multi-track information. For example, extracting a separate vocal track (i.e., an audio file containing only vocals) from a song. Figure 5 The separation network in the embodiment shown. Further, the structure of the object separation submodule can be as follows Figure 6 or Figure 7 As shown, no specific limitation is made here.
[0282] The rendering submodule is used to obtain the sound source position obtained by the orientation calculation submodule and perform spatial rendering on the sound source position. Furthermore, the corresponding rendering method can be determined according to the playback device selected by the input information of the UI in the input module. For different playback devices, the rendering method is different, and the rendering process will be described in detail in the subsequent embodiments.
[0283] 3. Database module.
[0284] The database module includes a database selection submodule, a rendering rule editing submodule and a rendering rule sharing submodule.
[0285] The database selection submodule is used to store rendering rules. The rendering rules can be the default two-channel / multi-channel audio tracks that come with the system when initialized, which are converted into three-dimensional (3D) spatial sense rendering rules, or they can be rendering rules saved by users. Optionally, different objects can correspond to the same rendering rules, or different objects can correspond to different rendering rules.
[0286] The rendering rule editing submodule is used to re-edit the saved rendering rule. Optionally, the saved rendering rule may be a rendering rule stored in the database selection submodule, or a newly input rendering rule, which is not specifically limited here.
[0287] The rendering rule sharing submodule is used to upload the rendering rules to the cloud, and / or to download specific rendering rules from the rendering rule database in the cloud. For example, the rendering rule sharing module can upload user-defined rendering rules to the cloud and share them with other users. The user can select the rendering rules shared by other users that match the multi-track information to be played from the rendering rule database stored in the cloud, and download them to the database on the terminal side as a data file for audio 3D rendering rules.
[0288] 4. Output module.
[0289] The output module is used to play the rendered single-object audio track or the target audio track (obtained according to the original audio track and the rendered single-object audio track) through a playback device.
[0290] First, the application scenarios to which the rendering method provided in the embodiment of the present application is applicable are described:
[0291] See also Fig. 9 , the application scenario includes a control device 901 , a sensor device 902 and a playback device 903 .
[0292] The playback device 903 in the embodiment of the present application can be an external speaker, or headphones (such as in-ear headphones, headphones, etc.), or a large screen (such as a projection screen), etc., and is not limited here.
[0293] The control device 901 and the sensor device 902 , as well as the sensor device 902 and the playback device 903 , may be connected via wired, wireless fidelity (WIFI), mobile data network or other connection methods, which are not specifically limited here.
[0294] The control device 901 in the embodiment of the present application is a terminal device for serving users. The terminal device may include a head-mounted display device (head mount display, HMD), the head-mounted display device may be a combination of a virtual reality (virtual reality, VR) box and a terminal, a VR all-in-one machine, a personal computer (personal computer, PC) VR, an augmented reality (augmented reality, AR) device, a mixed reality (mixed reality, MR) device, etc. The terminal device may also include a cellular phone (cellular phone), a smart phone (smart phone), a personal digital assistant (personal digital assistant, PDA), a tablet computer, a laptop computer (laptop computer), a personal computer (personal computer, PC), a vehicle-mounted terminal, etc., which is not limited here.
[0295] The sensor device 902 in the embodiment of the present application is a device for sensing direction and / or position, which can be a laser pen, a mobile phone, a smart watch, a smart bracelet, a device with an inertial measurement unit (IMU), a device with a simultaneous localization and mapping (SLAM) sensor, etc., and is not limited to the specifics here.
[0296] The playback device 903 in the embodiment of the present application is a device for playing audio or video, which can be an external device (for example, a speaker, a terminal device with the function of playing audio or video), or an internal device (for example, in-ear headphones, headphones, AR devices, VR devices, etc.), etc., and is not limited to the specifics here.
[0297] Understandably, Fig. 9 The number of each device in the application scenario shown can be one or more. For example, the external speaker can include multiple devices. The specific number of each device is not limited here.
[0298] The control device, sensor device, and playback device in the embodiments of the present application may be three devices, two devices, or one device, and the specific details are not limited here.
[0299] One possible way to achieve this is Fig. 9 The control device and the sensor device in the application scenario shown are the same device. For example, the control device and the sensor device are the same mobile phone, and the playback device is headphones. Another example is that the control device and the sensor device are the same mobile phone, and the playback device is an external speaker device (also called an external speaker device system, which includes one or more external speakers).
[0300] Another possible way to implement this is Fig. 9 The control device and the playback device in the application scenario shown are the same device. For example, the control device and the playback device are the same computer. Another example is that the control device and the playback device are the same large screen.
[0301] Another possible way to implement this is Fig. 9 The control device, sensor device, and playback device in the application scenario shown are the same device. For example, the control device, sensor device, and playback device are the same tablet computer.
[0302] The rendering method of the embodiment of the present application is described in detail below in combination with the above application scenarios and drawings.
[0303] See also Fig.10 An embodiment of a rendering method provided in an embodiment of the present application can be executed by a rendering device or by a component of a rendering device (such as a processor, a chip, or a chip system, etc.). The embodiment includes steps 1001 to 1004.
[0304] In the embodiment of the present application, the rendering device may have the following Fig. 9 The functions of the control device, the sensor device and / or the playback device are not specifically limited here. The rendering method is described below by taking the rendering device as a control device (such as a notebook), the sensor device as a device with an IMU (such as a mobile phone), and the playback device as an external device (such as a speaker) as an example.
[0305] The sensor described in the embodiments of the present application may refer to a sensor in a rendering device, or may refer to a sensor in a device other than a rendering device (such as the aforementioned sensor device), and is not specifically limited here.
[0306] Step 1001: Calibrate the playback device. This step is optional.
[0307] Optionally, before the playback device plays the rendered audio track, the playback device may be calibrated first, the purpose of the calibration being to enhance the authenticity of the spatial effect of the rendered audio track.
[0308] In the embodiment of the present application, there are many ways to calibrate the playback device. The following is an example of a playback device calibration process using the playback device being an external playback device as an example. Fig.11 This embodiment provides a method for calibrating a playback device, which includes steps 1 to 5.
[0309] Optionally, before calibration, the user's mobile phone is connected to the external speaker in the same manner as described above. Fig. 9 The connection method between the sensor device and the playback device in the illustrated embodiment is similar and will not be described again here.
[0310] Step 1. Determine the playback device type.
[0311] In the embodiment of the present application, the rendering device can determine the playback device type through user operations, or can adaptively detect the playback device type, or can determine the playback device type through default settings, or can determine the playback device type in other ways, which are not limited here.
[0312] For example, if the rendering device determines the playback device type through the user's operation, the rendering device may display the following information: Fig.12 The interface shown in FIG. 1 includes an icon for selecting the playback device type. In addition, the interface may also include an icon for selecting an input file, an icon for selecting a rendering method (i.e., a reference information option), a calibration icon, an audio hunter icon, an object bar, a volume, a duration progress, and a ball view (or a three-dimensional view). Fig.13 As shown, the user can click on the "select playback device type icon" 101. Fig.14 As shown, in response to the click operation, the rendering device displays a drop-down menu, which may include "external speaker option" and "headphone option". Further, the user may click "external speaker option" 102 to determine the type of the playback device as an external speaker. Fig.15 As shown, in the interface displayed by the rendering device, "Select playback device type" can be replaced by "External playback device" to prompt the user that the current playback device type is an external playback device. It can also be understood that the rendering device displays as follows Fig.12 The rendering device receives the fifth operation of the user (ie, Fig.13 and Fig.14 The rendering device responds to the fifth operation by selecting the playback device type as an external playback device from the playback device type options.
[0313] Furthermore, since this method is intended to calibrate playback devices, such as Fig.16 As shown, the user can also click on the "calibration icon" 103, such as Fig.17 As shown, in response to the click operation, the rendering device displays a drop-down menu, which may include a "default option" and a "manual calibration option". Further, the user may click on the "manual calibration option" 104 to determine that the calibration method is automatic calibration, which may be understood as the user calibrating the playback device using a mobile phone (i.e., a sensor device).
[0314] Fig.14 In the example, the drop-down menu of "Select playback device type icon" includes "external speaker options" and "headphone options". In actual applications, the drop-down menu can also include specific type options of headphones, such as headphones, in-ear headphones, wired headphones, Bluetooth headphones, etc., which are not limited here.
[0315] Fig.17 In the example, only the drop-down menu of the "calibration icon" includes "default options" and "manual calibration options". In actual applications, the drop-down menu may also include other types of options, which are not limited here.
[0316] Step 2: Confirm the test audio.
[0317] The test audio in the embodiment of the present application can be a default test signal (such as pink noise), or can be a test signal generated by the above Figure 5 The single-object audio track corresponding to the human voice separated from the song (i.e., the multimedia file is a song) by the separation network in the illustrated embodiment may also be the audio corresponding to other single-object audio tracks in the song, or may be audio that only includes single-object audio tracks, etc., and no specific limitation is made here.
[0318] Exemplarily, the user can click on the “select input file icon” in the interface displayed by the rendering device to select the test audio.
[0319] Step 3: Get the attitude angle of the phone and the distance between the sensor and the external speaker.
[0320] After determining the test audio source, the external speaker plays the test audio in turn, and the user holds a sensor device (such as a mobile phone) and points it at the external speaker that is playing the test audio. After the position of the mobile phone is stable, the current orientation of the mobile phone and the signal energy of the received test audio are recorded, and the distance between the mobile phone and the external speaker is calculated according to the following formula 1. When there are multiple external speakers, the operation is similar and will not be repeated here. Among them, the stability of the mobile phone position can be understood as the variance of the mobile phone orientation is less than a threshold value (for example, 5 degrees) within a period of time (for example, 200 milliseconds).
[0321] Optionally, if the playback device is two external speakers, the first external speaker plays the test audio first, and the user points the mobile phone at the first external speaker. After the first external speaker is calibrated, the user points the mobile phone at the second external speaker for calibration.
[0322] In the embodiments of the present application, the orientation of the mobile phone may refer to the attitude angle of the mobile phone, which may include an azimuth angle and a tilt angle (or a roll angle), or the attitude angle may include an azimuth angle, a tilt angle, and a pitch angle. The azimuth angle represents the angle around the z-axis, the tilt angle represents the angle around the y-axis, and the pitch angle represents the angle around the x-axis. The relationship between the orientation of the mobile phone and the x-axis, y-axis, and z-axis may be as follows: Fig.18 shown.
[0323] Exemplarily, continuing the above example, the playback devices are two external speakers. The first external speaker plays the test audio first, and the user points the phone at the first external speaker, and records the current orientation of the phone and the signal energy received from the test audio. Then, the second external speaker plays the test audio, and the user points the phone at the second speaker, and records the current orientation of the phone and the signal energy received from the test audio.
[0324] Furthermore, during the calibration of the external speaker, the rendering device may display Fig.19 The interface shown, where the right side of the interface is a ball view, in which calibrated external speakers and external speakers being calibrated can be displayed. In addition, uncalibrated external speakers (not shown in the figure) can also be displayed, which is not limited here. The center of the ball view is the user's position (which can also be understood as the position where the user holds the mobile phone. Since the user holds the mobile phone, the position of the mobile phone is close to the user's position), and the radius can be the distance between the user's position (or the mobile phone position) and the external speaker, or it can be a default value (for example, 1 meter), etc., which is not limited here.
[0325] For ease of understanding, Fig. 20 As shown, an example effect diagram of a user holding a mobile phone toward an external speaker device.
[0326] In the embodiment of the present application, the number of external speaker devices is N, and N is a positive integer. The i-th external speaker device refers to a certain external speaker device among the N external speaker devices, wherein i is a positive integer and i≤N. The formulas in the embodiment of the present application are all calculated based on the i-th external speaker device as an example, and the calculations for other external speakers are similar to those for the i-th external speaker device.
[0327] The formula 1 used to calibrate the i-th external speaker device can be described as follows:
[0328] Formula 1:
[0329] Where x(t) is the energy of the test signal received by the mobile phone at time t, X(t) is the energy of the test signal played by the external speaker at time t, t is a positive number, r i is the distance between the mobile phone and the i-th external speaker (since the user holds the mobile phone, it can also be understood as the distance between the user and the i-th external speaker), r s is the normalized distance, which can be understood as a coefficient used to convert the ratio of x(t) to X(t) to distance. This coefficient can be set according to the actual situation of the external speaker. s The specific value of is not limited here.
[0330] In addition, when there are multiple external speakers, play the test signals in sequence toward the external speakers, and use formula 1 to obtain the distance.
[0331] It is understandable that the above formula is only an example. In practical applications, formula 1 can also have other forms, such as removing r s Etc., no specific limitation is made here.
[0332] Step 4: Determine the location information of the external device based on the attitude angle and distance.
[0333] In step 3, the mobile phone has recorded the attitude angle of the mobile phone toward each external speaker device and calculated the distance between the mobile phone and each external speaker device by formula 1. Of course, the mobile phone can also send the measured x(t) and X(t) to the rendering device, and the rendering device calculates the distance between the mobile phone and each external speaker device by formula 1, which is not limited here.
[0334] After the rendering device obtains the attitude angle of the mobile phone and the distance between the mobile phone and the external speaker, the attitude angle of the mobile phone and the distance between the mobile phone and the external speaker can be converted into the position information of the external speaker in the spherical coordinate system through Formula 2. The position information includes azimuth, tilt angle and distance (i.e., the distance between the sensor device and the playback device). When there are multiple external speakers in the external speaker system, determining the position information of other external speakers is similar and will not be repeated here.
[0335] The above formula 2 can be described as follows:
[0336] Formula 2:
[0337] Where, λ(t) is the azimuth angle of the i-th external speaker in the spherical coordinate system at time t, Φ(t) is the tilt angle of the i-th external speaker in the spherical coordinate system at time t, d(t) is the distance between the mobile phone and the i-th external speaker; Ω(t)[0] is the azimuth angle of the mobile phone at time t (i.e., the rotation angle of the mobile phone around the z-axis), Ω(t)[1] is the pitch angle of the mobile phone at time t (i.e., the rotation angle of the mobile phone around the x-axis); r i is the distance calculated by formula 1. sign represents positive and negative values. If Ω(t)[1] is positive, then sign is positive; if Ω(t)[1] is negative, then sign is negative. %360 is used to adjust the angle range to 0-360 degrees. For example, if the angle of Ω(t)[0] is -80 degrees, then Ω(t)[0]%360 represents -80+360=280 degrees.
[0338] It can be understood that the above formula 2 is only an example. In practical applications, formula 2 may also have other forms, which are not specifically limited here.
[0339] For example, after the user has calibrated the external speaker, the rendering device may display the following Fig.21 The interface shown in the figure displays a “calibrated icon” and the position of the calibrated playback device can be displayed in the ball view on the right.
[0340] Through the above-mentioned calibration playback device, the problem of calibrating irregular external speakers can be solved, allowing users to obtain the spatial positioning of each external speaker in subsequent operations, thereby accurately rendering the required position of a single object audio track and improving the authenticity of the spatial effect of the rendered audio track.
[0341] Step 1002: Acquire a first single object audio track based on the multimedia file.
[0342] In the embodiment of the present application, the rendering device obtains the multimedia file by directly recording the sound of the first sound-emitting object, or by sending it through other devices, for example, by receiving it through a collection device (for example, a camera, a recorder, a mobile phone, etc.). In actual applications, there are other ways to obtain multimedia files, and the specific method of obtaining the multimedia files is not limited here.
[0343] The multimedia file in the embodiment of the present application may specifically be audio information, such as stereo audio information, multi-channel audio information, etc. The multimedia file may specifically be multimodal information, such as video information, image information corresponding to the audio information, text information, etc. It can also be understood that the multimedia file may include a video track or a text track (or a bullet screen track) in addition to the audio track, which is not specifically limited here.
[0344] In addition, the multimedia file may include a first single object audio track, or include an original audio track, which is synthesized from at least two single object audio tracks, which is not specifically limited here. The original audio track may be a single audio track or multiple audio tracks, which is not specifically limited here. The original audio track may include a vocal track, an instrument track (e.g., a drum track, a piano track, a trumpet track, etc.), an airplane sound, and other audio tracks generated by a sound-emitting object (or a sound-emitting object), and the specific type of the sound-emitting object corresponding to the original audio track is not limited here.
[0345] Depending on the various situations of the original audio tracks in the multimedia file, the processing methods of this step may be different, which are described below:
[0346] First, the audio track in the multimedia file is a single object audio track.
[0347] In this case, the rendering device may directly obtain the first single object audio track from the multimedia file.
[0348] Second, the audio track in the multimedia file is a multi-object audio track.
[0349] This situation can also be understood as the original audio track in the multimedia file corresponding to multiple sound objects. Optionally, in addition to corresponding to the first sound object, the original audio track also corresponds to the second sound object. That is, the original audio track is obtained by synthesizing at least the first single object audio track and the second single object audio track, the first single object audio track corresponds to the first sound object, and the second single object audio track corresponds to the second sound object.
[0350] In this case, the rendering device may separate the first single object audio track from the original audio track, or may separate the first single object audio track and the second single object audio track from the original audio track, and the details are not limited here.
[0351] Optionally, the rendering device can Figure 5 The separation network in the illustrated embodiment separates the first single object audio track from the original audio track. In addition, the rendering device can also separate the first single object audio track and the second single object audio track from the original audio track through the separation network. The specifics are not limited here. The difference in output depends on the different ways of training the separation network. For details, please refer to Figure 5 The description in the illustrated embodiment will not be repeated here.
[0352] Optionally, after the rendering device determines the multimedia file, the rendering device can identify the sound objects of the original audio track in the multimedia file through the identification network or the separation network. For example, the sound objects contained in the original audio track include a first sound object and a second sound object. The rendering device can randomly select one of the sound objects as the first sound object, or determine the first sound object through the user's selection. Furthermore, after the rendering device determines the first sound object, it can obtain the first single object audio track through the separation network. Of course, after determining the multimedia file, the rendering device can first obtain the sound object through the identification network, and then obtain the single object audio track of the sound object through the separation network. It is also possible to directly obtain the sound objects included in the multimedia file and the single object audio track corresponding to the sound object through the identification network and / or the separation network, which is not limited here.
[0353] Exemplarily, continuing the above example, after the playback device is calibrated, the rendering device may display the following: Fig.21 The interface shown or Fig. 22 The interface shown. The user can select a multimedia file by clicking on the “select input file icon” 105, where the multimedia file is “Dream it possible.wav” for example. It can also be understood that the rendering device receives the fourth operation of the user, and the rendering device responds to the fourth operation, and determines to select “Dream it possible.wav” (i.e., the target file) as the multimedia file from at least one multimedia file stored in the storage area. Among them, the storage area can be a storage area in the rendering device, or a storage area in an external device (such as a USB flash drive, etc.), which is not limited here. After the user selects the multimedia file, the rendering device can display the following Fig.23 In the interface shown in the figure, "Select input file" can be replaced by "Dream itpossible.wav" to prompt the user that the current multimedia file is: Dream itpossible.wav. In addition, the rendering device can use Figure 4 The recognition network and / or separation network in the illustrated embodiment recognizes the sound objects in "Dream it possible.wav" and separates the single object audio track corresponding to each sound object. For example, the rendering device recognizes that the sound objects included in "Dream it possible.wav" are human, piano, violin, and guitar. Fig.23 As shown, the interface displayed by the rendering device may also include an object bar, in which icons such as a "human voice icon", a "piano icon", a "violin icon", and a "guitar icon" may be displayed for the user to select the sound object to be rendered. Optionally, a "close icon" may also be displayed in the object bar, and the user may stop selecting the sound object by clicking the "close icon".
[0354] Further, such as Fig.24 As shown, the user can click the "voice icon" 106 to determine that the audio track to be rendered is a single object audio track corresponding to the voice. It can also be understood that the rendering device recognizes "Dream it possible.wav" and obtains the rendering device display as shown in FIG. Fig.24 In the interface shown, the rendering device receives a click operation from the user, and in response to the click operation, selects the first icon (ie, the "voice icon" 106) from the interface, thereby enabling the rendering device to determine that the first single object audio track is a vocal.
[0355] Understandably, Figure 22 to Figure 24 The playback device type shown is only an external speaker as an example. Of course, the user can choose the type of playback device as headphones. The following is only a schematic explanation using the example of the playback device type selected by the user during calibration as an external speaker.
[0356] In addition, the rendering device can also copy one or several single-object tracks in the original audio track, for example Fig.25 As shown, the user can also copy the "Voice Icon" in the object bar to obtain the "Voice 2 Icon", and the single object track corresponding to Vocal 2 is the same as the single object track corresponding to the vocal. The copying method can be that the user double-clicks the "Voice Icon" or double-clicks the vocal on the ball view, and the specifics are not limited here. When the user copies and obtains the "Voice 2 Icon", it can be assumed that the user loses control of the vocal and starts to control Vocal 2. Optionally, after the user copies and obtains Vocal 2, the first sound source position of the vocal can also be displayed in the ball view. Of course, in addition to copying the sound object, the user can also delete the sound object.
[0357] Step 1003: Determine a first sound source position of a first sound emitting object based on reference information.
[0358] In an embodiment of the present application, when the original audio track of a multimedia file includes multiple sound objects, the sound source position of one sound object can be determined based on reference information, or multiple sound source positions corresponding to multiple sound objects can be determined, which is not limited here.
[0359] Exemplarily, continuing the above example, the rendering device determines that the first sound object is a human voice, and the rendering device may display Fig.26 In the interface shown, the user can click the “select rendering mode icon” 107 to select reference information, which is used to determine the first sound source position of the first sound-emitting object. Fig. 27As shown, in response to the user's first operation (i.e., the aforementioned click operation), the rendering device may display a drop-down menu, which may include an "automatic rendering option" and an "interactive rendering option." The "interactive rendering option" corresponds to the reference position information, and the "automatic rendering option" corresponds to the media information.
[0360] It can also be understood that the rendering method includes an automatic rendering method and an interactive rendering method, wherein the automatic rendering method means that the rendering device automatically obtains the rendered first single object audio track according to the media information in the multimedia file. The interactive rendering method means that the rendered first single object audio track is obtained through the interaction between the user and the rendering device. In other words, when the automatic rendering method is determined, the rendering device can obtain the rendered first single object audio track based on a preset method; or when the interactive rendering method is determined, the rendering device obtains the reference position information in response to the user's second operation; determines the first sound source position of the first sound object based on the reference position information; renders the first single object audio track based on the first sound source position to obtain the rendered first single object audio track. Among them, the preset method includes: obtaining the media information of the multimedia file; determining the first sound source position of the first sound object based on the media information; rendering the first single object audio track based on the first sound source position to obtain the rendered first single object audio track.
[0361] In addition, the sound source positions (first sound source positions and second sound source positions) in the embodiments of the present application may be fixed positions at a certain moment, or may be multiple positions (such as motion trajectories) within a certain time period, and are not specifically limited here.
[0362] There are many cases of reference information in the embodiments of this application, which are described below:
[0363] First, the reference information includes reference location information.
[0364] The reference position information in the embodiment of the present application is used to indicate the sound source position of the first sound-emitting object. The reference position information may be the first position information of the sensor device or the second position information selected by the user, etc., which is not limited here.
[0365] There are multiple cases of reference position information in the embodiments of the present application, which are described below respectively:
[0366] 1. The reference position information is the first position information of the sensor device (hereinafter referred to as the sensor).
[0367] For example, continuing the above examples, Fig. 27As shown, further, the user can click "interactive rendering option" 108 to determine the rendering mode as interactive rendering. In response to the click operation, the rendering device can display a drop-down menu, which can include "direction control options", "position control options" and "interface control options".
[0368] There are multiple situations for the first location information in the embodiment of the present application, which are described below respectively:
[0369] 1.1. The first position information includes a first attitude angle of the sensor.
[0370] Similar to the previous calibration of the external speaker device by the orientation of the sensor, the user can adjust the orientation of the handheld sensor device (such as a mobile phone) through a second operation (such as translation up, down, left, and right) to determine the first sound source position of the first single object audio track. It can also be understood that the rendering device can receive the first attitude angle of the mobile phone and use the following formula 3 to obtain the first sound source position of the first single object audio track, and the first sound source position includes the azimuth angle, the tilt angle, and the distance between the external speaker device and the mobile phone.
[0371] Furthermore, the user can also determine the second sound source position of the second single object audio track by adjusting the orientation of the handheld mobile phone. It can also be understood that the rendering device can receive the first attitude angle (including azimuth and tilt angle) of the mobile phone, and use the following formula 3 to obtain the second sound source position of the second single object audio track, and the second sound source position includes the azimuth, tilt angle and the distance between the external speaker and the mobile phone.
[0372] Optionally, if the mobile phone and the rendering device are not connected, the rendering device may send a reminder message to the user, which is used to remind the user to connect the mobile phone and the rendering device. Of course, the mobile phone and the rendering device may also be the same mobile phone, in which case no reminder message needs to be sent.
[0373] The above formula 3 can be expressed as follows:
[0374] Formula 3:
[0375] Where λ(t) is the azimuth angle of the i-th external speaker in the spherical coordinate system at time t, Φ(t) is the tilt angle of the i-th external speaker in the spherical coordinate system at time t, and d(t) is the distance between the mobile phone and the i-th external speaker at time t; Ω(t)[0] is the azimuth angle of the mobile phone at time t (i.e., the rotation angle of the mobile phone around the z-axis), Ω(t)[1] is the tilt angle of the mobile phone at time t (i.e., the rotation angle of the mobile phone around the y-axis); d(t) is the distance between the mobile phone and the i-th external speaker at time t. The distance between the two sides can be the distance calculated by formula 1 during calibration, or it can be the default value (for example, 1 meter). d(t) can be adjusted as needed. sign represents positive and negative values. If Ω(t)[1] is positive, then sign is positive; if Ω(t)[1] is negative, then sign is negative. %360 is used to adjust the angle range to 0 degrees - 360 degrees. For example, if the angle of Ω(t)[0] is -80 degrees, then Ω(t)[0]%360 represents -80+360=280 degrees.
[0376] It can be understood that the above formula 3 is only an example. In practical applications, formula 3 may also have other forms, which are not specifically limited here.
[0377] For example, continuing the above examples, Fig.28 As shown, the rendering device may display a drop-down menu, which may include "orientation control options", "position control options" and "interface control options". The user may click on "orientation control options" 109 to determine that the rendering mode is orientation control in interactive rendering. In addition, after the user selects orientation control, the rendering device may display Fig.29 In the interface shown in FIG. 1 , “Select rendering mode” can be replaced by “Direction control rendering” to prompt the user that the current rendering mode is direction control. At this time, the user can adjust the direction of the phone. When the user adjusts the direction of the phone, Fig.30 As shown, the ball view in the rendering device display interface can display a dotted line, which is used to indicate the current orientation of the phone, so that the user can intuitively see the orientation of the phone in the ball view, which makes it easier for the user to determine the position of the first sound source. After the orientation of the phone is stable (refer to the aforementioned explanation about the stability of the phone's position, which will not be repeated here), determine the first attitude angle of the current phone. Then, obtain the first sound source position using the above formula three. In addition, if the user's position has not changed compared to the calibration position, the distance between the phone and the external speaker obtained during calibration can be used as d(t) in the above formula three. Then, the user determines the first sound source position of the first sound-emitting object based on the first attitude angle, or understands it as the first sound source position of the first single-object audio track. Furthermore, the rendering device can display the following: Fig.31 The interface shown in the figure includes the first position information of the sensor (ie, the first position information of the mobile phone) corresponding to the first sound source position in the ball view of the interface.
[0378] In addition, the above example describes an example of determining the position of the first sound source. Further, the user can also determine the position of the second sound source of the second single object audio track. Fig.32 As shown, the user can click the "violin icon" 110 to determine that the second sound source is a violin. The rendering device monitors the attitude angle of the mobile phone and determines the position of the second sound source through formula three. Fig.32 As shown, the spherical view in the display interface of the rendering device can display the currently determined first sound source position of the first sound-emitting object (person) and the second sound source position of the second sound-emitting object (violin).
[0379] In this way, users can use the orientation (i.e., the first attitude angle) provided by the sensor to dynamically render the selected sound object in real time or later. At this time, the sensor is like a laser pen, and the direction of the laser is the location of the sound source. This control can give the sound object a specific spatial orientation and movement, enabling users to interact with audio and provide users with a new experience.
[0380] 1.2. The first position information includes a second attitude angle and acceleration of the sensor.
[0381] The user can control the position of the sensor device (such as a mobile phone) through the second operation to determine the position of the first sound source. It can also be understood that the rendering device can receive the second attitude angle (including azimuth, tilt angle and pitch angle) and acceleration of the mobile phone, and use the following formulas 4 and 5 to obtain the first sound source position, which includes the azimuth, tilt angle and the distance between the external device and the mobile phone. That is, the second attitude angle and acceleration of the mobile phone are first converted into the coordinates of the mobile phone in the spatial rectangular coordinate system through formula 4, and then the coordinates of the mobile phone in the spatial rectangular coordinate system are converted into the coordinates of the mobile phone in the spherical coordinate system through formula 5, that is, the first sound source position.
[0382] The above formula 4 and formula 5 can be expressed as follows:
[0383] Formula 4:
[0384]
[0385] Formula 5:
[0386] Where x(t), y(t), z(t) are the position information of the mobile phone in the rectangular coordinate system at time t, g is the gravitational acceleration, a(t) is the acceleration of the mobile phone at time t, Ω(t)[0] is the azimuth angle of the mobile phone at time t (i.e., the rotation angle of the mobile phone around the z-axis), Ω(t)[1] is the pitch angle of the mobile phone at time t (i.e., the rotation angle of the mobile phone around the x-axis), Ω(t)[2] is the tilt angle of the mobile phone at time t (i.e., the rotation angle of the mobile phone around the y-axis), λ(t) is the azimuth angle of the i-th external speaker at time t, Φ(t) is the tilt angle of the i-th external speaker at time t, and d(t) is the distance between the i-th external speaker and the mobile phone at time t.
[0387] It can be understood that the above formula 4 and formula 5 are just examples. In practical applications, formula 4 and formula 5 may also have other forms, which are not limited here.
[0388] Exemplarily, continuing the above example, if the rendering device displays Fig. 27 In the interface shown in FIG. 1 , after the user determines that the rendering mode is interactive rendering, the rendering device may display a drop-down menu, which may include "orientation control options", "position control options" and "interface control options". The user may click on "orientation control options" to Fig.33 As shown, the user can click on the "position control option" 111 to determine that the rendering method is position control in interactive rendering. In addition, after the user selects position control, the "position control rendering" can replace the "select rendering method" in the rendering device display interface to prompt the user that the current rendering method is position control. At this time, the user can adjust the position of the mobile phone, and after the position of the mobile phone is stable (refer to the aforementioned explanation about the stability of the mobile phone position, which will not be repeated here), determine the second posture angle and acceleration of the current mobile phone. Then the first sound source position is obtained through the above formula four and formula five. Then the user determines the first sound source position of the first sound-emitting object based on the second posture angle and acceleration, or understands it as the first sound source position of the first single-object audio track. Furthermore, in the process of the user adjusting the mobile phone, or after the position of the mobile phone is stable, the rendering device can display the following Fig.34 The interface shown in the figure includes the first position information of the sensor corresponding to the first sound source position (i.e., the first position information of the mobile phone) in the ball view of the interface. This allows the user to intuitively see the position of the mobile phone in the ball view, thereby facilitating the user to determine the position of the first sound source. If the interface of the rendering device displays the first position information in the ball view while the user adjusts the position of the mobile phone, it can change in real time according to the change of the position of the mobile phone.
[0389] In addition, the above example describes an example of determining the position of the first sound source. Further, the user can also determine the position of the second sound source of the second single object audio track. The method of determining the position of the second sound source is similar to the method of determining the position of the first sound source, which will not be repeated here.
[0390] It is understandable that the above two modes of the first position information are only examples. In actual applications, the first position information may have other forms, which are not specifically limited here.
[0391] In this way, by separating the single-object audio track corresponding to the sound object in the audio, and using the actual position information of the sensor as the sound source position to control the sound object and perform real-time or post-production dynamic rendering, the movement trajectory of the sound object can be simply and completely controlled by us, greatly increasing the flexibility of editing.
[0392] 2. The reference location information is the second location information selected by the user.
[0393] The rendering device can provide a spherical view for the user to select the second position information. The center of the spherical view is the user's position, and the radius of the spherical view is the distance between the user's position and the external speaker. The rendering device obtains the second position information selected by the user in the spherical view, and converts the second position information into the first sound source position. It can also be understood that the rendering device obtains the second position information of a point selected by the user in the spherical view, and converts the second position information of the point into the first sound source position. The second position information includes the two-dimensional coordinates and depth (i.e., the distance between the section and the center of the sphere) of the point selected by the user in the spherical view.
[0394] Exemplarily, continuing the above example, if the rendering device displays Fig. 27 In the interface shown in FIG. 1 , after the user determines that the rendering mode is interactive rendering, the rendering device may display a drop-down menu, which may include "orientation control options", "position control options" and "interface control options". The user may click on "orientation control options" to Fig.35 As shown, the user can click on the "interface control option" 112 to determine the rendering mode as the interface control in the interactive rendering. In addition, after the user selects the interface control, the rendering device can display the following Fig.36 In the interface shown, "Select rendering mode" can be replaced by "Interface control rendering" to prompt the user that the current rendering mode is interface control.
[0395] There are multiple situations for the second location information in the embodiment of the present application, which are described below respectively:
[0396] 2.1. The second position information is obtained according to the user's selection on the vertical slice.
[0397] The rendering device obtains the two-dimensional coordinates of the point selected by the user on the vertical section and the distance between the vertical section where the point is located and the center of the circle (hereinafter referred to as depth), and uses the following formula 6 to convert the two-dimensional coordinates and depth into the first sound source position, which includes the azimuth, tilt angle and the distance between the external speaker and the mobile phone.
[0398] Exemplarily, continuing the above example, further, if the vertical slice is jumped by default, such as Fig.37 As shown, the right side of the rendering device interface can display a spherical view, a vertical slice, and a depth control bar. The depth control bar is used to adjust the distance between the vertical slice and the center of the sphere. The user can click a point (x, y) on the horizontal plane (as shown in 114), and the corresponding spherical view in the upper right corner will display the position of the point in the spherical coordinate system. In addition, if the horizontal slice is jumped by default, the user can click the meridian in the spherical view (as shown in Fig.37 113 in the figure), the interface will display as follows Fig.37 Of course, the user can also adjust the distance between the vertical section and the center of the ball by sliding (such as Fig.37 The second position information includes the two-dimensional coordinates (x, y) and the depth r of the point. The first sound source position is obtained using formula 6.
[0399] The above formula 6 can be expressed as follows:
[0400] Formula 6:
[0401]
[0402] Where x is the horizontal coordinate of the point selected by the user on the vertical section, y is the vertical coordinate of the point selected by the user on the vertical section, r is the depth, λ is the azimuth angle of the i-th external speaker, Φ is the inclination angle of the i-th external speaker, d is the distance between the i-th external speaker and the mobile phone (it can also be understood as the distance between the i-th external speaker and the user); %360 is used to adjust the angle range to 0 degrees - 360 degrees, for example: if The angle is -60 degrees, then Represents -60+360=300 degrees.
[0403] It can be understood that the above formula 6 is only an example. In practical applications, formula 6 may also have other forms, which are not specifically limited here.
[0404] 2.2. The second position information is obtained according to the user's selection on the horizontal slice.
[0405] The rendering device obtains the two-dimensional coordinates of the point selected by the user on the horizontal section and the distance between the horizontal section where the point is located and the center of the circle (hereinafter referred to as depth), and uses the following formula seven to convert the two-dimensional coordinates and depth into the first sound source position, which includes the azimuth, inclination angle and the distance between the external speaker and the mobile phone.
[0406] Exemplarily, continuing the above example, further, if the horizontal slice is jumped by default, such as Fig.38 As shown, the right side of the rendering device interface can display a spherical view, a horizontal slice, and a depth control bar. The depth control bar is used to adjust the distance between the horizontal slice and the center of the sphere. The user can click a point (x, y) on the horizontal plane (as shown in 117), and the corresponding spherical view in the upper right corner will display the position of the point in the spherical coordinate system. In addition, if the vertical slice is jumped by default, the user can click the latitude line in the spherical view (as shown in Fig.38 116 in the figure), the interface will display as follows Fig.38 Of course, the user can also adjust the distance between the horizontal section and the center of the ball by sliding (such as Fig.38 The second position information includes the two-dimensional coordinates (x, y) and the depth r of the point. The first sound source position is obtained using Formula 7.
[0407] The above formula 7 can be expressed as follows:
[0408] Formula 7:
[0409]
[0410] Where x is the horizontal coordinate of the point selected by the user on the vertical section, y is the vertical coordinate of the point selected by the user on the vertical section, r is the depth, λ is the azimuth angle of the i-th external speaker, Φ is the inclination angle of the i-th external speaker, d is the distance between the i-th external speaker and the mobile phone (it can also be understood as the distance between the i-th external speaker and the user); %360 is used to adjust the angle range to 0 degrees - 360 degrees, for example: if The angle is -50 degrees, then Represents -50+360=310 degrees.
[0411] It can be understood that the above formula 7 is only an example. In practical applications, formula 7 may also have other forms, which are not specifically limited here.
[0412] It is understandable that the above two methods of reference position information are only examples. In actual applications, there may be other situations for reference position information, which are not specifically limited here.
[0413] In this way, the user can select the second position information (such as click, drag, slide, etc.) through the ball view to control the selected sound object and perform real-time or post-dynamic rendering, giving it a specific spatial orientation and movement, so that the user can interact with the audio and provide a new experience for the user. In addition, the sound and image of the sound object can be edited without any sensor.
[0414] Second, the reference information includes media information of the multimedia file.
[0415] The media information in the embodiment of the present application includes: at least one of: text that needs to be displayed in the multimedia file, images that need to be displayed in the multimedia file, music features of music in the multimedia file, the sound source type corresponding to the first sound object, etc., and the specific information is not limited here.
[0416] In addition, determining the first sound source position of the first sound object based on the music features of the music in the multimedia file or the sound source type corresponding to the first sound object can be understood as automatic 3D remixing. Determining the first sound source position of the first sound object based on the position text to be displayed in the multimedia file or the image to be displayed in the multimedia file can be understood as multimodal remixing. The following describes respectively:
[0417] Exemplarily, continuing the above example, the rendering device determines that the first sound object is a human voice, and the rendering device may display Fig.26 In the interface shown, the user can click the “select rendering mode icon” 107 to select a rendering mode, which is used to determine the first sound source position of the first sound-emitting object. Fig.39 As shown, in response to the click operation, the rendering device may display a drop-down menu, which may include an "automatic rendering option" and an "interactive rendering option". The "interactive rendering option" corresponds to the reference position information, and the "automatic rendering option" corresponds to the media information. Fig.39 As shown, the user can click “Automatic Rendering Option” 119 to determine that the rendering mode is automatic rendering.
[0418] 1. Automatic 3D remixing.
[0419] For example, Fig.39 As shown, the user can click on the "automatic rendering option" 119, and the rendering device can respond to the click operation and display the following Fig.40 The drop-down menu shown in FIG. 1 may include “Automatic 3D Remix Option” and “Multi-modal Remix Option”. Further, the user may click “Automatic 3D Remix Option” 120. After the user selects Automatic 3D Remix, the rendering device may display the following information: Fig.41In the interface shown, "Select rendering mode" can be replaced by "Automatic 3D remixing" to prompt the user that the current rendering mode is automatic 3D remixing.
[0420] The following describes several scenarios of automatic 3D remixing:
[0421] 1.1. Media information includes the music features of the music in the multimedia file.
[0422] The music feature in the embodiment of the present application may refer to at least one of music structure, music emotion, singing mode, etc. The music structure may include prelude, prelude vocals, verse, transition section, or chorus, etc.; the music emotion may include happiness, sadness, or fear, etc.; the singing mode may include solo, chorus, or accompaniment, etc.
[0423] After the rendering device determines the multimedia file, it can analyze the music features in the audio track (which can also be understood as audio, songs, etc.) of the multimedia file. Of course, the music features can also be identified manually or by a neural network, which is not limited here. After identifying the music features, the first sound source position corresponding to the music features can be determined according to a preset association relationship, which is the relationship between the music features and the first sound source position.
[0424] Exemplarily, continuing the above example, the rendering device determines that the first sound source position is surround, and the rendering device can display Fig.41 In the interface shown, the motion trajectory of the first sound source position is displayed in the ball view of the interface.
[0425] As mentioned above, the music structure may generally include at least one of a prelude, prelude vocals, a verse, a transition section, or a chorus. The following is a schematic illustration using the analysis of a song structure as an example.
[0426] Optionally, the vocals and instrumentals of the song can be separated, either manually or through a neural network, and no specific limitation is imposed here. After separating the vocals, the song can be segmented by judging the silent segments of the vocals and the variance of the pitch. The specific steps include: if the vocal silence is greater than a certain threshold (e.g., 2 seconds), it is considered that the segment ends, thereby dividing the large segments of the song. If there is no vocal in the first large segment, it is determined that this large segment is the instrumental prelude; if there is vocal in the first large segment, it is determined that the first large segment is the vocal prelude. And the large segment with intermediate silence is determined as the transition segment. Further, calculate the center frequency of each large segment including vocals (hereinafter referred to as the vocal large segment) through the following formula eight, and calculate the variance of the center frequencies at all moments in the vocal large segment. Sort the vocal large segments according to the variance. The vocal large segments with the variance in the top 50% are marked as the chorus, and the vocal large segments with the variance in the bottom 50% are marked as the main verse. In other words, determine the musical characteristics of the song through the magnitude of the frequency fluctuations. Then, in subsequent rendering, for different large segments, the sound source position or the movement trajectory of the sound source position can be determined through a preset association relationship, and then different large segments of the song can be rendered.
[0427] Exemplarily, if the musical characteristic is the prelude, determine that the first sound source position is to circle around above the user (or understood as surround). First, downmix the multi-channel to mono (such as averaging), and then set the entire vocal to circle around the head during the entire prelude stage. The speed at each moment is determined according to the magnitude of the vocal energy (characterized by RMS or variance). The higher the energy, the faster the rotation speed. If the musical characteristic is panic, determine that the first sound source position is from right to left suddenly. If the musical characteristic is a chorus, the vocals of the left and right channels can be expanded and widened, and the delay can be increased. Judge the number of instruments in each time period. If there is an instrument solo situation, let the instrument circle according to the energy during the solo time period.
[0428] The above formula eight can be described as follows:
[0429] Formula eight:
[0430] where f c is the center frequency of the vocal large segment per 1 second, N is the number of large segments, N is a positive integer, 0 < n < N - 1. f(n) is the frequency domain obtained by Fourier transform of the time domain waveform corresponding to the large segment, and x(n) is the energy corresponding to a certain frequency.
[0431] It can be understood that the above formula eight is just an example. In practical applications, formula eight can also have other forms, and no specific limitation is imposed here.
[0432] In this way, the position and dynamics of the extracted specific sound objects are set according to the musical characteristics of the music, making our 3D rendering more natural and better reflecting the artistry.
[0433] 1.2. The media information includes the sound source type corresponding to the first sound object.
[0434] The sound source type in the embodiment of the present application can be a person, a musical instrument, a drum sound, a piano sound, etc. In practical applications, it can be divided as needed, and is not limited here. Of course, the rendering device can identify the sound source type manually or by a neural network, and is not limited here.
[0435] After identifying the sound source type, the first sound source position corresponding to the sound source type can be determined according to a preset association relationship, which is the relationship between the sound source type and the first sound source position (similar to the aforementioned music features, which will not be repeated here).
[0436] It is understandable that the above two methods of automatic 3D remixing are just examples. In actual applications, there may be other situations for automatic 3D remixing, which are not specifically limited here.
[0437] 2. Multimodal remixing.
[0438] For example, Fig.42 As shown, the user can select a multimedia file by clicking the “select input file icon” 121, where “car.mkv” is taken as an example. It can also be understood that the rendering device receives the fourth operation of the user, and the rendering device responds to the fourth operation and selects “car.mkv” (i.e., the target file) from the storage area as the multimedia file. The storage area may be a storage area in the rendering device or a storage area in an external device (such as a USB flash drive, etc.), which is not limited here. After the user selects the multimedia file, the rendering device may display the following: Fig.43 In the interface shown in the figure, "Select input file" can be replaced by "car.mkv" to prompt the user that the current multimedia file is: car.mkv. In addition, the rendering device can use Figure 4 The identification network and / or separation network in the illustrated embodiment identifies the sound objects in "car.mkv" and separates the single object audio track corresponding to each sound object. For example, the rendering device identifies that the sound objects included in "car.mkv" are people, cars, and wind. Fig.43 as well as Fig.44 As shown, the interface displayed by the rendering device may also include an object bar, in which icons such as “human voice icon”, “car icon”, and “wind sound icon” may be displayed for the user to select the sound object to be rendered.
[0439] The following describes multiple situations of multi-modal remixing:
[0440] 2.1. Media information includes images that need to be displayed in multimedia files.
[0441] Optionally, after the rendering device obtains a multimedia file (containing an image or an audio track of a video), it can split the video into frame images (the number can be one or more), obtain third position information of the first sound-emitting object based on the frame images, and obtain the first sound source position based on the third position information. The third position information includes the two-dimensional coordinates and depth of the first sound-emitting object in the image.
[0442] Optionally, the specific steps of obtaining the first sound source position based on the third position information may include: inputting the frame image into the detection network, and obtaining the tracking frame information (x0, y0, w0, h0) corresponding to the first sound object in the frame image. Of course, the frame image and the first sound object may also be used as inputs to the detection network, and the detection network outputs the tracking frame information of the first sound object. The tracking frame information includes the two-dimensional coordinates (x0, y0) of a corner point of the tracking frame and the height h0 and width w0 of the tracking frame. The rendering device uses formula nine to calculate the tracking frame information (x0, y0, w0, h0) to obtain the coordinates of the center point of the tracking frame (x c ,y c ), and then the coordinates of the center point of the tracking frame (x c ,y c ) is input into the depth estimation network to obtain the relative depth of each point in the tracking box Then use formula 10 to calculate the relative depth of each point in the tracking frame Get the average depth z of all points in the tracking box c Based on the image size (height h1, width w1) and formula 11, the previously obtained (x c ,y c ,z c ) is normalized to [-1,1] (x norm ,y norm ,z norm ), and then obtain the first sound source position based on the playback device information and formula twelve.
[0443] The above formula nine can be described as follows:
[0444] Formula 9:
[0445]
[0446] Formula 10:
[0447] Formula 11:
[0448]
[0449] Formula 12: λ i =x norm *θ x_max ;
[0450] Φ i =y norm *θ y_max ;
[0451]
[0452] Where (x0, y0) is the 2D coordinate of a corner point of the tracking frame (e.g., the lower left corner point), h0 is the height of the tracking frame, w0 is the width of the tracking frame; h1 is the height of the image, w1 is the width of the image; is the relative depth of each point in the tracking box, z c is the average depth of all points in the tracking box; θ x_max is the maximum horizontal angle of the playback device (if the playback device is N external playback devices, the playback device information of the N external playback devices is the same), θ y_max is the maximum vertical angle of the playback device, d y_max is the maximum depth of the playback device; i is the azimuth of the i-th external speaker, Φ i is the tilt angle of the i-th external speaker, r i is the distance between the i-th external speaker and the user.
[0453] It can be understood that the above formula 9 to formula 12 are only examples. In practical applications, formula 9 to formula 12 may also have other forms, which are not specifically limited here.
[0454] For example, Fig.43 As shown, the user can click on the “multi-modal remix option” 122, and the rendering device can respond to the click operation by displaying the following Fig.43 The right side of the interface includes a frame (for example, the first frame) of "car.mkv" and playback device information, and the playback device information includes the maximum horizontal angle, the maximum vertical angle, and the maximum depth. If the playback device is a headset, the user can enter the playback device information. If the playback device is an external speaker, the user can enter the playback device information or directly use the calibration information obtained during the calibration phase as the playback device information. The specific information is not limited here. In addition, after the user selects the multimodal remix option, the rendering device can display the following: Fig.44 In the interface shown, "Select rendering mode" can be replaced by "Multimodal remix" to prompt the user that the current rendering mode is multimodal remix.
[0455] In the case where the media information includes an image to be displayed in a multimedia file, there are multiple ways to determine the first sound emitting object in the image, which are described below:
[0456] (1) The first sound object is determined by the user clicking in the object column.
[0457] Optionally, the rendering device may determine the first sound-emitting object based on a click of the user in the object bar.
[0458] For example, Fig.44 As shown, the user can click the "car icon" 123 to determine that the sound object to be rendered is a car. The rendering device displays the tracking frame of the car in the image "car.mkv" on the right, and then obtains the third position information, and converts the third position information into the first sound source position through formulas 9 to 12. In addition, the interface also includes the coordinates of the corner point (x0, y0) of the lower left corner of the tracking frame and the coordinates of the center point (x c ,y c ). For example, the maximum horizontal angle in the external speaker information is 120 degrees, the maximum vertical angle is 60 degrees, and the maximum depth is 10 (the unit can be meter, decimeter, etc., which is not limited here).
[0459] (2) The first sound object is determined by the user clicking on the image.
[0460] Optionally, the rendering device may use the sound emission object determined by the user's third operation (eg, click) in the image as the first sound emission object.
[0461] For example, Fig.45 As shown, the user can determine the first sound-generating object by clicking on the sound-generating object in the image (as shown in 124 ).
[0462] (3) The first sound object is determined according to the default settings.
[0463] Optionally, the rendering device can identify the sound-emitting object through the audio track corresponding to the image, can track the default sound-emitting object or all sound-emitting objects in the image, and determine the third position information. The third position information includes the two-dimensional coordinates of the sound-emitting object in the image and the depth of the sound-emitting object in the image.
[0464] Exemplarily, the rendering device may select “joint” by default in the object column, that is, all sound-emitting objects in the image are tracked, and the third position information of the first sound-emitting object is determined respectively.
[0465] It can be understood that the above-mentioned methods of determining the first sound emitting object in an image are only examples. In actual applications, there may be other methods of determining the first sound emitting object in an image, which are not specifically limited here.
[0466] In this method, the coordinates of the sound object and the single-object audio track are extracted by combining the multi-modality of audio, video and images, and then a 3D immersive feeling is rendered in a headset or external speaker environment, so that the sound moves with the picture, allowing users to obtain the best sound experience. In addition, the technology of tracking and rendering the object audio in the entire video after selecting the sound object can also be applied in professional mixing post-production to improve the work efficiency of the mixer. By separating the single-object audio track of the audio in the video, and by analyzing and tracking the sound object in the video image, the motion information of the sound object is obtained, and the selected sound object is dynamically rendered in real time or in post-production. The video picture is matched with the direction of the audio source to improve the user experience.
[0467] 2.2. Media information includes the location text that needs to be displayed in the multimedia file.
[0468] In this case, the rendering device may determine the first sound source position based on the position text to be displayed in the multimedia file, where the position text is used to indicate the first sound source position.
[0469] Optionally, the position text can be understood as text with meanings such as position and orientation, such as: the wind blows north, front, back, left, right, etc., which is not limited here. Of course, the position text can specifically be lyrics, subtitles, advertising slogans, etc., which is not limited here.
[0470] Optionally, the semantics of the displayed location text can be identified based on reinforcement learning or a neural network, and then the first sound source position can be determined based on the semantics.
[0471] In this method, by identifying the location text related to the location, a 3D immersive feeling is rendered in a headphone or external speaker environment, achieving a sense of space corresponding to the location text, allowing users to obtain the best sound experience.
[0472] It is understandable that the above two modes of media information are just examples. In actual applications, media information may also have other forms, which are not specifically limited here.
[0473] In addition, in step 1003, a variety of situations are described for how to determine the first sound source position based on the reference information. In practical applications, the first sound source position can also be determined by a combination of methods. For example, after determining the first sound source position by the sensor orientation, the motion trajectory of the first sound source position is determined by the music feature. For example, Fig.46 As shown, the rendering device has determined the sound source position of the human voice based on the first posture angle of the sensor as Fig.46 As shown on the right side of the interface. Further, the user can determine the movement trajectory of the human voice by clicking the "circle option" 125 in the menu on the right side of the "human voice icon". It can also be understood as first using the sensor direction to determine the first sound source position at a certain moment, and then using the music features or preset rules to determine that the movement trajectory of the first sound source position is a circle. Correspondingly, as Fig.46 As shown, the interface of the rendering device can display the motion trajectory of the object.
[0474] Optionally, in the process of determining the first sound source position, the user may control the distance in the first sound source position by controlling the volume button of the mobile phone or clicking, dragging, sliding, etc. on the ball view.
[0475] Step 1004: spatially render the first single-object audio track based on the first sound source position.
[0476] After determining the position of the first sound source, the rendering device may perform spatial rendering on the first single-object audio track to obtain the rendered first single-object audio track.
[0477] Optionally, the rendering device spatially renders the first single object audio track based on the first sound source position to obtain the rendered first single object audio track. Of course, the rendering device may also spatially render the first single object audio track based on the first sound source position, and render the second single object audio track based on the second sound source position to obtain the rendered first single object audio track and the second single object audio track.
[0478] Optionally, in the case of spatial rendering of multiple single-object audio tracks corresponding to multiple sound-emitting objects in the original audio track, the method for determining the sound source position in the embodiment of the present application can apply a combination of multiple methods in step 1003, which is not limited here.
[0479] For example, Fig.32 As shown, the first sound object is a person, the second sound object is a violin, the first sound source position of the first single object audio track corresponding to the first sound object can be determined by one of the interactive rendering methods, and the second sound source position of the second single object audio track corresponding to the second sound object can be determined by one of the automatic rendering methods. The specific determination methods for the first sound source position and the second sound source position can be any two of the above-mentioned step 1003. Of course, the specific determination methods for the first sound source position and the second sound source position can also be the same, which is not limited here.
[0480] In addition, in the above figure containing the ball view, the ball view may also include a volume bar, and the user may control the volume of the first single object audio track by sliding the volume bar with a finger, dragging the mouse, or using a mouse wheel, thereby improving the real-time performance of the rendered audio track. Fig.47As shown, the user can adjust the volume bar 126 to adjust the volume of the single object track corresponding to the guitar.
[0481] The rendering method of this step may be different depending on the type of playback device. It can also be understood as the method used by the rendering device to spatially render the original audio track or the first single-object audio track based on the first sound source position and the type of playback device. It varies depending on the type of playback device, and is described below:
[0482] The first one is that the playback device type is headphones.
[0483] In this case, after the rendering device determines the first sound source position, it can render the audio track based on Formula 13 and the HRTF filter coefficient table. The audio track can be the first single object audio track, the second single object audio track, or the first single object audio track and the second single object audio track, which is not limited here. Among them, the HRTF filter coefficient table is used to represent the correlation between the sound source position and the coefficient, which can also be understood as one sound source position corresponds to one HRTF filter coefficient.
[0484] The above formula 13 can be expressed as follows:
[0485] Formula 13:
[0486] in, is the first single-object audio track after rendering, S is the sound object of the multimedia file and includes the first sound object, i indicates the left channel or the right channel, a s (t) is the adjustment coefficient of the first sound object at time t, h i,s (t) is the head-related transfer function HRTF filter coefficient of the left channel or the right channel corresponding to the first sound object at time t, wherein the HRTF filter coefficient of the left channel corresponding to the first sound object at time t is generally different from the HRTF filter coefficient of the right channel corresponding to the first sound object at time t, and the HRTF filter coefficient is related to the position of the first sound source, o s (t) is the first single object audio track at time t, and τ is the integral term.
[0487] It can be understood that the above formula 13 is only an example. In practical applications, formula 13 may also have other forms, which are not specifically limited here.
[0488] The second type is that the playback device type is an external speaker.
[0489] In this case, after determining the first sound source position, the rendering device may render the audio track based on Formula 14. The audio track may be a first single object audio track, a second single object audio track, or a first single object audio track and a second single object audio track, which is not limited here.
[0490] The above formula 14 can be expressed as follows:
[0491] Formula 14:
[0492]
[0493] The number of external speakers can be N. is the first single-object audio track after rendering, i indicates the i-th channel in the multi-channel, S is the sound object of the multimedia file and includes the first sound object, a s (t) is the adjustment coefficient of the first sound object at time t, g s (t) represents the translation coefficient of the first sound object at time t, o s (t) is the first single object track at time t, λ i The azimuth angle obtained by calibrating the ith external device for the calibrator (such as the aforementioned sensor device), Φ i The tilt angle obtained by the calibrator to calibrate the i-th external device, r i is the distance between the i-th external speaker and the calibrator, N is a positive integer, i is a positive integer and i≤N, and the position of the first sound source is within the tetrahedron formed by the N external speakers.
[0494] In addition, for the spatial rendering of the original audio track, the single object audio track corresponding to a certain sound object in the original audio track can be rendered and replaced, for example, by S1 in the above formula. Alternatively, the single object audio track corresponding to a certain sound object in the original audio track can be copied and added, for example, by S2 in the above formula. Of course, it can also be a combination of S1 and S2.
[0495] It can be understood that the above formula 14 is only an example. In practical applications, formula 14 may also have other forms, which are not specifically limited here.
[0496] For easier understanding of N, see Fig.48 , this figure is a schematic diagram of the architecture of the external speaker in the spherical coordinate system, where when the sound source position of the sound object is within the tetrahedron formed by four external speaker devices, N=4; when the sound source position of the sound object is on the surface of the area formed by three external speaker devices, N=3; if it is on the line connecting two external speaker devices, N=2; when it is directly pointing to one external speaker device, N=1. Fig.48 Since the point in is on the line between external speaker device 1 and external speaker device 2, Fig.48 N is shown as 2.
[0497] Step 1005: Obtain a target audio track based on the rendered first single object audio track.
[0498] Depending on the type of playback device, the target audio track obtained in this step may be different. It can also be understood as the method used by the rendering device to obtain the target audio track. It varies depending on the type of playback device. The following describes them separately:
[0499] The first one is that the playback device type is headphones.
[0500] In this case, after the rendering device obtains the rendered first single object audio track and / or the second single object audio track, it can obtain the target audio track based on Formula 15 and the rendered audio track. The audio track can be the first single object audio track, the second single object audio track, or the first single object audio track and the second single object audio track, which is not limited here.
[0501] The above formula 15 can be expressed as follows:
[0502] Formula 15:
[0503] Among them, i indicates the left channel or the right channel, is the target track at time t, X i (t) is the original audio track at time t, is the first single object track that has not been rendered at time t, is the first single object track after rendering, a s (t) is the adjustment coefficient of the first sound object at time t, h i,s (t) is the head-related transfer function HRTF filter coefficient of the left channel or right channel corresponding to the first sound source at time t. The HRTF filter coefficient is related to the position of the first sound source. s (t) is the first single-object audio track at time t, τ is the integral term, S1 is the sound object that needs to be replaced in the original audio track, if the first sound object is to replace the sound object in the original audio track, then S1 is an empty set; S2 is the sound object added to the target audio track compared to the original audio track, if the first sound object is to copy the sound object in the original audio track, then S2 is an empty set; S1 and / or S2 are the sound objects of the multimedia file and include the first sound object. If S2 is an empty set, it can be understood that the spatial rendering of the audio track is to replace the sound object, and the single-object audio track corresponding to the sound object is spatially rendered, and the rendered single-object audio track is used to replace the initial single-object audio track in the multimedia file. In other words, compared with the multimedia file, the target audio track does not have a single-object audio track corresponding to multiple sound objects, but replaces the initial single-object audio track in the multimedia file with the rendered single-object audio track.
[0504] It can be understood that the above formula 15 is only an example. In practical applications, formula 15 may also have other forms, which are not limited here.
[0505] The second type is that the playback device type is an external speaker.
[0506] In this case, after the rendering device obtains the rendered first single object audio track and / or the second single object audio track, it can obtain the target audio track based on Formula 16 and the rendered audio track. The audio track can be the first single object audio track, the second single object audio track, or the first single object audio track and the second single object audio track, which is not limited here.
[0507] The above formula 16 can be expressed as follows:
[0508] Formula 16:
[0509]
[0510] The number of external speakers may be N, i indicates a channel among multiple channels, is the target track at time t, X i (t) is the original audio track at time t, is the first single audio track that has not been rendered at time t, is the first single object track after rendering, a s (t) is the adjustment coefficient of the first sound object at time t, g s (t) represents the translation coefficient of the first sound object at time t, g i,s (t) represents g s The i-th row in (t), o s (t) is the first single-object audio track at time t, S1 is the sound object that needs to be replaced in the original audio track. If the first sound object replaces the sound object in the original audio track, S1 is an empty set; S2 is the sound object added to the target audio track compared to the original audio track. If the first sound object is a copy of the sound object in the original audio track, S2 is an empty set; S1 and / or S2 are the sound objects of the multimedia file and include the first sound object, λ i The azimuth angle obtained by the calibrator to calibrate the i-th external device, Φ i The tilt angle obtained by the calibrator to calibrate the i-th external device, r i is the distance between the i-th external speaker and the calibrator, N is a positive integer, i is a positive integer and i≤N, and the position of the first sound source is within the tetrahedron formed by the N external speakers.
[0511] It can be understood that the above formula 16 is only an example. In practical applications, formula 16 can also have other forms, which are not specifically limited here.
[0512] Of course, a new multimedia file may also be generated according to the multimedia file and the target audio track, which is not specifically limited here.
[0513] In addition, after rendering the first single object audio track, the user can upload the settings of the sound source position during the rendering process to the aforementioned Figure 8 In the corresponding database module, it is convenient for other users to use this setting method to render other audio tracks. Of course, users can also download the setting method from the database module and modify it to facilitate the spatial rendering of the audio track. In this method, modifications to the rendering rules and sharing between different users are added. On the one hand, repeated object recognition and tracking for the same file can be avoided in multimodal mode, reducing the overhead on the terminal side; on the other hand, the user's free creation in the interactive mode can be shared with other users, further enhancing the interactivity of the application.
[0514] For example, Fig.49 As shown in , the user can choose to synchronize the rendering rule files stored in the local database to other devices of the user. Fig.50 As shown, the user can choose to upload the rendering rule file stored in the local database to the cloud for sharing with other users, and other users can choose to download the corresponding rendering rule file from the cloud database to the client side.
[0515] The metadata files stored in the database are mainly used to render the sound objects separated by the system or the objects specified by the user in the automatic mode, or to render the sound objects specified by the user that need to be automatically rendered according to the stored rendering rules in the mixed mode. The metadata files stored in the database can be pre-made by the system, as shown in Table 1.
[0516] Table 1
[0517]
[0518] Among them, the serial numbers 1 and 2 in Table 1 can also be created by the user when using the interactive mode of the present invention, such as serial numbers 3-6 in Table 1; they can also be stored after the system automatically identifies the motion trajectory of the specified sound-emitting object in the video screen in the multimodal mode, such as serial number 7 in Table 1. The metadata file can be strongly related to the audio content in the multimedia file or the multimodal file content: for example, serial number 3 in Table 1 is the metadata file corresponding to the audio file A1, and serial number 4 is the metadata file corresponding to the audio file A2; it can also be decoupled from the multimedia file: the user performs interactive operations on the object X of the audio file A in the interactive mode, and saves the motion trajectory of the object X as the corresponding metadata file (for example, serial number 5 in Table 1 is the free circling and rising state). When using automatic rendering next time, the user can select the metadata file of the free circling and rising state from the database module to render the object Y of the audio file B.
[0519] In one possible implementation, the rendering method provided in the embodiment of the present application includes steps 1001 to 1005. In another possible implementation, the rendering method provided in the embodiment of the present application includes steps 1002 to 1005. In another possible implementation, the rendering method provided in the embodiment of the present application includes steps 1001 to 1004. In another possible implementation, the rendering method provided in the embodiment of the present application includes steps 1002 to 1004. In addition, in the embodiment of the present application Fig.10 The steps shown do not limit the time sequence relationship. For example, step 1001 in the above method can also be performed after step 1002, that is, after obtaining the audio track, before calibrating the playback device.
[0520] In the embodiment of the present application, the user can control the sound image, quantity, and volume of a specific sound object of the audio through the mobile phone sensor, control the sound image, quantity, and volume of a specific sound object by dragging it through the mobile phone interface, and spatially render the specific sound object in the music through automated rules to enhance the spatiality, automatically render the sound source position through multimodal recognition, and render the single sound object through a method to provide a sound experience that is completely different from the traditional music and movie interaction mode. It provides a brand-new interactive mode for music appreciation. Among them, the automated 3D re-production enhances the spatial sense of two-channel music and brings listening to a new level. In addition, the separate addition of the interactive method we have designed enhances the user's ability to edit audio, which can be applied to the production of sound objects in music and film and television works, and simply edits the motion information of specific sound objects. It also increases the user's controllability and playability of music, allowing users to experience the fun of making their own audio and the ability to control specific sound objects.
[0521] In addition to the above-mentioned training method and rendering method, this application also provides two specific application scenarios for applying the above-mentioned rendering method, which are described below respectively:
[0522] The first one is the “Sound Hunter” game scenario.
[0523] This scenario can also be understood as the user pointing to the sound source location, and judging whether the user's pointing is consistent with the actual sound source location, scoring the user's operation, and improving the user's entertainment experience.
[0524] Exemplarily, continuing the above example, after the user determines the multimedia file and renders the single object audio track, as shown in FIG. Fig.51 As shown, the user can click the "Hunter Icon" 126 to enter the Hunter game scene, and the rendering device can display the following Fig.51 In the interface shown, the user can click the play button at the bottom of the interface to confirm the start of the game, and the playback device will play at least one single object track in a certain order and at any position. When the playback device plays the single object track of the piano, the user determines the location of the sound source by hearing, and points the mobile phone at the location of the sound source determined by the user. If the location pointed by the user's mobile phone is consistent with the actual sound source location of the single object track of the piano (or the error is within a certain range), the rendering device can display the following Fig.51 The prompt on the right side of the interface is "Hit the first instrument, it took 5.45 seconds". In addition, after the user points to the correct position of a sound source, the corresponding sound source in the object bar can change from red to green. Of course, if the user fails to correctly point to the sound source within a certain period of time, it can be displayed as a failure. After the first single object track is played, the preset time period (such as Fig.54 After a time interval T), the next single-object audio track is played to continue the game, such as Fig.52 as well as Fig.53 In addition, after the user points to the wrong position of a sound-generating object, the corresponding sound-generating object in the object bar can remain red. Fig.54 As shown in the figure, after the user presses the pause button at the bottom of the interface or the single object track is played, the game is confirmed to be over. Further, after the game is over, if the user's pointing is correct, the rendering device can display the following Fig.53 The interface shown.
[0525] In this scenario, through real-time information interaction between the user and the playback system, the position of the object audio is rendered in the playback system in real time, and the game is designed to enable users to obtain the ultimate "listening and positioning" experience. It can be applied to home entertainment, AR, VR games, etc. Compared with the existing technology of "listening and positioning" which is only for a whole song, this application provides a game that separates the human voice and instruments from a song.
[0526] The second type is the multi-person interactive scenario.
[0527] This scenario can be understood as multiple users controlling the sound source position of a specific sound source, thereby rendering the sound track by multiple people, increasing entertainment and communication between multiple users. For example, this interactive scenario can be an online band or an online anchor controlling a symphony.
[0528] For example, if the multimedia file is a piece of music played by multiple instruments, user A can select the multi-person interaction mode and invite user B to jointly create the music. Each user can select a different instrument as the interactive sound object to control, and render the instruments according to the rendering trajectory given by the user corresponding to the sound object, and then remix the instruments. The remixed audio file is then sent to each participating user. Different users can select different interaction modes, which are not limited here. For example, Fig.55 As shown, user A selects the interactive mode to interactively control the position of object A by changing the orientation of the mobile phone he uses, and user B selects the interactive mode to interactively control the position of object B by changing the orientation of the mobile phone he uses. Fig.56 As shown, the system (rendering device or cloud server) can send the remixed audio file to each user participating in the multi-person interactive application, and the position of object A and the position of object B in the audio file correspond to the manipulation of user A and user B respectively.
[0529] An example of the specific interaction process between user A and user B in the above: user A chooses to input a multimedia file, the system identifies the object information in the input file, and feeds back to user A through the UI interface. User A selects a mode. If user A selects a multi-person interaction mode, user A sends a multi-person interaction request to the system and sends the information of the designated invited object to the system. In response to the request, the system sends an interaction request to user B selected by user A. If user B accepts the request, it sends an acceptance instruction to the system to join the multi-person interaction application created by user A. User A and user B respectively select the sound object to be operated, and use the above rendering mode to manipulate the selected sound object, and the corresponding rendering rule file. The system separates the single object audio track through the separation network, and renders the separated single object audio track separately according to the rendering track provided by the user corresponding to the sound object, and then remixes the rendered single object audio track to obtain the target audio track, and then sends the target audio track to each participating user.
[0530] In addition, the multi-person interaction mode can be the real-time online multi-person interaction described in the above example, or it can be the offline multi-person interaction. For example, the multimedia file selected by user A is duet music, including singer A and singer B. Fig.57As shown, user A can select the interactive mode to manipulate the rendering effect of singer A, and share the re-rendered target audio track with user B; user B can use the received target audio track shared by user A as an input file to manipulate the rendering effect of singer B. The interactive modes selected by different users can be the same or different, and are not specifically limited here.
[0531] It is understandable that the above application scenarios are only examples. In actual applications, there may be other application scenarios, which are not specifically limited here.
[0532] In this scenario, real-time and non-real-time interactive rendering control involving multiple people is supported. Users can invite other users to jointly re-render different sound objects of multimedia files, enhancing the interactive experience and the fun of the application. The above method is used to realize the collaborative sound and image control of different objects by multiple people, thereby realizing the rendering of multimedia files by multiple people.
[0533] The rendering method in the embodiment of the present application is described above. The rendering device in the embodiment of the present application is described below. Fig.58 , an embodiment of the rendering device in the embodiment of the present application includes:
[0534] An acquiring unit 5801 is configured to acquire a first single object audio track based on a multimedia file, where the first single object audio track corresponds to a first sound object;
[0535] A determining unit 5802 is configured to determine a first sound source position of a first sound emitting object based on reference information, where the reference information includes reference position information and / or media information of a multimedia file, and the reference position information is used to indicate a first sound source position;
[0536] The rendering unit 5803 is configured to perform spatial rendering on the first single object audio track based on the first sound source position to obtain a rendered first single object audio track.
[0537] In this embodiment, the operations performed by each unit in the rendering device are the same as those described above. Figures 5 to 11 The description in the illustrated embodiment is similar and will not be repeated here.
[0538] In this embodiment, the acquisition unit 5801 acquires a first single object audio track based on a multimedia file, and the first single object audio track corresponds to a first sound object; the determination unit 5802 determines a first sound source position of the first sound object based on reference information, and the rendering unit 5803 performs spatial rendering on the first single object audio track based on the first sound source position to obtain a rendered first single object audio track. The stereoscopic spatial sense of the first single object audio track corresponding to the first sound object in the multimedia file can be enhanced, providing an immersive stereo sound effect for the user.
[0539] See also Fig.59 Another embodiment of the rendering device in the embodiment of the present application includes:
[0540] An acquiring unit 5901 is configured to acquire a first single object audio track based on a multimedia file, where the first single object audio track corresponds to a first sound object;
[0541] A determining unit 5902 is configured to determine a first sound source position of a first sound emitting object based on reference information, where the reference information includes reference position information and / or media information of a multimedia file, and the reference position information is used to indicate a first sound source position;
[0542] The rendering unit 5903 is configured to perform spatial rendering on the first single object audio track based on the first sound source position to obtain a rendered first single object audio track.
[0543] The rendering device in this embodiment also includes:
[0544] The providing unit 5904 is used to provide a ball view for the user to select, the center of the ball view is the user's location, and the radius of the ball view is the distance between the user's location and the playback device;
[0545] The sending unit 5905 is used to send the target audio track to the playback device, and the playback device is used to play the target audio track.
[0546] In this embodiment, the operations performed by each unit in the rendering device are the same as those described above. Figures 5 to 11 The description in the illustrated embodiment is similar and will not be repeated here.
[0547] In this embodiment, the acquisition unit 5901 acquires the first single object audio track based on the multimedia file, and the first single object audio track corresponds to the first sound object; the determination unit 5902 determines the first sound source position of the first sound object based on the reference information, and the rendering unit 5903 performs spatial rendering on the first single object audio track based on the first sound source position to obtain the rendered first single object audio track. The stereoscopic spatial sense of the first single object audio track corresponding to the first sound object in the multimedia file can be enhanced to provide users with an immersive stereo sound effect. In addition, a sound effect experience completely different from the traditional music and movie interaction mode is provided. A new interactive mode is provided for music appreciation. Among them, the automated 3D re-production enhances the spatial sense of two-channel music and brings listening to a new level. In addition, the interaction method designed by us is added separately to enhance the user's ability to edit audio, which can be applied to the production of sound objects in music and film and television works, and simply edit the motion information of specific sound objects. It also increases the user's controllability and playability of music, allowing users to experience the fun of making their own audio and the ability to control specific sound objects.
[0548] See also Fig.60, an embodiment of the rendering device in the embodiment of the present application includes:
[0549] An acquisition unit 6001 is used to acquire multimedia files;
[0550] The acquisition unit 6001 is further configured to acquire a first single object audio track based on the multimedia file, where the first single object audio track corresponds to the first sound object;
[0551] Display unit 6002, used to display a user interface, the user interface includes a rendering mode option;
[0552] A determining unit 6003 is used to determine an automatic rendering mode or an interactive rendering mode from rendering mode options in response to a first operation of the user on the user interface;
[0553] The acquiring unit 6001 is further configured to acquire the rendered first single object audio track based on a preset method when the determining unit determines that the rendering method is automatic; or
[0554] The acquisition unit 6001 is also used to obtain reference position information in response to a second operation of the user when the determination unit determines the interactive rendering mode; determine the first sound source position of the first sound-emitting object based on the reference position information; and render the first single-object audio track based on the first sound source position to obtain the rendered first single-object audio track.
[0555] In this embodiment, the operations performed by each unit in the rendering device are the same as those described above. Figures 5 to 11 The description in the illustrated embodiment is similar and will not be repeated here.
[0556] In this embodiment, the determination unit 6003 determines the automatic rendering mode or the interactive rendering mode from the rendering mode options according to the first operation of the user. On the one hand, the acquisition unit 6001 can automatically acquire the rendered first single object audio track based on the first operation of the user. On the other hand, the spatial rendering of the audio track corresponding to the first sound object in the multimedia file can be realized through the interaction between the rendering device and the user, providing the user with an immersive stereo sound effect.
[0557] See also Fig.61 , a schematic diagram of the structure of another rendering device provided by the present application. The rendering device may include a processor 6101, a memory 6102, and a communication interface 6103. The processor 6101, the memory 6102, and the communication interface 6103 are interconnected via a line. The memory 6102 stores program instructions and data.
[0558] The memory 6102 stores the aforementioned Figures 5 to 11 In the corresponding implementation manner shown, the steps executed by the rendering device correspond to program instructions and data.
[0559] Processor 6101, used to execute the above Figures 5 to 11 The steps performed by the rendering device in any of the illustrated embodiments.
[0560] The communication interface 6103 can be used to receive and send data and to perform the aforementioned Figures 5 to 11 The steps related to obtaining, sending and receiving in any of the embodiments shown.
[0561] In one implementation, the rendering device may include a Fig.61 More or fewer components are merely illustrative in this application and are not limiting.
[0562] The present application also provides a sensor device, such as Fig.62 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The sensor device can be any terminal device including a mobile phone, a tablet computer, etc., taking the sensor as a mobile phone as an example:
[0563] Fig.62 The block diagram shows a partial structure of a sensor device - a mobile phone provided in an embodiment of the present application. Fig.62 The mobile phone includes: a radio frequency (RF) circuit 6210, a memory 6220, an input unit 6230, a display unit 6240, a sensor 6250, an audio circuit 6260, a wireless fidelity (WiFi) module 6270, a processor 6280, and a power supply 6290. Those skilled in the art will appreciate that Fig.62 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0564] Combine the following Fig.62 A detailed introduction to the various components of the mobile phone:
[0565] The RF circuit 6210 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 6280 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 6210 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 6210 can also communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to the global system of mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc.
[0566] The memory 6220 can be used to store software programs and modules. The processor 6280 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 6220. The memory 6220 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 6220 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0567] The input unit 6230 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 6230 may include a touch panel 6231 and other input devices 6232. The touch panel 6231, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 6231 or near the touch panel 6231 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 6231 may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 6280, and can receive and execute commands sent by the processor 6280. In addition, the touch panel 6231 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 6231, the input unit 6230 may also include other input devices 6232. Specifically, the other input devices 6232 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.
[0568] The display unit 6240 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 6240 may include a display panel 6241. Optionally, the display panel 6241 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 6231 may cover the display panel 6241. When the touch panel 6231 detects a touch operation on or near it, it is transmitted to the processor 6280 to determine the type of touch event. Subsequently, the processor 6280 provides a corresponding visual output on the display panel 6241 according to the type of touch event. Although in Fig.62 In the embodiment, the touch panel 6231 and the display panel 6241 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 6231 and the display panel 6241 can be integrated to realize the input and output functions of the mobile phone.
[0569] The mobile phone may also include at least one sensor 6250, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 6241 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 6241 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared, IMU, SLAM sensors, etc., they will not be repeated here.
[0570] The audio circuit 6260, the speaker 6262, and the microphone 6262 can provide an audio interface between the user and the mobile phone. The audio circuit 6260 can transmit the received audio data to the speaker 6262 after converting the electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 6262 converts the collected sound signal into an electrical signal, which is received by the audio circuit 6260 and converted into audio data, and then the audio data is output to the processor 6280 for processing, and then sent to another mobile phone through the RF circuit 6210, or the audio data is output to the memory 6220 for further processing.
[0571] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 6270. It provides users with wireless broadband Internet access. Fig.62 A WiFi module 6270 is shown, but it is understandable that it is not an essential component of the mobile phone.
[0572] The processor 6280 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 6220, and calling data stored in the memory 6220, it executes various functions of the mobile phone and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 6280 may include one or more processing units; preferably, the processor 6280 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 6280.
[0573] The mobile phone also includes a power supply 6290 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 6280 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption management.
[0574] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.
[0575] In the embodiment of the present application, the processor 6280 included in the mobile phone can execute the aforementioned Figures 5 to 11 The functions in the illustrated embodiment will not be described in detail here.
[0576] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0577] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0578] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0579] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0580] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.
[0581] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
Claims
1. A rendering method, characterized in that: include: Acquire a first single object audio track based on the multimedia file, the first single object audio track corresponding to the first sound object; Determine a first sound source position of the first sound-emitting object based on reference information, wherein the reference information includes reference position information and media information of the multimedia file, and the reference position information is used to indicate the first sound source position; The first single-object audio track is spatially rendered based on the first sound source position to obtain a rendered first single-object audio track.
2. The method according to claim 1, characterized in that The media information includes: text to be displayed in the multimedia file, images to be displayed in the multimedia file, music features of music to be played in the multimedia file, and at least one of a sound source type corresponding to the first sound object.
3. The method according to claim 1 or 2, characterized in that: The reference position information includes first position information of the sensor or second position information selected by a user.
4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Determine a type of a playback device, the playback device being used to play a target audio track, the target audio track being obtained according to the rendered first single object audio track; The spatially rendering the first single-object audio track based on the first sound source position includes: The first single object audio track is spatially rendered based on the first sound source position and the type of the playback device.
5. The method according to claim 2, characterized in that: When the media information includes the image and the image includes the first sound emitting object, determining the first sound source position of the first sound emitting object based on the reference information includes: Determine third position information of the first sound emitting object in the image, the third position information including a two-dimensional coordinate and a depth of the first sound emitting object in the image; The first sound source position is acquired based on the third position information.
6. The method according to claim 2 or 5, characterized in that: When the media information includes music features of the music to be played in the multimedia file, determining the first sound source position of the first sound object based on the reference information includes: The first sound source position is determined based on the association relationship and the music feature, where the association relationship is used to represent the association between the music feature and the first sound source position.
7. The method according to claim 2, characterized in that When the media information includes text to be displayed in the multimedia file and the text includes position text related to a position, determining the first sound source position of the first sound emitting object based on the reference information includes: identifying the location text; The first sound source position is determined based on the position text.
8. The method according to claim 3, characterized in that When the reference position information includes the first position information, before determining the first sound source position of the first sound emitting object based on the reference information, the method further includes: Acquire the first position information, where the first position information includes a first attitude angle of the sensor and a distance between the sensor and a playback device; The determining the first sound source position of the first sound emitting object based on the reference information includes: The first position information is converted into the first sound source position.
9. The method according to claim 3, characterized in that: When the reference position information includes the first position information, before determining the first sound source position of the first sound emitting object based on the reference information, the method further includes: Acquire the first position information, where the first position information includes a second attitude angle of the sensor and an acceleration of the sensor; The determining the first sound source position of the first sound emitting object based on the reference information includes: The first position information is converted into the first sound source position.
10. The method according to claim 3, characterized in that When the reference position information includes the second position information, before determining the first sound source position of the first sound emitting object based on the reference information, the method further includes: A spherical view is provided for the user to select, the center of the spherical view is the position of the user, and the radius of the spherical view is the distance between the position of the user and the playback device; Acquire the second position information selected by the user in the ball view; The determining the first sound source position of the first sound emitting object based on the reference information includes: The second position information is converted into the first sound source position.
11. The method according to any one of claims 1 to 10, characterized in that: The step of obtaining a first single object audio track based on the multimedia file includes: The first single-object audio track is separated from the original audio track in the multimedia file, the original audio track is synthesized by at least the first single-object audio track and a second single-object audio track, and the second single-object audio track corresponds to a second sound object.
12. The method according to claim 11, characterized in that The step of separating the first single object audio track from the original audio track in the multimedia file comprises: The first single-object audio track is separated from the original audio track by using a trained separation network.
13. The method according to claim 12, characterized in that The trained separation network is obtained by training the separation network with training data as input of the separation network and with the loss function value being less than a first threshold as a goal, the training data includes a training track, the training track is synthesized and obtained by at least an initial third single-object track and an initial fourth single-object track, the initial third single-object track corresponds to a third sound object, the initial fourth single-object track corresponds to a fourth sound object, the third sound object belongs to the same type as the first sound object, the second sound object belongs to the same type as the fourth sound object, and the output of the separation network includes the separated third single-object track; The loss function is used to indicate a difference between the separated third single-object audio track and the initial third single-object audio track.
14. The method according to claim 4, characterized in that The spatially rendering the first single-object audio track based on the first sound source position and the type of the playback device includes: If the playback device is a headset, the rendered first single-object audio track is obtained by the following formula; in, is the rendered first single-object audio track, S is the sound object of the multimedia file and includes the first sound object, i indicates the left channel or the right channel, a s (t) is the adjustment coefficient of the first sound object at time t, h i,s (t) is the head-related transfer function HRTF filter coefficient of the left channel or the right channel corresponding to the first sound object at time t, and the HRTF filter coefficient is related to the first sound source position, s (t) is the first single object audio track at the time t, and τ is the integral term.
15. The method according to claim 4, characterized in that The spatially rendering the first single-object audio track based on the first sound source position and the type of the playback device includes: If the playback device is N external playback devices, the rendered first single object audio track is obtained by the following formula; in, in, in, is the rendered first single-object audio track, i indicates the i-th channel in the multi-channel, S is the sound object of the multimedia file and includes the first sound object, a s (t) is the adjustment coefficient of the first sound object at time t, g s (t) represents the translation coefficient of the first sound object at the time t, o s (t) is the first single object track at the time t, λ i The azimuth angle obtained by the calibrator to calibrate the i-th external device, Φ i The tilt angle obtained by the calibrator to calibrate the i-th external speaker device, r i is the distance between the i-th external speaker device and the calibrator, N is a positive integer, i is a positive integer and i≤N, and the first sound source position is within the tetrahedron formed by the N external speakers.
16. The method according to claim 4, characterized in that The method further comprises: Acquire a target audio track based on the rendered first single object audio track, the original audio track in the multimedia file, and the type of the playback device; The target audio track is sent to the playback device, and the playback device is used to play the target audio track.
17. The method according to claim 16, characterized in that The acquiring the target audio track based on the rendered first single object audio track, the original audio track in the multimedia file, and the type of the playback device includes: If the type of the playback device is headphones, the target audio track is obtained by the following formula: Among them, i indicates the left channel or the right channel, is the target track at time t, X i (t) is the original audio track at time t, is the first single object audio track that has not been rendered at the time t, is the first single object audio track after rendering, a s (t) is the adjustment coefficient of the first sound object at the time t, h i,s (t) is the head-related transfer function HRTF filter coefficient of the left channel or the right channel corresponding to the first sound object at the time t, and the HRTF filter coefficient is related to the first sound source position, s (t) is the first single-object audio track at the time t, τ is the integral term, S1 is the sound object that needs to be replaced in the original audio track, if the first sound object is to replace the sound object in the original audio track, then S1 is an empty set; S2 is the sound object added to the target audio track compared to the original audio track, if the first sound object is a copy of the sound object in the original audio track, then S2 is an empty set; S1 and / or S2 are the sound objects of the multimedia file and include the first sound object.
18. The method according to claim 16, characterized in that The acquiring the target audio track based on the rendered first single object audio track, the original audio track in the multimedia file, and the type of the playback device includes: If the type of the playback device is N external playback devices, the target audio track is obtained by the following formula: in, Where i indicates the i-th channel in the multi-channel, is the target track at time t, X i (t) is the original audio track at time t, is the first single object audio track that has not been rendered at the time t, is the first single object audio track after rendering, a s (t) is the adjustment coefficient of the first sound object at the time t, g s (t) represents the translation coefficient of the first sound object at the time t, g i,s (t) represents g s The i-th row in (t), o s (t) is the first single-object audio track at the time t, S1 is the sound object to be replaced in the original audio track, if the first sound object is to replace the sound object in the original audio track, then S1 is an empty set; S2 is the sound object added to the target audio track compared to the original audio track, if the first sound object is to copy the sound object in the original audio track, then S2 is an empty set; S1 and / or S2 are the sound objects of the multimedia file and include the first sound object, λ i The azimuth angle obtained by the calibrator to calibrate the i-th external device, Φ i The tilt angle obtained by the calibrator when calibrating the i-th external speaker device, r i is the distance between the i-th external speaker device and the calibrator, N is a positive integer, i is a positive integer and i≤N, and the first sound source position is within the tetrahedron formed by the N external speakers.
19. A rendering device, characterized in that: include: An acquisition unit, configured to acquire a first single object audio track based on the multimedia file, wherein the first single object audio track corresponds to a first sound object; a determining unit, configured to determine a first sound source position of the first sound emitting object based on reference information, wherein the reference information includes reference position information and media information of the multimedia file, and the reference position information is used to indicate the first sound source position; A rendering unit is used to perform spatial rendering on the first single-object audio track based on the first sound source position to obtain a rendered first single-object audio track.
20. The rendering device according to claim 19, characterized in that The media information includes: text to be displayed in the multimedia file, images to be displayed in the multimedia file, music features of music to be played in the multimedia file, and at least one of a sound source type corresponding to the first sound object.
21. The rendering device according to claim 19 or 20, characterized in that The reference position information includes first position information of the sensor or second position information selected by a user.
22. The rendering device according to any one of claims 19 to 21, characterized in that The determining unit is further used to determine the type of a playback device, the playback device is used to play a target audio track, and the target audio track is obtained according to the rendered first single object audio track; The rendering unit is specifically configured to perform spatial rendering on the first single-object audio track based on the first sound source position and the type of the playback device.
23. The rendering device according to claim 20, characterized in that When the media information includes the image and the image includes the first sound emitting object, the determining unit is specifically configured to determine third position information of the first sound emitting object in the image, wherein the third position information includes a two-dimensional coordinate and a depth of the first sound emitting object in the image; The determining unit is specifically configured to obtain the first sound source position based on the third position information.
24. The rendering device according to claim 20 or 23, characterized in that When the media information includes music features of the music to be played in the multimedia file, the determination unit is specifically used to determine the first sound source position based on an association relationship and the music features, wherein the association relationship is used to represent the association between the music features and the first sound source position.
25. The rendering device according to claim 20, characterized in that When the media information includes text to be displayed in the multimedia file and the text includes location text related to a location, the determining unit is specifically configured to identify the location text; The determining unit is specifically configured to determine the first sound source position based on the position text.
26. The rendering device according to claim 21, characterized in that When the reference position information includes the first position information, the acquisition unit is further used to acquire the first position information, where the first position information includes a first attitude angle of the sensor and a distance between the sensor and the playback device; The determining unit is specifically configured to convert the first position information into the first sound source position.
27. The rendering device according to claim 21, characterized in that When the reference position information includes the first position information, the acquiring unit is further used to acquire the first position information, where the first position information includes a second attitude angle of the sensor and an acceleration of the sensor; The determining unit is specifically configured to convert the first position information into the first sound source position.
28. The rendering device according to claim 21, characterized in that When the reference position information includes the second position information, the rendering device further includes: A providing unit, configured to provide a spherical view for selection by a user, wherein the center of the spherical view is the position of the user, and the radius of the spherical view is the distance between the position of the user and a playback device; The acquisition unit is further used to acquire the second position information selected by the user in the ball view; The determining unit is specifically configured to convert the second position information into the first sound source position.
29. The rendering device according to any one of claims 19 to 28, characterized in that The acquisition unit is specifically used to separate the first single object audio track from the original audio track in the multimedia file, the original audio track is synthesized by at least the first single object audio track and the second single object audio track, and the second single object audio track corresponds to the second sound object.
30. The rendering device according to claim 29, characterized in that The acquisition unit is specifically configured to separate the first single-object audio track from the original audio track by using a trained separation network.
31. The rendering device according to claim 30, characterized in that The trained separation network is obtained by training the separation network with training data as input of the separation network and with the loss function value being less than a first threshold as a goal, the training data includes a training track, the training track is synthesized and obtained by at least an initial third single-object track and an initial fourth single-object track, the initial third single-object track corresponds to a third sound object, the initial fourth single-object track corresponds to a fourth sound object, the third sound object belongs to the same type as the first sound object, the second sound object belongs to the same type as the fourth sound object, and the output of the separation network includes the separated third single-object track; The loss function is used to indicate a difference between the separated third single-object audio track and the initial third single-object audio track.
32. The rendering device according to claim 22, characterized in that If the playback device is a headset, the acquisition unit is specifically used to acquire the rendered first single-object audio track through the following formula; in, is the rendered first single-object audio track, S is the sound object of the multimedia file and includes the first sound object, i indicates the left channel or the right channel, a s (t) is the adjustment coefficient of the first sound object at time t, h i,s (t) is the head-related transfer function HRTF filter coefficient of the left channel or the right channel corresponding to the first sound object at time t, and the HRTF filter coefficient is related to the first sound source position, s (t) is the first single object audio track at the time t, and τ is the integral term.
33. The rendering device according to claim 22, characterized in that If the playback device is N external playback devices, the acquisition unit is specifically used to acquire the rendered first single object audio track through the following formula; in, in, in, is the rendered first single-object audio track, i indicates the i-th channel in the multi-channel, S is the sound object of the multimedia file and includes the first sound object, a s (t) is the adjustment coefficient of the first sound object at time t, g s (t) represents the translation coefficient of the first sound object at the time t, o s (t) is the first single object track at the time t, λ i The azimuth angle obtained by the calibrator to calibrate the i-th external device, Φ i The tilt angle obtained by the calibrator to calibrate the i-th external speaker device, r i is the distance between the i-th external speaker device and the calibrator, N is a positive integer, i is a positive integer and i≤N, and the first sound source position is within the tetrahedron formed by the N external speakers.
34. The rendering device according to claim 22, characterized in that The acquisition unit is further used to acquire a target audio track based on the rendered first single object audio track and the original audio track in the multimedia file; The rendering device also includes: A sending unit is used to send the target audio track to the playing device, and the playing device is used to play the target audio track.
35. The rendering device according to claim 34, characterized in that If the playback device is a headset, the acquisition unit is specifically configured to acquire the target audio track by using the following formula: Among them, i indicates the left channel or the right channel, is the target track at time t, X i (t) is the original audio track at time t, is the first single object audio track that has not been rendered at the time t, is the first single object audio track after rendering, a s (t) is the adjustment coefficient of the first sound object at the time t, h i,s (t) is the head-related transfer function HRTF filter coefficient of the left channel or the right channel corresponding to the first sound object at the time t, and the HRTF filter coefficient is related to the first sound source position, s (t) is the first single-object audio track at the time t, τ is the integral term, S1 is the sound object that needs to be replaced in the original audio track, if the first sound object is to replace the sound object in the original audio track, then S1 is an empty set; S2 is the sound object added to the target audio track compared to the original audio track, if the first sound object is to copy the sound object in the original audio track, then S2 is an empty set; S1 and / or S2 are the sound objects of the multimedia file and include the first sound object.
36. The rendering device according to claim 34, characterized in that If the playback device is N external playback devices, the acquisition unit is specifically used to acquire the target audio track by the following formula: in, Where i indicates the i-th channel in the multi-channel, is the target track at time t, X i (t) is the original audio track at time t, is the first single object audio track that has not been rendered at the time t, is the first single object audio track after rendering, a s (t) is the adjustment coefficient of the first sound object at the time t, g s (t) represents the translation coefficient of the first sound object at the time t, g i,s (t) represents g s The i-th row in (t), o s (t) is the first single-object audio track at the time t, S1 is the sound object to be replaced in the original audio track, if the first sound object is to replace the sound object in the original audio track, then S1 is an empty set; S2 is the sound object added to the target audio track compared to the original audio track, if the first sound object is to copy the sound object in the original audio track, then S2 is an empty set; S1 and / or S2 are the sound objects of the multimedia file and include the first sound object, λ i The azimuth angle obtained by the calibrator to calibrate the i-th external device, Φ i The tilt angle obtained by the calibrator to calibrate the i-th external speaker device, r i is the distance between the i-th external speaker device and the calibrator, N is a positive integer, i is a positive integer and i≤N, and the first sound source position is within the tetrahedron formed by the N external speakers.
37. A rendering device, characterized in that include: A processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the rendering device executes the method according to any one of claims 1 to 18.
38. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 18.
