Video playing method, device and equipment and readable storage medium

Abstract

The embodiment of the invention discloses a video playing method, apparatus and device, and a readable storage medium. An electronic device determines a spatial position of a target role for sounding in a video frame relative to a reference point. Afterwards, a dialogue signal corresponding to the video is determined, after the dialogue signal is sent by the target role, the dialogue signal is rendered according to the spatial position of the target role relative to the reference point so as to obtain multiple paths of voice signals, and the path number of the multiple paths of voice signals is the same as the first number of loudspeakers of the electronic equipment. And finally, the electronic equipment synchronously plays the video frames and the multiple paths of voice signals. By adopting the scheme, the electronic equipment renders the dialogue signal according to the spatial position of the target role in the video frame, so that the spatial position of the target role is consistent with the sound image position of the dialogue signal, and the presence of a user is enhanced while the video playing quality is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of multimedia technology, and in particular to a video playback method, apparatus, device, and readable storage medium. Background Art

[0002] With the rapid development of technology, watching multimedia videos through electronic devices has become a common entertainment activity in people's lives.

[0003] Multimedia videos, such as TV series and movies, include important components such as character images and human voice dialogues. Human voice dialogues are mainly provided in the following two ways: they are obtained by using a specific microphone array on the set and then mixed into the channel audio signal, such as 2.1 channel stereo and 5.1 channel surround sound; they are recorded in a recording studio and then directly mixed into the corresponding channel audio signal after recording.

[0004] However, no matter which method is used to produce the human voice dialogue, when the user watches the video, there will be a mismatch between the spatial position of the character image and the sound image position of the human voice dialogue. For example, the user sits 3 meters in front of the screen, the character in the human image is located in the left half of the display screen, but the human voice dialogue heard by the user comes from the middle of the screen. Summary of the invention

[0005] The embodiments of the present application provide a video playback method, apparatus, device and readable storage medium, which renders a dialogue signal according to the spatial position of a target character in a video frame so that the spatial position of the target character is consistent with the sound and image position of the dialogue signal, thereby improving the video playback quality and enhancing the user's sense of presence.

[0006] In a first aspect, an embodiment of the present application provides a video playback method, comprising:

[0007] Determine a spatial position of a target character in a video frame relative to a reference point, wherein the target character is a character that makes a sound in the video frame;

[0008] When the dialogue signal corresponding to the video frame is emitted by the target character, the dialogue signal is rendered according to the spatial position to obtain a multi-channel voice signal, wherein the number of channels of the multi-channel voice signal is the same as the first number of speakers of the electronic device;

[0009] The video frames and the multi-channel voice signals are played synchronously.

[0010] In a second aspect, an embodiment of the present application provides a video playback device, including:

[0011] A determination module, used to determine a spatial position of a target character in a video frame relative to a reference point, wherein the target character is a character that makes a sound in the video frame;

[0012] a processing module, configured to render the dialogue signal corresponding to the video frame according to the spatial position to obtain a multi-channel voice signal when the dialogue signal is emitted by the target character, wherein the number of channels of the multi-channel voice signal is the same as the first number of speakers of the electronic device;

[0013] A rendering module is used to synchronously play the video frames and the multi-channel voice signals.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the method described in the first aspect or various possible implementation methods of the first aspect.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed by a processor, they are used to implement the method described in the first aspect or various possible implementation methods of the first aspect.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product comprising a computing program, and when the computer program is executed by a processor, the method described in the first aspect or various possible implementation methods of the first aspect is implemented.

[0017] The video playback method, apparatus, device and readable storage medium provided by the embodiments of the present application, the electronic device determines the spatial position of the target character that makes a sound in the video frame relative to the reference point. Afterwards, the dialogue signal corresponding to the video is determined. When the dialogue signal is emitted by the target character, the dialogue signal is rendered according to the spatial position of the target character relative to the reference point to obtain a multi-channel voice signal. The number of channels of the multi-channel voice signal is the same as the first number of speakers of the electronic device. Finally, the electronic device synchronously plays the video frame and the multi-channel voice signal. With this solution, the electronic device renders the dialogue signal according to the spatial position of the target character in the video frame, so that the spatial position of the target character is consistent with the sound and image position of the dialogue signal, thereby improving the video playback quality and enhancing the user's sense of presence. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1It is a schematic diagram of a scenario to which the video playback method provided in the embodiment of the present application is applicable;

[0020] Figure 2 is a flow chart of a video playback method provided in an embodiment of the present application;

[0021] Figure 3 is a flow chart of determining the spatial position of a target character in a video playback method provided in an embodiment of the present application;

[0022] Figure 4 is a flowchart of a narration signal rendering process in a video playback method provided in an embodiment of the present application;

[0023] Figure 5 is a preprocessing flow chart of the video playback method provided in an embodiment of the present application;

[0024] Figure 6 is another flow chart of the video playback method provided in an embodiment of the present application;

[0025] Figure 7 A schematic diagram of a video playback device provided in an embodiment of the present application;

[0026] Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] Currently, users often use electronic devices to watch multimedia videos such as movies, TV series, and live broadcasts in their daily lives.

[0028] The important components of multimedia videos include character images and human voice dialogues. For viewers, when actually playing, the matching consistency between the spatial position of the characters in the character images and the sound and image positions can bring users an immersive audio-visual experience. Therefore, enhancing the matching between spatial position and sound and image positions is an inevitable pursuit of multimedia videos. Among them, spatial position refers to the position of the voiced character in the character images relative to the audience, and sound and image position refers to the position of the human voice dialogue corresponding to the character in the character images perceived by the audience.

[0029] At present, there are two main ways to produce human voice dialogue in multimedia videos: 1) Use a specific microphone to obtain it at the shooting scene, and then use upmixing or downmixing processing technology to mix it into the channel audio signal. Channel audio signals are, for example, 2.1 channel stereo, 5.1 channel surround sound, etc. In this way, in order to make the spatial position and sound and image position of the characters in the character picture match each other, strict requirements are placed on the layout position of the speakers of the electronic device during later playback. When the layout position of the speakers is appropriate, a rough match between the spatial position and the sound and image position can be achieved. In practice, the layout positions of the speakers of various electronic devices vary greatly. Obviously, this matching method is difficult to implement, has high limitations, and has limited accuracy.

[0030] 2) It is recorded in a recording studio later and mixed directly into the corresponding channel audio signal after recording. This method completely ignores the need to match the spatial position of the character in the character picture with the sound and image position.

[0031] Obviously, no matter which method of producing human voice dialogue is used, when users watch it later, there will be a mismatch between the spatial position of the character in the human image and the sound and image position.

[0032] Based on this, the embodiments of the present application provide a video playback method, apparatus, device and readable storage medium, which renders the dialogue signal according to the spatial position of the target character in the video frame, so that the spatial position of the target character and the sound and image position of the dialogue signal are consistent, thereby improving the video playback quality while enhancing the user's sense of presence.

[0033] Figure 1 This is a schematic diagram of a scenario applicable to the video playback method provided in the embodiment of the present application. Figure 1 , the scenario includes a terminal device 11 and a server 12. A network connection is established between the terminal device 11 and the server 12.

[0034] In the embodiment of the present application, the terminal device 11 has a display screen and at least one speaker, such as a mobile phone, a tablet computer, a television, a smart interactive tablet, etc. In one embodiment, at least one speaker is arranged inside the terminal device 11 and is integrated with the terminal device 11. In another embodiment, at least one speaker is arranged outside the terminal device 11 and is connected to the terminal device 11 by wired, wireless, Bluetooth, etc. The terminal device 11 is used to obtain and play the media stream from the server 12, and the user watches movies, TV, live broadcasts, and holds video conferences through the terminal device 11.

[0035] In the embodiment of the present application, the number of speakers of the terminal device 11 is referred to as the first number Ms, and the first number Ms is related to the audio system of the terminal device. For example, if the audio system is a 2.1-channel system, the first number Ms is 2. For another example, if the audio system is a 5.1-channel multi-channel surround sound system, the first number Ms is 5. The first number Ms is also referred to as the number of speaker channels, and the speaker is also referred to as a loudspeaker, a speaker, etc.

[0036] The number of channels of the sound source, that is, the number of audio channels of the media stream is the second number Mc, and the second number Mc is related to the generation method of the media stream. When the audio of the media stream is two-channel stereo, the second number Mc is 2; when the audio of the media stream is 5-channel surround sound, the second number Mc is 5. When the audio of the media stream is 9-channel surround sound, the second number Mc is 9.

[0037] According to the above, in the embodiment of the present application, the first number Ms of the speakers of the terminal device 11 and the second number Mc corresponding to the number of audio channels of the media stream are the same or different.

[0038] The server 12 has huge computing power, storage capacity, etc., and can provide services to terminal devices. The server 12 can be hardware or software. When the server 12 is hardware, the server 12 is a single server or a distributed server cluster composed of multiple servers. When the server 12 is software, it can be multiple software modules or a single software module, etc., which is not limited by the embodiments of the present application.

[0039] In the video playback scenario, the server 12 stores media streams, such as TV series, movies, short videos, etc. The terminal device 11 obtains the media streams from the server 12 and plays them.

[0040] In the live broadcast scenario, the media stream is the real-time audio and video data uploaded by the anchor, and the server 12 stores the media streams. The terminal device obtains the media stream from the server 12 and plays it.

[0041] In the conference scenario, the media stream is real-time audio and video data from the remote conference room. The server 12 stores these media streams. The terminal device 11 obtains the media stream from the server 12 and plays it.

[0042] In one method, after the terminal device 11 obtains the media stream from the server 12, it calculates the spatial position of the target character in each video frame in the media stream, renders the dialogue signal corresponding to the video frame according to the spatial position to obtain multiple voice signals, and plays the video frames and multiple voice signals simultaneously.

[0043] In another embodiment, the server 12 pre-stores the reference point, the positional relationship between the reference point and the display screen of the terminal device 11, and the arrangement position of each speaker of the terminal device 11. During the media stream playback process, the server 12 determines the spatial position of the target character in the video frame according to the pre-stored information, and renders the dialogue signal corresponding to the video frame according to the spatial position to obtain a multi-channel voice signal. Afterwards, the server 12 sends the video frame and the multi-channel voice signal to the terminal device 11 for simultaneous playback by the terminal device 11.

[0044] Figure 2 : is a flow chart of a video playback method provided by an embodiment of the present application. The execution subject of this embodiment is an electronic device, such as the above-mentioned terminal device or server, and this embodiment includes:

[0045] 201. Determine a spatial position of a target character in a video frame relative to a reference point, wherein the target character is a voice-generating character in the video frame.

[0046] In the embodiment of the present application, the media stream includes a video stream and an audio stream. For each video frame in the video stream, the electronic device identifies the target character that makes the sound. The target character is, for example, a person, an animal, a talking plant, a talking object such as a cloud, a stone, etc. in the picture of the video frame, or a vase that makes a shattering sound, a car that makes a braking sound, etc. in the picture of the video frame.

[0047] Usually, the target character is often a close-up shot, and the target character occupies a large area of ​​the electronic device display screen. For example, the target character occupies 20% or even more of the screen area. Moreover, the target character often has features such as lip movement. Therefore, the electronic device can determine the target character from the video frame through target detection, image segmentation and other technologies.

[0048] In the embodiment of the present application, the reference point is also referred to as the viewer position. In actual implementation, the reference point can be flexibly set. In one way, the position of the reference point is fixed. For example, the position of the reference point is at a preset distance directly in front of the display screen. The preset distance is, for example, 3 meters, 5 meters, etc., which is not limited in the embodiment of the present application. For another example, a circle is made with a position directly in front of the display screen and 4 meters away from the center point of the display screen as the center and a radius of 1.5 meters, and any point in the circular area is a reference point. When the video is playing, when there is only one viewer in the circular area, the position of the viewer is used as the reference point. When there are multiple viewers in the circular area, the position of any viewer is used as the reference point; or, the average position of the positions of each viewer is determined, and the average position is used as the reference point.

[0049] In another way, the position of the reference point is the position of the viewer captured by the electronic device. For example, when the electronic device plays a media stream, it captures an image within the field of view, determines the position of the viewer based on the image, and uses the viewer as the reference point. When there is only one viewer, the viewer may be located directly in front of, to the left, or to the right of the display screen, and therefore, the reference point may be located directly in front of, to the left, or to the right of the display screen, and so on. When there are multiple viewers, the position of any viewer is used as the reference point; or, the positions of multiple viewers are averaged to obtain an average position, and the average position is used as the reference point.

[0050] In actual implementation, the electronic device defaults to a reference point at a preset distance directly in front of the display screen. The electronic device opens a setting menu for the user to set the position of the reference point as needed. When setting, the user can select an adaptive reference point position, or set the reference point at a fixed position. When the user selects an adaptive reference point position, the electronic device captures the viewer and uses the viewer's position as the position of the reference point. In addition, the electronic device can also set the reference point according to a voice command issued by the user to set a fixed reference point, or set an adaptive reference point.

[0051] With this solution, the reference point can be flexibly set by fixed setting or real-time capture of the position of the reference point to meet different playback scenarios.

[0052] After the electronic device determines the target character from the video frame, it can determine the position of the target character in the display screen. Then, according to the position of the target character in the display screen, the positional relationship between the reference point and the display screen of the electronic device, the spatial position of the target character relative to the reference point is determined.

[0053] For example, if the reference point is 3 meters in front of the display screen and the target character is in the middle of the display screen, then the spatial position of the target character relative to the reference point is in front of it; for another example, if the reference point is 3 meters in front of the display screen and the target character is in the left half of the display screen, then the spatial position of the target character relative to the reference point is left front.

[0054] 202. When a dialogue signal corresponding to the video frame is emitted by the target character, render the dialogue signal according to the spatial position to obtain a multi-channel voice signal, wherein the number of channels of the multi-channel voice signal is the same as the first number of speakers of the electronic device.

[0055] After the electronic device determines the spatial position of the target character relative to the reference point, it determines the audio frame corresponding to the video frame from the media stream, and separates the audio frame to separate the dialogue signal and the multi-channel accompaniment signal. For example, the audio of the media stream is a two-channel stereo, including a left channel signal and a right channel signal. The electronic device separates the human voice signal 1 from the left channel signal and separates the human voice signal 2 from the right channel signal, and then synthesizes the human voice signal 1 and the human voice signal 2 to obtain the dialogue signal. The remaining signal after extracting the human voice signal 1 from the left channel signal is the left accompaniment signal, and the remaining signal after extracting the human voice from the right channel signal is the right accompaniment signal.

[0056] For another example, the audio of the media stream is dual-channel stereo, including a left channel signal and a right channel signal. The electronic device averages the left channel signal and the right channel signal to obtain an average signal, and inputs the average signal into the vocal separation network to obtain a dialogue signal. The remaining signal after extracting the dialogue signal from the left channel signal is the left accompaniment signal, and the remaining signal after extracting the dialogue signal from the right channel signal is the right accompaniment signal.

[0057] For another example, if the audio of the media stream is 5-channel surround sound, the electronic device separates the audio frames to obtain one channel of dialogue signal and 5 channels of accompaniment signals.

[0058] The dialogue signal corresponding to the video frame can be divided into at least two cases. In one case, there is a target character in the video frame, and the dialogue signal is indeed emitted by the target character. For example, the target character is a person, and the dialogue signal is a line; another example, the target character is a plant, and the dialogue signal is a line; another example, the target character is a car, and the dialogue signal is the sound of brakes; another example, the target character is a cloud, and the dialogue signal is the sound of thunder.

[0059] In another case, there is no target character in the video frame, that is, any character in the video frame does not make any sound, and there is even no human, animal or other character in the video frame, only scenery or still life. In this case, the dialogue signal is a voice-over signal.

[0060] After separating the dialogue signal corresponding to the video frame from the audio frame, the electronic device determines whether the dialogue signal is emitted by the target character. For example, the electronic device detects whether the target character has lip movement characteristics. When the target character has lip movement characteristics, it is determined that the dialogue signal is emitted by the target character; when the target character does not have lip movement characteristics, it is determined that the dialogue signal is a narration signal.

[0061] When the electronic device determines that the dialogue signal is emitted by the target character, it renders the dialogue signal according to the spatial position of the target character relative to the reference point, thereby obtaining a multi-channel voice signal, which is recorded as Si, i=1, 2...Ms, and is an integer. For example, if the number of speaker channels, i.e., the first number, is 5, the electronic device renders the dialogue signal into 5 channels, which are played by different speakers respectively.

[0062] In actual implementation, the electronic device can set the weight of each speaker. Taking the reference point being 3 meters in front of the display screen, the audio system of the electronic device being a 5-channel surround sound system, and the target character being located in the center of the display screen as an example, the electronic device has 5 speaker channels, namely front left channel, center channel, front right channel, rear right channel and rear left channel, and the weights are weight 1, weight 2, weight 3, weight 4 and weight 5 respectively, then weight 2>weight 1, weight 1=weight 4=weight 3=weight 5=0.

[0063] Assuming that the target character is located in the left half of the display screen, weight 3 = weight 4 = weight 5 = 0, and the size of weight 1 and weight 2 is related to the position of the target character. For example, if the target character is close to the front left channel speaker, weight 1> weight 2; if the target character is close to the center channel speaker, weight 2 is greater than weight 1; if the target character is between the front left channel speaker and the center channel speaker, weight 1 = weight 2; if the target angle is located at the front left channel speaker, weight 1 is the largest, weight 2 = weight 4 = weight 3 = weight 5 = 0; if the target character is located at the center channel speaker, weight 2 is the largest, weight 1 = weight 4 = weight 3 = weight 5 = 0. It can be understood that when the weight of a certain channel is 0, the amplitude of the voice signal of that channel is 0.

[0064] 203. Synchronously play the video frame and the multi-channel voice signals.

[0065] After the electronic device synchronizes the video frame and the multi-channel voice signal corresponding to the video frame, that is, after the electronic device performs audio and video synchronization, the video frame is displayed on the display screen while the multi-channel voice signal is played by each speaker of the electronic device. At this time, for the viewer at the reference point, the sound and image position of the multi-channel voice signal perceived by hearing matches the spatial position of the target character in the video frame. For example, the user sees that the target character is located in the right half of the display screen, and the sound heard is also located in the right half of the display screen. The target character will not be located in the right half of the display screen, but the sound will not be located in the middle or even on the left side of the display screen.

[0066] In the video playback method provided by the embodiment of the present application, the electronic device determines the spatial position of the target character that makes a sound in the video frame relative to the reference point. After that, the dialogue signal corresponding to the video is determined. When the dialogue signal is emitted by the target character, the dialogue signal is rendered according to the spatial position of the target character relative to the reference point to obtain a multi-channel voice signal. The number of channels of the multi-channel voice signal is the same as the first number of speakers of the electronic device. Finally, the electronic device synchronously plays the video frame and the multi-channel voice signal. With this solution, the electronic device renders the dialogue signal according to the spatial position of the target character in the video frame, so that the spatial position of the target character and the sound and image position of the dialogue signal are consistent, thereby improving the video playback quality and enhancing the user's sense of presence.

[0067] Figure 3 1 is a flow chart of determining the spatial position of a target character in a video playback method provided in an embodiment of the present application. This embodiment includes:

[0068] 301. Determine, from the video frame, a character whose pixel ratio is greater than a preset ratio, to obtain the target character.

[0069] In this step, the electronic device identifies the target character from the video frame and calibrates the position of the target character. During the target character identification process, the electronic device uses the close-up character in the video frame as the target character. The close-up character is, for example, a character whose pixel ratio in the video frame is greater than a preset ratio, and the preset ratio is, for example, 20%, etc., which is not limited in the embodiments of the present application.

[0070] 302. Determine a pixel point position of the target character in the video frame according to the pixel points included in the target character.

[0071] The electronic device can flexibly determine the pixel position of the target character, and the pixel position represents the position of the target character in the video frame.

[0072] In one method, the electronic device determines the center position of the mouth, the center position of the head, etc. of the target character, and uses the center position as the pixel point position.

[0073] In another method, the electronic device determines the average value of the pixel points of a specific part of the target character, and uses the average value as the pixel point position of the target character. For example, the specific part is the head of the target character, and the electronic device detects the head area through face detection technology. The coordinates (x o ,y o ), o = 1, 2...N, N is the number of pixels in the head area.

[0074] After determining the coordinates of each pixel point in the human head area, the electronic device determines the average value (x ave ,y ave ), and use the average value as the pixel position of the target character.

[0075] 303. Taking the reference point as a coordinate origin, determine the vertex positions of each vertex of a display screen, where the display screen is used to display the video frame.

[0076] The display screen of an electronic device is usually rectangular. The electronic device uses the reference point as the center of the circle to determine the vertex positions (x q ,y q , z q), q = 1, 2, 3, 4. The vertex position is the coordinate of the vertex of the display screen relative to the reference point.

[0077] It should be noted that there is no strict order between the above steps 301 and 303.

[0078] 304. Determine the spatial position according to the vertex positions of each vertex of the display screen and the pixel positions.

[0079] After the electronic device determines the pixel position of the target character in the video frame and the vertex position of each vertex of the display screen relative to the reference point, it converts the pixel position into a spatial position (x p ,y p , z p ).

[0080] With this solution, the electronic device determines the spatial position of the target character relative to the reference point based on the vertex positions of each vertex of the display screen and the pixel position of the target character, with high speed and accuracy.

[0081] Optionally, in the above embodiment, when the dialogue signal is emitted by the target character, the electronic device renders the dialogue signal according to the spatial position to obtain a multi-channel voice signal, first, the electronic device uses the reference point as the coordinate origin to determine the arrangement position of each speaker of the electronic device relative to the reference point. Thereafter, the electronic device renders the dialogue signal according to the arrangement position of each speaker and the spatial position to obtain a multi-channel voice signal.

[0082] In the embodiment of the present application, each speaker can be integrated inside the electronic device, or can be arranged outside the electronic device and establish a Bluetooth connection, a network connection, etc. with the electronic device. When the speaker is integrated inside the electronic device, the electronic device pre-stores the relative position of each speaker relative to the center point of the display screen of the electronic device, and determines the arrangement position of each speaker relative to the reference point based on the relative position, the position of the center point of the display screen, etc.

[0083] When the speaker is an external speaker, the electronic device can capture an image through a camera to obtain an image containing each speaker, and determine the arrangement position of each speaker relative to the reference point based on the image, etc. In addition, the electronic device can also pick up the sound of the speaker array and then estimate the direction of each speaker to determine the arrangement position of each speaker relative to the reference point.

[0084] In the above embodiment, the arrangement position of the loudspeaker can be expressed as a coordinate (xi, yi, zi), where i=1, 2, 3...Ms, and Ms is the number of loudspeakers, that is, the first number.

[0085] Afterwards, the electronic device renders the dialogue signal according to the arrangement position of each speaker, the spatial position of the target character relative to the reference point, etc. to obtain a multi-channel voice signal.

[0086] For example, the first number is 2, that is, the number of speaker channels is 2, and the left and right speakers are symmetrically arranged relative to the reference point. When the spatial position indicates that the target character is located in the left front, it means that the target character is located in the left half of the display screen. When playing, the effect of the sound heard by the viewer in the left front is achieved, so that the sound image position matches the spatial position of the target character. In order to achieve this effect, the rendered multi-channel voice signal includes a left channel voice signal and a right channel voice signal. The amplitude of the left channel signal is larger, and the amplitude of the right channel signal is smaller. The sound image synthesized by the left channel signal and the right channel signal is located in the left front of the user.

[0087] For another example, the first number is 2, that is, the number of speaker channels is 2, the left speaker is farther from the reference point, and the right speaker is closer to the reference point. When the spatial position indicates that the target character is located directly in front of the viewer, it means that the target character is located in the middle part of the display screen, and the sound heard by the viewer during playback should be in front of the viewer. In this way, the sound and image position matches the spatial position of the target character. In order to achieve this effect, the rendered multi-channel voice signal includes a left-channel voice signal and a right-channel voice signal. The delay of the left-channel signal is small, and the delay of the right-channel signal is large. The sound and image synthesized by the left-channel signal and the right-channel signal is located directly in front of the user.

[0088] It is understandable that when the same media stream is played on different electronic devices, the rendering algorithm of the dialogue signal is different. For example, if the audio system of electronic device 1 is a two-channel stereo system, one dialogue signal is rendered into two voice signals; for another example, if the audio system of electronic device 2 is a five-channel surround sound system, one dialogue signal is rendered into five voice signals; for another example, if electronic devices 3 and 4 are two-channel stereo systems, but the speaker arrangements are different, the two voice signals rendered by electronic device 3 are different from the two voice signals rendered by electronic device 4.

[0089] With this solution, the electronic device renders the dialogue signal according to the arrangement position of each speaker and the spatial position of the target character relative to the reference point to obtain a multi-channel voice signal. This solution can adapt to different speaker arrangements, has a simple rendering method, and has a wide range of applications.

[0090] The above describes how to render the dialogue signal emitted by the target character to achieve the purpose of matching the spatial position of the target character with the sound image position of the dialogue signal. The following describes the rendering methods of the narration signal and the accompaniment sound respectively.

[0091] First, the narration signal.

[0092] Figure 4 It is a flowchart of the narration signal rendering process in the video playback method provided in an embodiment of the present application.

[0093] This embodiment includes:

[0094] 401. Taking the reference point as a coordinate origin, determine the vertex positions of each vertex of a display screen, where the display screen is used to display the video frame.

[0095] For details, please refer to the above Figure 3 The description of step 303 is not repeated here.

[0096] 402. Determine a center position of the display screen relative to the reference point according to vertex positions of each vertex of the display screen.

[0097] For example, the electronic device can calculate the vertex position (x q ,y q , z q ), determine the center of the display relative to the center of the reference point (x z ,y z , z z ).

[0098] 403. Render the dialogue signal according to the arrangement positions of the speakers of the electronic device and the center position to obtain a multi-channel narration signal.

[0099] For example, the reference point is located in front of the display screen and the speakers are arranged symmetrically. To achieve the effect that the narration heard by the viewer is located in front of the screen, the rendered multi-channel voice signal includes a left-channel voice signal and a right-channel voice signal. The amplitude of the left-channel signal is the same as that of the right-channel signal, and the synthesized sound image of the left-channel signal and the right-channel signal is located in front of the reference point.

[0100] Assume that the first number is 2, that is, the number of speaker channels is 2, the left speaker is farther from the reference point, and the right speaker is closer to the reference point. In order to achieve the effect that the narration heard by the viewer is located directly in front of the viewer, the rendered multi-channel voice signal includes a left channel voice signal and a right channel voice signal. The delay of the left channel signal is small, and the delay of the right channel signal is large. The sound image synthesized by the left channel signal and the right channel signal is located directly in front of the user.

[0101] With this solution, the electronic device renders the narration signal according to the arrangement position of each speaker and the center position of the display screen relative to the reference point to obtain a multi-channel narration signal. This solution can adapt to different speaker arrangements, has a simple rendering method, and has a wide range of applications.

[0102] Second, the accompaniment signal.

[0103] In the embodiment of the present application, the number of paths of the accompaniment signal is the second number, that is, the number of audio paths of the media stream. After the electronic device separates the human voice from the audio frame, the remaining path signals are the accompaniment signals, denoted by C i , i=1, 2...Mc, Mc is the second number. For example, if the audio of the media stream is two-channel stereo, then after the dialogue signal is separated, there are still two channels of accompaniment signals. For another example, if the audio of the media stream is five-channel surround sound, then after the dialogue signal is separated, there are still five channels of accompaniment signals.

[0104] For multiple accompaniment signals, the electronic device determines a rendering method according to the relationship between the first quantity and the second quantity.

[0105] When the first number Ms and the second number Mc are different, the electronic device mixes the multiple accompaniment signals according to the arrangement positions of the speakers of the electronic device, the first number, and the second number to obtain a multiple background signal, and the number of the multiple background signals is the first number. i , i=1, 2...Ms, Ms is the first quantity.

[0106] When the first number Ms is less than the second number Mc, the electronic device uses an upmixing algorithm to mix the accompaniment signal of the Mc channel. During the mixing process, the arrangement position of each speaker is considered at the same time, so as to obtain the background signal of the Ms channel.

[0107] When the first number Ms is greater than the second number Mc, the electronic device uses a down-mixing algorithm to mix the accompaniment signal of the Mc path. During the mixing process, the arrangement position of each speaker is considered at the same time, so as to obtain the background signal of the Ms path.

[0108] When the first number Ms is equal to the second number Mc, in one way, the corresponding signals in the multi-channel accompaniment signals and the multi-channel background signals are the same. That is, the electronic device does not perform any processing on the multi-channel accompaniment signals. For example, the first number Ms=the second number Mc=2, the electronic device uses the accompaniment signal of the left channel as the background signal of the left channel, and uses the accompaniment signal of the right channel as the background signal of the right channel.

[0109] In another way, the electronic device renders each accompaniment signal in the multiple accompaniment signals according to the arrangement position of each speaker to obtain the corresponding speaker path signal. In this way, the electronic device adjusts the delay, amplitude, phase, etc. of each accompaniment signal in combination with the position of each speaker to obtain the multiple background signals.

[0110] In another method, the electronic device re-renders the multi-channel accompaniment signal according to the arrangement position of each speaker of the electronic device and the speaker arrangement corresponding to the media stream to be played, so as to obtain the multi-channel background signal of each speaker of the electronic device, and the video frame is included in the media stream. In this method, the electronic device obtains the arrangement position of each speaker when the media stream is produced, that is, the speaker arrangement corresponding to the sound source. Then, according to the arrangement position of each speaker of the current electronic device and the arrangement position of the speaker corresponding to the media stream when the media stream is played back, the multi-channel accompaniment signal is re-rendered to obtain the multi-channel background signal of each speaker of the electronic device.

[0111] The electronic device obtains the background signal Cp of Ms path i After that, the electronic device converts the background signal Cp of Ms path i and the speech signal S i Mix the sound to get the channel signal Y corresponding to each speaker i , Y i =Cp i +S i , i=1, 2...Ms, Ms is the first quantity.

[0112] With this solution, the electronic device mixes Mc-channel accompaniment signals according to the first number, the second number, the arrangement positions of the speakers, etc. to obtain background signals of the speakers of the electronic device, and the processing method is simple and fast.

[0113] Optionally, in the above embodiment, before the electronic device determines the spatial position of the target character in the video frame relative to the reference point, it also performs some preprocessing on the media stream. Figure 5 . Figure 5 : is a preprocessing flow chart of the video playback method provided in the embodiment of the present application. This embodiment includes:

[0114] 501. Determine a video frame sequence and an audio frame sequence according to a media stream to be played.

[0115] Media streams include video streams and audio streams. The electronic device obtains a video frame sequence based on the video stream and an audio frame sequence based on the audio stream. Video frames and audio frames are not necessarily one-to-one corresponding. The correspondence between video frames and audio frames is related to the frame rate, audio sampling frequency, etc.

[0116] 502. Determine a role set including different roles according to the video frame sequence, wherein any role in the role set appears in at least one video frame of the video frame sequence, and the roles in the role set are voice-speaking roles.

[0117] For each video frame to be played in the video frame, the electronic device determines the target role in the video frame, and the target roles of multiple video frames form a role set, and the roles in the role set are not repeated. For example, the video frame sequence includes video frame 1 to video frame 2000, a total of 2000 video frames. Among them, video frame 1 to video frame 1000 include target role 1, and video frame 1001 to video frame 2000 include target role 2, so there are 2 roles in the role set.

[0118] 503. Determine multiple dialogue sets from the audio frame sequence, where the dialogue sets include at least one dialogue signal, and different dialogue sets correspond to different voiceprints.

[0119] The electronic device separates the human voice from each audio frame in the audio frame to obtain multiple dialogue signals. At the same time, the dialogue signals are classified according to the voiceprint, etc., and the dialogue signals with the same voiceprint are sorted in the order of playback and placed in the same dialogue set. The dialogue signals with the same voiceprint are often emitted by the same character. A dialogue set contains at least one dialogue signal.

[0120] 504. For each video frame in the video frame sequence, determine the target character from the character set, and determine a dialogue signal from the dialogue set corresponding to the target character.

[0121] When playing the media stream, the electronic device determines the dialogue set corresponding to each character, and determines the dialogue signal corresponding to the current video frame from the dialogue set, synchronizes the video frame and the dialogue signal, and plays them.

[0122] With this solution, the electronic device pre-processes the media stream, matches and synchronizes the video frames and the corresponding dialogue signals, thereby achieving the purpose of improving the smoothness of video playback.

[0123] Optionally, in the above embodiment, when a target set exists in multiple dialogue sets, the dialogue signal in the target set is determined to be a narration signal, and the voiceprint corresponding to the target set is different from the voiceprint of any character in the character set.

[0124] Exemplarily, if there is no narration signal, the number of characters in the character set is the same as the number of dialogue sets. That is, for any character that speaks, there is a dialogue set corresponding to the character in multiple dialogue sets. When there is a narration signal, the number of dialogue sets is greater than the number of characters in the character set. The electronic device uses the dialogue set that cannot be matched to a character in the multiple dialogue sets as the narration signal set.

[0125] By adopting this solution, the electronic device determines the narration signal during the preprocessing of the streaming media, which facilitates the subsequent rendering of the narration signal and achieves the purpose of improving the smoothness of video playback.

[0126] Figure 6 is another flow chart of the video playback method provided by the embodiment of the present application. This embodiment includes:

[0127] 601. Obtain media stream.

[0128] The electronic device obtains a media stream to be played, the media stream including a video stream and an audio stream, and obtains a video frame sequence according to the video stream and an audio frame sequence according to the audio stream.

[0129] 602. Identify a target character in a video frame and calibrate a position of the target character.

[0130] For each video frame, the electronic device identifies the target character speaking in the video frame through AI face recognition and AI face detection related to deep learning algorithms, and calibrates the position of the target character. The target character is often a close-up shot, and the target character occupies a large area of ​​the electronic device display. For example, the target character occupies 20% or even more of the screen.

[0131] The electronic device can calibrate the position of the target object according to the center position of the mouth and the center position of the head of the target object, so as to determine the pixel position of the target character, that is, the position of the target character in the video frame. Figure 3 The description of step 302 is not repeated here.

[0132] In the implementation of this application, each time the electronic device identifies a target role, it sets a label for the target role. When a new target role appears, a new role label is added, such as label 1, label 2, label 3, etc. Different roles form a role set.

[0133] 603. Extract, separate and classify dialogue signals from audio frames.

[0134] In the embodiment of the present application, there are the following situations in the audio corresponding to the video frame:

[0135] Case a: There is a dialogue signal in the audio frame, and the dialogue signal is emitted by the target object in the video frame;

[0136] Case b: There is a dialogue signal in the audio frame, but there is no target object in the video frame corresponding to the audio frame, that is, the dialogue signal is a narration signal.

[0137] For cases a and b, after the electronic device extracts the dialogue signal from the audio frame, the remaining audio component is used as the Mc-channel accompaniment signal.

[0138] Case c: There is no dialogue signal in the audio frame. At this time, the electronic device uses all Mc-channel audio signals as accompaniment signals, namely, Mc-channel accompaniment signals.

[0139] In the embodiment of the present application, the dialogue signal extraction method includes but is not limited to speech separation algorithm, etc. After the electronic device extracts the dialogue signal from each audio frame in the audio frame sequence, it classifies the dialogue signal, and the classification method includes but is not limited to voiceprint recognition algorithm.

[0140] Each time the electronic device recognizes a dialogue signal, it sets a label for the dialogue signal. Dialogue signals with the same voiceprint feature have the same label. When a dialogue signal with a new voiceprint feature appears, a new label is set for the dialogue signal, such as label a, label b, etc. Dialogue signals with the same voiceprint form a dialogue set, thereby obtaining multiple dialogue sets.

[0141] 604. The character and dialogue sets match.

[0142] For each target character in the character set, the electronic device determines a dialogue set that matches the target character from multiple dialogue sets. The matching method is obtained by the character classification label and the dialogue classification label. When a dialogue set does not have a target character that matches it, the dialogue signal in the dialogue set is a narration signal.

[0143] 605. Render the dialogue signal to obtain a multi-channel voice signal.

[0144] The electronic device determines the spatial position of the target character relative to the reference point based on the pixel position of the target character in the video frame, the position of the vertex of the display screen, etc. The specific process can be seen in the above Figure 3 The relevant description will not be repeated here.

[0145] The electronic device renders the dialogue signal of the matched target character according to the spatial position of the target character relative to the reference point, the arrangement position of each speaker, etc., thereby obtaining a multi-channel voice signal, which is recorded as S i , i=1, 2...Ms, and is an integer.

[0146] For the narration signal, the electronic device renders the narration signal according to the arrangement position and center position of each speaker to obtain a multi-channel narration signal, which is recorded as S i , i=1, 2...Ms, and is an integer.

[0147] It is understandable that, in most cases, the dialogue signal corresponding to the same video frame is either emitted by the target character or a narration signal.

[0148] 606. Render the multiple accompaniment signals to obtain multiple background signals.

[0149] The electronic device performs audio upmixing or downmixing on the Mc-channel accompaniment signal obtained in step 603 in combination with the arrangement positions of the speakers, thereby obtaining the Ms-channel background signal Cp. i , i=1, 2...Ms, Ms is the first quantity.

[0150] 607. For multiple voice signals Si and multiple background signals Cp i Mixing.

[0151] Exemplarily, the electronic device converts the Ms path background signal Cp in step 606 into i and the voice S of Ms in step 605 i Mix and get the rendered multi-channel speaker signal Y i , Y i =Cp i +S i , i=1,2…MS.

[0152] 608. The electronic device synchronously plays the processed multi-channel loudspeaker signal Y i and video frames.

[0153] Exemplarily, the electronic device displays the video frame using a display screen and plays the multi-channel speaker signal Y using each speaker. i .

[0154] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0155] Figure 7 A schematic diagram of a video playback device provided in an embodiment of the present application. The video playback device 700 is integrated in an electronic device, and the video playback device 700 includes: a determination module 71 , a processing module 72 , and a rendering module 73 .

[0156] A determination module 71, configured to determine a spatial position of a target character in a video frame relative to a reference point, wherein the target character is a character that makes a sound in the video frame;

[0157] A processing module 72, configured to render the dialogue signal corresponding to the video frame according to the spatial position to obtain a multi-channel voice signal when the dialogue signal is emitted by the target character, wherein the number of channels of the multi-channel voice signal is the same as the first number of speakers of the electronic device;

[0158] The playing module 73 is used to synchronously play the video frame and the multi-channel voice signal.

[0159] In a feasible implementation, the processing module 72 is used to determine the layout position of each speaker of the electronic device relative to the reference point with the reference point as the coordinate origin; render the dialogue signal according to the layout position of each speaker of the electronic device and the spatial position to obtain a multi-channel voice signal.

[0160] In a feasible implementation, the processing module 72 is also used to, when the dialogue signal is a voice-over signal, determine the vertex positions of each vertex of a display screen using the reference point as a coordinate origin, and the display screen is used to display the video frame; determine the center position of the display screen relative to the reference point based on the vertex positions of each vertex of the display screen; and render the dialogue signal based on the arrangement positions of each speaker of the electronic device and the center position to obtain a multi-channel voice-over signal.

[0161] In a feasible implementation, the processing module 72 is further used to determine a multi-channel accompaniment signal according to the audio frame, and the number of the multi-channel accompaniment signals is a second number; when the first number and the second number are different, the multi-channel accompaniment signal is mixed according to the arrangement position of each speaker of the electronic device, the first number, and the second number to obtain a multi-channel background signal, and the number of the multi-channel background signal is the first number; when the first number and the second number are the same, the multi-channel accompaniment signal is re-rendered according to the arrangement position of each speaker of the electronic device and the speaker arrangement corresponding to the media stream to be played, so as to obtain a multi-channel background signal of each speaker of the electronic device, and the video frame is included in the media stream; and a multi-channel speaker path signal is obtained according to the multi-channel voice signal and the multi-channel background signal;

[0162] The playing module 73 is used to synchronously play the video frame and the multi-channel speaker channel signal.

[0163] In a feasible implementation, the reference point is located at a preset distance in front of the display screen of the electronic device, or the reference point is the position of the viewer captured by the electronic device.

[0164] In a feasible implementation, the determination module 71 is used to determine a character whose pixel ratio is greater than a preset ratio from the video frame to obtain the target character; determine the pixel point position of the target character in the video frame based on the pixel points contained in the target character; determine the vertex positions of each vertices of the display screen using the reference point as the coordinate origin, and the display screen is used to display the video frame; determine the spatial position based on the vertex positions of each vertices of the display screen and the pixel point positions.

[0165] In a feasible implementation, the processing module 72 is further used to determine a video frame sequence and an audio frame sequence according to the media stream to be played before the determination module 71 determines the spatial position of the target character in the video frame relative to the reference point; determine a role set containing different characters according to the video frame sequence, any character in the role set appears in at least one video frame of the video frame sequence, and the characters in the role set are speaking characters; determine multiple dialogue sets from the audio frame sequence, the dialogue sets contain at least one dialogue signal, and different dialogue sets correspond to different voiceprints; for each video frame in the video frame sequence, determine the target character from the role set, and determine the dialogue signal from the dialogue set corresponding to the target character.

[0166] In a feasible implementation, the processing module 72 is further used to determine that the dialogue signal in the target set is a narration signal when there is a target set in multiple dialogue sets, and the voiceprint corresponding to the target set is different from the voiceprint of any character in the character set.

[0167] The video playback device provided in the embodiment of the present application can execute the actions of the electronic device in the above embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0168] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 8 The electronic device 800 described in the embodiment of the present application includes: at least one processor 81, at least one network interface 84, a user interface 83, a memory 85, at least one communication bus 82 and multiple speakers 86.

[0169] The communication bus 82 is used to realize the connection and communication between these components.

[0170] The user interface 83 may include a display screen (Display) and a camera (Camera), and the optional user interface 83 may also include a standard wired interface and a wireless interface.

[0171] The network interface 84 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0172] Among them, the processor 81 may include one or more processing cores. The processor 81 uses various interfaces and lines to connect various parts of the entire electronic device 800, and executes various functions and processes data of the electronic device 80 by running or executing instructions, programs, code sets or instruction sets stored in the memory 85, and calling data stored in the memory 85. Optionally, the processor 81 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 81 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 81, and it can be implemented separately through a chip.

[0173] After the processor 81 determines the spatial position of the target character in the video frame relative to the reference point, it renders the dialogue signal according to the spatial position, etc. The display screen in the user interface 83 displays the video frame, and each speaker 86 is used to play the multi-channel speaker signal.

[0174] Among them, the memory 85 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 85 includes a non-transitory computer-readable storage medium. The memory 85 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 85 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area may store data involved in the above-mentioned method embodiments, etc. The memory 85 may also be optionally at least one storage device located away from the aforementioned processor 81. As Figure 8 As shown, the memory 85 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an operating application of the electronic device.

[0175] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0176] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0177] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0178] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0179] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0180] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0181] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0182] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0183] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A video playback method, characterized in that: Applied to electronic equipment, the method comprises: Determine a spatial position of a target character in a video frame relative to a reference point, wherein the target character is a character that makes a sound in the video frame; When the dialogue signal corresponding to the video frame is emitted by the target character, the dialogue signal is rendered according to the spatial position to obtain a multi-channel voice signal, wherein the number of channels of the multi-channel voice signal is the same as the first number of speakers of the electronic device; The video frames and the multi-channel voice signals are played synchronously.

2. The method according to claim 1, characterized in that When the dialogue signal corresponding to the video frame is emitted by the target character, rendering the dialogue signal according to the spatial position to obtain a multi-channel voice signal includes: Taking the reference point as the coordinate origin, determining the arrangement position of each speaker of the electronic device relative to the reference point; The dialogue signal is rendered according to the arrangement positions of the speakers of the electronic device and the spatial position to obtain a multi-channel voice signal.

3. The method according to claim 1, characterized in that Also includes: When the dialogue signal is a voice-over signal, taking the reference point as a coordinate origin, determining the vertex positions of each vertex of a display screen, the display screen being used to display the video frame; Determining the center position of the display screen relative to the reference point according to the vertex positions of each vertex of the display screen; The dialogue signal is rendered according to the arrangement positions of the speakers of the electronic device and the center position to obtain a multi-channel narration signal.

4. The method according to any one of claims 1 to 3, characterized in that: The synchronously playing the video frame and the multi-channel voice signal includes: Determine a plurality of accompaniment signals according to the audio frame, wherein the number of the plurality of accompaniment signals is a second number; When the first number and the second number are different, the multi-channel accompaniment signals are mixed according to the arrangement positions of the speakers of the electronic device, the first number, and the second number to obtain multi-channel background signals, and the number of the multi-channel background signals is the first number; when the first number and the second number are the same, the multi-channel accompaniment signals are re-rendered according to the arrangement positions of the speakers of the electronic device and the speaker arrangement corresponding to the media stream to be played to obtain multi-channel background signals of the speakers of the electronic device, and the video frame is included in the media stream; Obtaining multi-channel speaker path signals according to the multi-channel voice signals and the multi-channel background signals; The video frames and the multi-channel speaker path signals are played synchronously.

5. The method according to any one of claims 1 to 3, characterized in that: The reference point is located at a preset distance in front of the display screen of the electronic device, or the reference point is the position of the viewer captured by the electronic device.

6. The method according to any one of claims 1 to 3, characterized in that: Determining the spatial position of the target character in the video frame relative to the reference point includes: Determine from the video frame a character whose pixel ratio is greater than a preset ratio to obtain the target character; Determine the pixel position of the target character in the video frame according to the pixel points included in the target character; Taking the reference point as a coordinate origin, determining the vertex position of each vertex of a display screen, the display screen being used to display the video frame; The spatial position is determined according to the vertex positions of each vertex of the display screen and the pixel positions.

7. The method according to any one of claims 1 to 3, characterized in that: Before determining the spatial position of the target character in the video frame relative to the reference point, the method further includes: Determine a video frame sequence and an audio frame sequence according to the media stream to be played; Determining a role set including different roles according to the video frame sequence, any one of the roles in the role set appears in at least one video frame of the video frame sequence, and the roles in the role set are voice-speaking roles; Determining a plurality of dialogue sets from the audio frame sequence, wherein the dialogue sets include at least one dialogue signal, and different dialogue sets correspond to different voiceprints; For each video frame in the video frame sequence, the target character is determined from the character set, and a dialogue signal is determined from the dialogue set corresponding to the target character.

8. The method according to claim 7, characterized in that Also includes: When a target set exists in the multiple dialogue sets, it is determined that the dialogue signal in the target set is a narration signal, and the voiceprint corresponding to the target set is different from the voiceprint of any character in the character set.

9. A video playback device, characterized in that: include: A determination module, used to determine a spatial position of a target character in a video frame relative to a reference point, wherein the target character is a character that makes a sound in the video frame; a processing module, configured to render the dialogue signal corresponding to the video frame according to the spatial position to obtain a multi-channel voice signal when the dialogue signal is emitted by the target character, wherein the number of channels of the multi-channel voice signal is the same as the first number of speakers of the electronic device; A rendering module is used to synchronously play the video frames and the multi-channel voice signals.

10. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the electronic device implements the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images