A playback processing method, device, equipment and storage medium

Abstract

The embodiment of the present application discloses a playback processing method, device, equipment and storage medium, the playback processing method includes: displaying a video playback interface; in response to an audio playback triggering event, hiding the image screen of the video; and playing the content audio corresponding to the video; in the process of playing the content audio, playing the narration audio of the video, the narration audio is generated based on the recognition of the plot content of the video. By using the embodiment of the present application, the plot content can be prompted during the audio playback process, effectively improving the audio playback effect.

Description

Technical Field

[0001] The present application relates to the field of computer technology, in particular to the field of audio and video processing technology, and specifically to a playback processing method, a playback processing device, a playback processing equipment, and a computer-readable storage medium. Background Art

[0002] At present, when it is inconvenient for users to watch videos (such as TV series, movies, variety shows, etc.), they can play the content audio corresponding to the video; the so-called content audio refers to the synchronous audio of the video, which may include character dialogues, background music, environmental sound effects (such as wind, water, etc.) in the video, etc. In actual application scenarios, when playing the content audio corresponding to the video, due to the lack of the prompt function of the video image, the user cannot know some key information in the plot content of the video, which is not convenient for the user to understand the plot content, and the audio playback effect is poor. Summary of the invention

[0003] The embodiments of the present application provide a playback processing method, device, equipment and storage medium, which can prompt the plot content during audio playback and effectively improve the audio playback effect.

[0004] On the one hand, an embodiment of the present application provides a playback processing method, the playback processing method comprising:

[0005] Display the video playback interface;

[0006] In response to an audio playback trigger event, hiding the image of the video; and playing the content audio corresponding to the video;

[0007] During the process of playing the content audio, the narration audio of the video is played, and the narration audio is generated based on the recognition of the plot content of the video.

[0008] In the embodiment of the present application, in response to the audio playback triggering event, the image of the video can be hidden, the content audio corresponding to the video can be played, and the narration audio of the video can be played during the playback of the content audio. The narration audio is generated based on the recognition of the plot content of the video, so the narration audio can prompt the plot content of the video. Playing the narration audio during the playback of the content audio can help users have a richer and deeper understanding of the plot content, effectively improving the audio playback effect.

[0009] On the other hand, an embodiment of the present application provides a playback processing method, the playback processing method comprising:

[0010] Get the video to be processed;

[0011] Identify the plot content of the video and obtain key information of the plot content;

[0012] The narration audio of the video is generated according to the key information, and the narration audio is used to be played during the process of playing the content audio corresponding to the video when the image screen of the video is hidden.

[0013] In the embodiment of the present application, the plot content of the video can be identified to obtain key information of the plot content, and the narration audio of the video can be generated according to the key information of the plot content; the narration audio can be used to be played in the process of playing the content audio corresponding to the video when the image screen of the video is hidden. The narration audio is generated according to the key information obtained by identifying the plot content, so the narration audio can prompt the key information in the plot content of the video. When the image screen of the video is hidden, playing the narration audio in the process of playing the content audio can help users have a richer and deeper understanding of the plot content, and effectively improve the audio playback effect.

[0014] On the other hand, an embodiment of the present application provides a playback processing device, the playback processing device comprising:

[0015] A display unit, used to display a video playback interface;

[0016] A processing unit, configured to hide the image of the video in response to an audio playback triggering event; and to play the content audio corresponding to the video;

[0017] The processing unit is also used to play the narration audio of the video during the process of playing the content audio, and the narration audio is generated based on the recognition of the plot content of the video.

[0018] In one implementation, the narration audio includes N audio clips, each audio clip corresponds to a playback time point of the content audio; any audio clip among the N audio clips is represented as the i-th audio clip, and the i-th audio clip corresponds to the i-th playback time point of the content audio; wherein i and N are both positive integers and i≤N;

[0019] The processing unit is used to perform the following steps when playing the narration audio of the video during the process of playing the content audio:

[0020] When the content audio is played to the i-th playback time point, the content audio is paused and the i-th audio segment is played; and after the i-th audio segment is finished playing, the content audio continues to be played.

[0021] In one implementation, the playback interface is a service interface of an application program, and the application program includes a narration audio playback switch; the processing unit is further configured to perform the following steps:

[0022] During the playing of the i-th audio segment, in response to the narration audio playing switch being triggered to be turned off, the playing of the i-th audio segment is stopped and the playing of the content audio continues.

[0023] In one implementation, the narration audio is used to prompt key information of the plot content of the video; wherein the i-th audio clip is used to prompt at least one of the following key information: scene switching information occurring at the i-th playback time point of the video, character objects appearing at the i-th playback time point of the video, and action information performed by the character objects appearing at the i-th playback time point of the video.

[0024] In one implementation, the display unit is further configured to perform the following steps:

[0025] When the content audio is played to the first progress, in response to the video playback triggering event, the image screen of the video is restored from the first progress in the playback interface.

[0026] In one implementation, the processing unit is further configured to perform the following steps:

[0027] In the process of displaying the video playback interface, in response to the first gravity sensing operation, an audio playback triggering event is generated;

[0028] Alternatively, during the playback of the content audio or the narration audio, a video playback trigger event is generated in response to a second gravity sensing operation.

[0029] In one implementation, the playback interface includes an audio playback option and a video playback option; the processing unit is further configured to perform the following steps:

[0030] When the audio playback option is selected, an audio playback trigger event is generated;

[0031] Alternatively, when the video play option is selected, a video play trigger event is generated.

[0032] In one implementation, the playback interface is a service interface of an application program; and the processing unit is further configured to perform the following steps:

[0033] If the application switches from the foreground to the background, an audio playback trigger event is generated;

[0034] Alternatively, if the application is switched from running in the background to running in the foreground, a video playback trigger event is generated.

[0035] In one implementation, the playback interface is a service interface of an application program, and the application program includes a narration audio playback switch; the processing unit is further configured to perform the following steps:

[0036] When the narration audio playback switch is turned on, the narration audio of the video will be played during the playback of the content audio.

[0037] In the embodiment of the present application, in response to the audio playback triggering event, the image of the video can be hidden, the content audio corresponding to the video can be played, and the narration audio of the video can be played during the playback of the content audio. The narration audio is generated based on the recognition of the plot content of the video, so the narration audio can prompt the plot content of the video. Playing the narration audio during the playback of the content audio can help users have a richer and deeper understanding of the plot content, effectively improving the audio playback effect.

[0038] On the other hand, an embodiment of the present application provides a playback processing device, the playback processing device comprising:

[0039] An acquisition unit, used for acquiring a video to be processed;

[0040] A processing unit, used to identify the plot content of the video and obtain key information of the plot content;

[0041] The processing unit is also used to generate narration audio for the video according to the key information. The narration audio is used to be played during the process of playing the content audio corresponding to the video when the image screen of the video is hidden.

[0042] In one implementation, a video includes a plurality of video frames arranged in sequence, and the jth frame is any video frame in the video; the processing unit is used to identify the plot content of the video, and when obtaining key information of the plot content, specifically to perform the following steps:

[0043] Perform scene recognition on the image picture corresponding to the jth frame and the image picture corresponding to the j+1th frame of the video respectively;

[0044] Compare the scene change values ​​between the jth frame and the j+1th frame;

[0045] If the scene change value is greater than the change threshold, the i-th playback time point corresponding to the j+1-th frame in the video and the scene label of the j+1-th frame obtained by scene recognition are recorded as a key information. The recorded key information is used to indicate the scene switching information occurring at the i-th playback time point in the video.

[0046] In one implementation, a video includes a plurality of video frames arranged in sequence, and the j+1th frame is any video frame in the video; the processing unit is used to identify the plot content of the video, and when obtaining key information of the plot content, specifically to perform the following steps:

[0047] Perform face recognition on the image corresponding to the j+1th frame of the video to obtain one or more character facial features and the facial coordinate area corresponding to each character facial feature;

[0048] Obtain a face template set, the face template set including multiple template face features and a role object label corresponding to each template face feature;

[0049] If the facial features of the target character obtained by face recognition match the target template facial features included in the face template set, the i-th playback time point corresponding to the j+1-th frame in the video, the character object label corresponding to the target template facial features, and the face coordinate area corresponding to the target character facial features are recorded as a key information. The recorded key information is used to represent a character object that appears in the video at the i-th playback time point.

[0050] In one implementation, a video includes a plurality of video frames arranged in sequence; a processing unit is used to identify the plot content of the video, and when obtaining key information of the plot content, specifically performs the following steps:

[0051] Performing action recognition on the image pictures of multiple video frames contained in the video frame by frame to obtain one or more character action features, the face coordinate area corresponding to each character action feature, and the duration range of each character action feature;

[0052] Acquire an action template set, wherein the action template set includes a plurality of template action features and an action label corresponding to each template action feature;

[0053] If the identified target character action feature matches the target template action feature included in the action template set, the action label corresponding to the target template action feature, the facial coordinate area corresponding to the target character action feature, and the duration range corresponding to the target character action feature are recorded as a key information. The recorded key information is used to represent the action information performed by the character object appearing in the video within a duration range.

[0054] In one implementation, the key information includes a scene tag of the video at the i-th playback time point; when the processing unit is used to generate the narration audio of the video according to the key information, it is specifically used to perform the following steps:

[0055] The scene label of the i-th playback time point is used as a plot label;

[0056] The plot tag is converted into an audio segment corresponding to the i-th playback time point, and the narration audio includes the audio segment corresponding to the i-th playback time point.

[0057] In one implementation, the key information includes a scene tag of the video at the i-th playback time point and a character object tag of the video at the i-th playback time point; when the processing unit is used to generate the narration audio of the video according to the key information, it is specifically used to perform the following steps:

[0058] The scene label at the i-th playback time point and the character object label at the i-th playback time point are concatenated into a plot label;

[0059] The plot label is converted into an audio clip corresponding to the i-th playback time point, and the narration audio includes the audio clip corresponding to the i-th playback time point.

[0060] In one implementation, the key information includes: a scene label of the video at the i-th playback time point, a character object label of the video at the i-th playback time point, a face coordinate region corresponding to the character object label at the i-th playback time point, an action label of the video within a target duration range, and a face coordinate region corresponding to the action label within the target duration range; wherein the i-th playback time point is within the target duration range, and the face coordinate region corresponding to the character object label at the i-th playback time point matches the face coordinate region corresponding to the action label within the target duration range;

[0061] When the processing unit is used to generate the narration audio of the video according to the key information, it is specifically used to perform the following steps:

[0062] The scene label at the i-th playback time point, the character object label at the i-th playback time point, and the action label of the target duration range are concatenated into a plot label;

[0063] The plot tag is converted into an audio segment corresponding to the i-th playback time point, and the narration audio includes the audio segment corresponding to the i-th playback time point.

[0064] In the embodiment of the present application, the plot content of the video can be identified to obtain key information of the plot content, and the narration audio of the video can be generated according to the key information of the plot content; the narration audio can be used to be played in the process of playing the content audio corresponding to the video when the image screen of the video is hidden. The narration audio is generated according to the key information obtained by identifying the plot content, so the narration audio can prompt the key information in the plot content of the video. When the image screen of the video is hidden, playing the narration audio in the process of playing the content audio can help users have a richer and deeper understanding of the plot content, and effectively improve the audio playback effect.

[0065] On the other hand, an embodiment of the present application provides a playback processing device, the playback processing device comprising:

[0066] A processor, adapted to implement a computer program; and a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program is adapted to be loaded by the processor and execute the above-mentioned playback processing method.

[0067] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is read and executed by a processor of a computer device, the computer device executes the above-mentioned playback processing method.

[0068] On the other hand, an embodiment of the present application provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above-mentioned playback processing method.

[0069] In an embodiment of the present application, when the image screen of the video is hidden, the narration audio of the video can be played during the process of playing the content audio corresponding to the video; the narration audio is generated by identifying the plot content of the video, so the narration audio can prompt the plot content of the video, and playing the narration audio during the process of playing the content audio can help users have a richer and deeper understanding of the plot content, and effectively improve the audio playback effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0071] Figure 1 A schematic diagram of the architecture of a playback processing system provided by an exemplary embodiment of the present application is shown;

[0072] Figure 2 A flowchart of a playback processing solution provided by an exemplary embodiment of the present application is shown;

[0073] Figure 3 A flowchart of a playback processing method provided by an exemplary embodiment of the present application is shown;

[0074] Figure 4a A schematic diagram of an interface of a playback processing method provided by an exemplary embodiment of the present application is shown;

[0075] Figure 4bA schematic diagram of an interface of a playback processing method provided by another exemplary embodiment of the present application is shown;

[0076] Figure 4c A schematic diagram of an interface of a playback processing method provided by another exemplary embodiment of the present application is shown;

[0077] Figure 4d A schematic diagram of an interface of a playback processing method provided by another exemplary embodiment of the present application is shown;

[0078] Figure 4e A schematic diagram showing a flow chart of a method for playing narration audio provided by an exemplary embodiment of the present application;

[0079] Figure 4f A schematic diagram of a narration audio playback switch provided by an exemplary embodiment of the present application is shown;

[0080] Figure 5 A flowchart of a playback processing method provided by another exemplary embodiment of the present application is shown;

[0081] Figure 6a A schematic diagram of a process of identifying a role object provided by an exemplary embodiment of the present application is shown;

[0082] Figure 6b A schematic flow chart of a role object identification process provided by another exemplary embodiment of the present application is shown;

[0083] Figure 7 A schematic diagram of a process flow of an action recognition process provided by an exemplary embodiment of the present application is shown;

[0084] Figure 8 A schematic diagram of the structure of a playback processing device provided by an exemplary embodiment of the present application is shown;

[0085] Fig. 9 A schematic structural diagram of a playback processing device provided by another exemplary embodiment of the present application is shown;

[0086] Fig.10 A schematic structural diagram of a playback processing device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0087] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0088] When it is inconvenient for users to watch videos (such as TV series, movies, variety shows, etc.), they can choose to play the video in the background, or choose to play the content audio corresponding to the video. Content audio refers to the synchronous audio of the video, that is, the audio that is played synchronously during the playback of the video is the content audio corresponding to the video. The content audio may include character dialogues, background music, environmental sound effects (such as wind, water, etc.), etc. in the video. By playing the content audio corresponding to the video, you can have a general understanding of the plot of the video. Among them, background playback refers to hiding the player (such as an application) that is currently playing the video, but the functions of the player are still running; when playing in the background, the player's playback interface is closed (or hidden) and is invisible to the user, but all functions of the player are running normally; therefore, when playing a video in the background, the image of the video is hidden, and the user cannot see the image of the video, but can hear the content audio corresponding to the video; for example, when pressing the Home button of the mobile phone during the playback of the video (the Home button refers to the control that can trigger the display of the main interface of the mobile phone), the image of the video is hidden, and the main interface of the mobile phone is displayed, which is not visible to the user. For example, the user sees the main interface of the phone, but can hear the audio and content corresponding to the video. This is background video playback. For another example, when the screen-off button of the phone is pressed during video playback (the screen-off button refers to a control that can be used to control the phone to enter or cancel the sleep state), the phone turns off the screen and enters the sleep state, and the image of the video is hidden. For the user, what he sees is the screen of the phone after the screen is turned off, but he can hear the audio and content corresponding to the video. This is also background video playback. For another example, when the "Minimize" button is selected in the video playback interface, the video playback interface is minimized and the video image is hidden, and only the audio and content corresponding to the video are output. This is also background video playback. Among them, choosing to play the content audio corresponding to the video may refer to: directly hiding the image of the video in the video playback interface through special operations, and only playing the content audio corresponding to the video, and other interface elements in the video playback interface (such as video title information, video introduction information and other interface elements) may maintain their original display state unchanged; for example, an audio playback option is provided in the video playback interface. During the video playback process, if the user operates the audio playback option, indicating that the user chooses to play the content audio corresponding to the video, then the image of the video is hidden in the playback interface, and only the content audio corresponding to the video is played in the playback interface.

[0089] In the process of playing the content audio corresponding to the video, due to the lack of the prompting effect of the video image screen, the user cannot know some key information in the plot content of the video, which is not convenient for the user to understand the plot content. Based on this, the embodiment of the present application provides a playback processing scheme, which can identify the plot content of the video and generate the narration audio of the video; the narration audio can refer to the audio that can prompt the key information in the plot content of the video, and the content audio corresponding to the video cannot prompt the key information of the plot content, and the key information plays a very important prompting role in the understanding of the plot content of the video; the key information of the plot content may include at least one of the following: scene switching information, character object, and action information performed by the character object; the scene switching information can be used to indicate that at a certain playback time point of the video, the image screen of the video switches from one scene (for example, it can be "snow") to another scene (for example, it can be "lake"); the character object can be used to indicate the character object that appears at a certain playback time point of the video (for example, "Zhang XX"); the action information performed by the character object can be used to indicate the action performed by the character object at a certain playback time point of the video (for example, "Zhang XX running"). During the playback of content audio, add the playback of narration audio. This can help users have a richer and deeper understanding of the plot content and effectively improve the audio playback effect.

[0090] Combine the following Figure 1 The playback processing system shown introduces the playback processing solution provided in the embodiment of the present application. Figure 1 The schematic diagram of the architecture of a playback processing system provided by an exemplary embodiment of the present application is shown. The playback processing system 10 may include a terminal 101 and a server 102. Among them, the terminal 101 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a vehicle-mounted device, a smart speaker, a smart watch, etc., but is not limited thereto. The server 102 may be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network) services, and big data and artificial intelligence platforms. The terminal 101 and the server 102 may be directly or indirectly connected via wired communication or wireless communication, and the present application does not limit this.

[0091] (1) The playback process of the narration audio is performed by the terminal 101 , and the recognition process of the narration audio is performed by the server 102 .

[0092] A video-on-demand application may be run in the terminal, and the server may be a data processing server of the video-on-demand application. When a user requests a video, the video-on-demand application may send a video-on-demand request to the server, and the video-on-demand request may carry the video identifier of the requested video (e.g., video ID (Identity document, identification number), etc.); the server may obtain the video according to the video identifier of the requested video, and transmit the obtained video to the video-on-demand application for playback. In response to an audio playback triggering event, the video-on-demand application may send an audio playback request to the server, and the audio playback request may include the video identifier of the requested video. The server may obtain the video and the content audio corresponding to the video according to the video identifier of the requested video, and identify the plot content of the obtained video to obtain key information of the plot content; the server may also generate narration audio according to the key information of the plot content, and send the content audio and narration audio to the video-on-demand application, so that the video-on-demand application can play the narration audio during the playback of the content audio.

[0093] It should be noted that the server 102 mentioned in the embodiment of the present application can be a server cluster consisting of a video background server, a voice conversion server and a plot content recognition server. The data interaction process between the video on demand application, the video background server, the voice conversion server and the plot content recognition server can be seen in Figure 2 , Figure 2 A flowchart of a playback processing solution provided by an exemplary embodiment of the present application is shown. Figure 2 The playback processing scheme shown may include the following ① to ⑧:

[0094] ① The user performs an audio playback trigger operation. The audio playback trigger operation may include any of the following: performing gravity sensing operations such as shaking the terminal, rotating the terminal, and flipping the terminal while the video on demand application displays the video playback interface; or selecting the audio playback option in the video playback interface; or switching the video on demand application from the foreground to the background, such as selecting the Home button while the video on demand application displays the video playback interface, selecting the screen off button while the video on demand application displays the video playback interface, selecting the "minimize" button in the video playback interface, etc.

[0095] ② The video-on-demand application generates an audio playback request according to the user's audio playback trigger operation, and sends the audio playback request to the video background server. Specifically, the video-on-demand application generates an audio playback trigger event in response to the user's audio playback trigger operation, generates an audio playback request, and then sends the audio playback request to the video background server; the audio playback request may include the video identifier of the requested video.

[0096] ③ The video background server responds to the audio playback request, obtains the content audio corresponding to the video, and sends the audio playback request to the plot content recognition server. Specifically, the video background server can respond to the audio playback request, obtain the content audio corresponding to the video according to the video identifier of the requested video included in the audio playback request; then the video background server can also send the audio playback request to the plot content recognition server, requesting the plot content recognition server to identify the plot content of the requested video and obtain key information of the plot content.

[0097] ④ The plot content recognition server responds to the audio playback request, obtains the video according to the audio playback request, and recognizes the plot content of the video to obtain key information of the plot content. Specifically, the plot content recognition server can respond to the audio playback request, obtain the video according to the video identifier of the requested video included in the audio playback request, and recognize the plot content of the video to obtain at least one key information of the plot content.

[0098] The plot content recognition server can also combine and splice each key information to obtain one or more plot tags, each plot tag corresponding to a play time point of the content audio. Then, the plot content recognition server can send each plot tag and the play time point corresponding to each plot tag to the video background server.

[0099] ⑤ The video background server sends the content audio, plot tags and the play time points corresponding to the plot tags corresponding to the video to the video on demand application. Specifically, the video background server can send the content audio corresponding to the video, the plot tags obtained by splicing and combining the key information and the play time points corresponding to each plot tag to the video on demand application.

[0100] ⑥ The video-on-demand application sends a voice conversion request to the voice conversion server. Specifically, after the video-on-demand application receives the content audio corresponding to the video sent by the video background server, the various plot tags obtained by splicing and combining the key information, and the playback time point corresponding to each plot tag, the video-on-demand application can play the content audio. During the process of playing the content audio, when the content audio is played to the playback time point corresponding to the target plot tag, the video-on-demand application can pause the content audio. The target plot tag is any plot tag received by the video-on-demand application. Then, the video-on-demand application sends a voice conversion request to the voice conversion server, and the voice conversion request can include the target plot tag.

[0101] ⑦ The voice conversion server responds to the voice conversion request, performs voice conversion on the target plot tag, and obtains an audio segment corresponding to the target plot tag. Specifically, the voice conversion server can respond to the voice conversion request, perform voice conversion on the target plot tag, and obtain an audio segment corresponding to the target plot tag; the narration audio includes the audio segment obtained by converting each plot tag; then, the voice conversion server sends the audio segment obtained by converting the target plot tag to the video on demand application.

[0102] ⑧ The video-on-demand application starts playing the audio clip corresponding to the target plot tag from the playback time point corresponding to the target plot tag. Specifically, the video-on-demand application can start playing the audio clip corresponding to the target plot tag from the playback time point corresponding to the target plot tag, and after the audio clip corresponding to the target plot tag is played, continue playing the content audio from the playback time point corresponding to the target plot tag.

[0103] It should be noted that when the content audio is played to the playback time point corresponding to the next plot tag, the above ⑥ to ⑧ are repeated; the next plot tag is a plot tag whose playback time point is located after the playback time point corresponding to the target plot tag among the plot tags obtained by splicing and combining. That is to say, in the process of playing the content audio, when the content audio is played to the playback time point corresponding to the next plot tag, the video on demand application pauses playing the content audio; then, the video on demand application sends a voice conversion request to the voice conversion server, and the voice conversion request may include the next plot tag. The voice conversion server can respond to the voice conversion request, perform voice conversion on the next plot tag, and obtain the audio clip corresponding to the next plot tag; then, the voice conversion server can send the audio clip corresponding to the next plot tag to the video on demand application. The video on demand application can start playing the audio clip corresponding to the next plot tag from the playback time point corresponding to the next plot tag, and after the audio clip corresponding to the next plot tag is played, continue playing the content audio from the playback time point corresponding to the next plot tag.

[0104] (2) The narration audio playback process and the narration audio recognition process are both executed by the terminal 101.

[0105] When the playing process of the narration audio and the recognition process of the narration audio are both executed by the terminal 101, the video-on-demand application has the functions of plot content recognition and voice conversion. During the process of playing a video in the playback interface of the video application, in response to an audio playback trigger event, the video-on-demand application can obtain the content audio corresponding to the video and play the content audio corresponding to the video; the video-on-demand application can also identify the plot content of the video, obtain at least one key information of the plot content and the playback time point corresponding to each key information; then, the video-on-demand application can splice and combine the key information to obtain one or more plot tags, each plot tag corresponding to a playback time point of the content audio; the video-on-demand application can perform voice conversion on each plot tag respectively to obtain the audio clip corresponding to each plot tag, and the narration audio is composed of the audio clips corresponding to each plot tag; when the content audio playback reaches the playback time point corresponding to the target plot tag, the video-on-demand application pauses the playback of the content audio and starts to play the audio clip corresponding to the target plot tag; after the audio clip corresponding to the target plot tag is played, the video-on-demand application can continue to play the content audio from the playback time point corresponding to the target plot tag; wherein the target plot tag is any plot tag obtained by splicing and combining.

[0106] In an embodiment of the present application, a plot content recognition server can recognize the plot content of a video and obtain key information corresponding to the plot content; a voice conversion server can perform voice conversion on the plot tags obtained by splicing and combining, and obtain an audio segment corresponding to the plot tag in the narration audio; thus, through a server cluster consisting of a video background server, a voice conversion server, and a plot content recognition server, a narration audio for a video can be generated, and the narration audio can prompt the plot content of the video; thus, the video-on-demand application plays the narration audio of the video during the process of playing the content audio, which can help users have a richer and deeper understanding of the plot content and improve the audio playback effect.

[0107] It can be understood that the playback processing system described in the embodiment of the present application is to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0108] It should be noted that the playback processing solution provided in the embodiment of the present application involves artificial intelligence voice technology, natural language processing technology, computer vision technology, machine learning technology, etc. Specifically, the process of converting key information into audio clips involves artificial intelligence voice technology; the process of identifying the plot content of the video to obtain key information involves artificial intelligence computer vision technology, machine learning technology, and natural language processing technology. Among them:

[0109] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level technology and software-level technology. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0110] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further processing graphics so that computer processing becomes an image that is more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D (3-Dimension) technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0111] The key technologies of speech technology include automatic speech recognition (ASR), text to speech (TTS) and voiceprint recognition. Enabling computers to listen, see, speak and feel is the future development direction of human-computer interaction, among which speech has become one of the most promising human-computer interaction methods in the future.

[0112] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, information extraction, text mining, information retrieval, machine translation, robot question answering, dialogue system, knowledge graph and other technologies.

[0113] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0114] The playback processing solution provided in the embodiment of the present application also involves cloud computing, artificial intelligence cloud services and other technologies. Specifically, the process of distinguishing the plot content of the video to obtain key information requires powerful computing power, and the computing service provided by cloud computing can be used; in addition, the process of distinguishing the plot content of the video to obtain key information can also use the service support provided by artificial intelligence cloud services. Among them:

[0115] Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing. Cloud technology refers to the general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model application, which can form a resource pool and be used on demand, flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0116] Cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, expanded at any time, and paid for by use. As a basic capability provider of cloud computing, a cloud computing resource pool (referred to as cloud platform, generally referred to as IaaS (Infrastructure as a Service)) platform will be established, and various types of virtual resources will be deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices. According to the logical function division, the PaaS (Platform as a Service) layer can be deployed on the IaaS layer, and the SaaS (Software as a Service) layer can be deployed on the PaaS layer. SaaS can also be deployed directly on IaaS. PaaS is a platform for software operation, such as databases, web containers, etc. SaaS is a variety of business software, such as web portals, SMS mass senders, etc. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0117] Cloud computing refers to the delivery and use model of IT (Internet Technology) infrastructure, which means obtaining required resources through the network in an on-demand and easily scalable manner; in a broad sense, cloud computing refers to the delivery and use model of services, which means obtaining required services through the network in an on-demand and easily scalable manner. This service can be related to IT and software, the Internet, or other services. Cloud computing is the product of the development and integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing. With the development of the Internet, real-time data streams, and the diversification of connected devices, as well as the promotion of search services, social networks, mobile commerce, and open collaboration, cloud computing has developed rapidly. Different from the previous parallel distributed computing, the emergence of cloud computing will promote revolutionary changes in the entire Internet model and enterprise management model in terms of concept.

[0118] Artificial intelligence cloud service is generally referred to as AIaaS (AI as a Service). This is the service mode of a mainstream artificial intelligence platform. Specifically, the AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service model is similar to opening an AI theme mall: all developers can access and use one or more artificial intelligence services provided by the platform through API interfaces. Some senior developers can also use the AI ​​framework and AI infrastructure provided by the platform to deploy and operate their own cloud artificial intelligence services.

[0119] The playback processing scheme of the embodiment of the present application also involves blockchain technology. Blockchain is a new application mode of computer technologies such as distributed data storage, P2P (Peer to Peer) transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain uses cryptography to ensure that data cannot be tampered with or forged. For example, terminals, video background servers, voice conversion servers, plot content recognition servers, etc. can all be nodes in the blockchain network. The video identification of the video, the content audio corresponding to the video, key information, plot tags, and audio clips corresponding to the plot tags involved in the embodiments of the present application can be stored in the blockchain network in the form of blocks; when the same video is on demand multiple times and the audio is requested to be played multiple times, the video identification associated with the video identification, the content audio corresponding to the video, key information, plot tags, and audio clips corresponding to the plot tags can be directly obtained, and audio playback can be quickly achieved to improve the efficiency of audio playback; in addition, the video identification of the video, the content audio corresponding to the video, key information, plot tags, and audio clips corresponding to the plot tags are transmitted in the blockchain network. Based on the characteristics that the blocks in the blockchain cannot be tampered with or forged, the transmission process is safer and more reliable.

[0120] See also Figure 3 , Figure 3 A flowchart of a playback processing method provided by an exemplary embodiment of the present application is shown. The playback processing method can be executed by a playback processing device provided by an embodiment of the present application, and the playback processing device can be Figure 1 In the terminal 101 of the playback processing system shown, the playback processing method may include but is not limited to the following steps S301 to S304:

[0121] S301, displaying a video playback interface.

[0122] The video playback interface is displayed, and the video playback interface may include an image screen of the video. Figure 4a A schematic diagram of an interface of a playback processing method provided by an exemplary embodiment of the present application is shown. Figure 4a As shown, the video playback interface 40 displays a video image 401 (eg Figure 4a The image shown includes a dog, a cow, a big tree, etc.).

[0123] S302, in response to the audio playback triggering event, hiding the video image.

[0124] S303, play the content audio corresponding to the video.

[0125] In step S302 to step S303, in response to the audio playback triggering event, the video image can be hidden and the content audio corresponding to the video can be played. Figure 4a In response to the audio playback trigger event, the video image can be hidden, and the content audio corresponding to the video can be played from the second progress. Figure 4a "20:00" shown), in response to the video playback trigger event, the image screen of the video can be restored from the first progress in the playback interface. It should be noted that the playback progress of the video is synchronized with the playback progress of the content audio, so the first progress and the second progress can both be used to describe the playback progress of the video, and can also be used to describe the playback progress of the content audio; in the embodiment of the present application, the numerical value corresponding to the first progress is greater than the numerical value corresponding to the second progress. Among them, taking the playback progress of the video as an example, the playback progress of the video refers to the proportion of the content that has been played in the video relative to the total content; when the video is not played, its playback progress is 0%; when the video is played completely, its playback progress is 100%; and when the video is not played completely, its playback progress is a numerical value between the interval (0%, 100%). For example, assuming that the playback time of the entire content of the video is 40 minutes, and the video is currently played to the 10th minute of content, the current playback progress of the video is 10 / 40=25%.

[0126] In the embodiment of the present application, the audio playback triggering event may include but is not limited to any of the following:

[0127] (1) Selecting an audio playback option generates an audio playback trigger event. Specifically, the playback interface may include an audio playback option; if the audio playback option is selected, an audio playback trigger event may be generated. Figure 4a As shown, the playback interface 40 includes an audio playback option 402. When the audio playback option 402 is selected, an audio playback trigger event may be generated. In this manner, hiding the video image may mean: not displaying the video image in the playback interface, and displaying an audio playback prompt in the playback interface. The audio playback prompt is used to prompt the audio corresponding to the currently playing video, such as Figure 4a As shown, the audio play prompt 403 is displayed in the play interface 40 .

[0128] (2) An audio playback trigger event is generated in response to a gravity sensing operation. Specifically, in the process of displaying the video playback interface, an audio playback trigger event can be generated in response to a first gravity sensing operation. The first gravity sensing operation may refer to an operation that causes the gravity of the terminal to change. After the gravity sensor of the terminal senses the change in the gravity of the terminal, it completes the conversion from the gravity change to the electrical signal, thereby generating an audio playback trigger event; the first gravity sensing operation includes any one of the following: a shaking operation, a rotating operation, a flipping operation, etc.; a shaking operation may refer to an operation of quickly shaking the terminal; a rotating operation may refer to an operation of rotating the terminal in a rotation direction, and the rotation direction may include from left to right, from right to left, from front to back, from top to bottom, etc., and the rotation angle of one rotation is generally less than 180 degrees; a flipping operation may refer to an operation of flipping the terminal in a flipping direction, and the flipping direction may include from left to right, from right to left, from front to back, from top to bottom, etc., and the flipping angle of one flip is generally 180 degrees. Figure 4b A schematic diagram of an interface of a playback processing method provided by another exemplary embodiment of the present application is shown. Figure 4b As shown, during the process of displaying the video playback interface 40, the terminal user can generate an audio playback trigger event by shaking the terminal. Figure 4c A schematic diagram of a rotation operation provided by another exemplary embodiment of the present application is shown. Figure 4c As shown, during the process of displaying the video playback interface 40, the terminal user can generate an audio playback trigger event by flipping the terminal. In this manner, hiding the video image may mean: not displaying the video image in the playback interface, and displaying an audio playback prompt in the playback interface, the audio playback prompt is used to prompt the audio corresponding to the currently playing video, such as Figure 4b or Figure 4c As shown, the audio play prompt 403 is displayed in the play interface 40 .

[0129] (3) An application (for example, the video-on-demand application mentioned above) switches from the foreground to the background, generating an audio playback trigger event. At this time, the playback interface is the service interface of the application; in the process of displaying the video playback interface in the application, if the application switches from the foreground to the background, an audio playback trigger event is generated. For example, in the process of displaying the video playback interface in the application, if the Home button is selected, the application switches from the foreground to the background, generating an audio playback trigger event; for another example, in the process of displaying the video playback interface in the application, if the screen-off button is selected, the application switches from the foreground to the background, generating an audio playback trigger event; for another example, in the process of displaying the video playback interface in the application, if the "minimize" button is selected, the application switches from the foreground to the background, generating an audio playback trigger event. When the Home button is selected, hiding the video image screen may mean: not displaying the video playback interface, and displaying the terminal's main interface; Figure 4d A schematic diagram of an interface of a playback processing method provided by another exemplary embodiment of the present application is shown. Figure 4d As shown, the video playback interface 40 includes a Home button 404. During the process of displaying the video playback interface 40 in the application, if the Home button 404 is selected, the main interface 41 of the terminal is displayed.

[0130] The above three audio playback triggering event triggering methods are only examples and do not constitute a limitation of this application. Users can flexibly select an applicable method to trigger the generation of an audio playback triggering event, thereby improving audio playback efficiency and audio playback effects, and meeting the user's needs to switch from video to audio playback.

[0131] In the embodiment of the present application, the video playback triggering event may include but is not limited to any of the following:

[0132] (1) Selecting a video play option generates an audio play trigger event. Specifically, the video play interface may include a video play option; if the video play option is selected, a video play trigger event is generated. Figure 4a As shown, the playback interface 40 includes an audio playback option 405. When the content audio is played to the first progress, if the video playback option 405 is selected and a video playback trigger event is generated, the video image screen 401 can be restored to display from the first progress in the playback interface 40.

[0133] (2) A video playback trigger event is generated in response to a gravity sensing operation. Specifically, during the playback of content audio or narration audio, a video playback trigger event may be generated in response to a second gravity sensing operation. The second gravity sensing operation may refer to an operation that causes a change in the gravity of the terminal. After the gravity sensor of the terminal senses the change in the gravity of the terminal, it completes the conversion from the gravity change to an electrical signal, thereby generating an audio playback trigger event. The second gravity sensing operation includes at least one of the following: a shaking operation, a rotating operation, a flipping operation, and the like. A shaking operation may refer to an operation of quickly shaking the terminal. A rotating operation may refer to an operation of rotating the terminal in a rotation direction. The rotation direction may include from left to right, from right to left, from front to back, from top to bottom, and the like. The rotation angle of one rotation is generally less than 180 degrees. A flipping operation may refer to an operation of flipping the terminal in a flipping direction. The flipping direction may include from left to right, from right to left, from front to back, from top to bottom, and the like. The flipping angle of one flip is generally 180 degrees. Figure 4b As shown, when the content audio is played to the first progress, if the terminal user generates a video playback trigger event by shaking the terminal, the image screen 401 of the video can be restored from the first progress in the playback interface 40. Figure 4c As shown, when the content audio is played to the first progress, if the terminal user generates a video play trigger event by flipping the terminal, the image screen 401 of the video can be restored from the first progress in the play interface 40 .

[0134] (3) An event in which an application (for example, the video-on-demand application described above) switches from background operation to foreground operation. At this time, the playback interface is the service interface of the application; when the content audio is played to the first progress, if the application switches from background operation to foreground operation, a video playback trigger event is generated. Figure 4d As shown, when the content audio is played to the first progress, if the icon 406 of the application displayed in the main interface 41 is selected, the application is switched from the background operation to the foreground operation, a video playback trigger event is generated, and the image screen 401 of the video can be restored from the first progress in the playback interface 40. For another example, when the content audio is played to the first progress, if the screen-off key is selected, the application is switched from the background operation to the foreground operation, an audio playback trigger event is generated, and the image screen of the video can be restored from the first progress in the playback interface.

[0135] The above three video playback triggering event triggering methods are only examples and do not constitute a limitation of the present application. Users can flexibly select an applicable method to trigger a video playback triggering event, so that audio playback can be conveniently switched to video playback, meeting the user's need to switch audio to video playback.

[0136] It should be noted that the above-mentioned various audio playback trigger events and various video playback trigger events can be used in combination flexibly; for example, when an audio playback trigger event is generated by clicking an audio playback option, it is not necessary to click a video playback option to generate a video playback trigger event. A video playback trigger event can also be generated by performing a gravity sensing operation on the terminal, and the embodiments of the present application do not limit this. The first gravity sensing operation and the second gravity sensing operation can be the same, for example, the first gravity sensing operation and the second gravity sensing operation are both shaking operations; the first gravity sensing operation and the second gravity sensing operation can also be different, for example, the first gravity sensing operation is a shaking operation, and the second gravity sensing operation is a flipping operation.

[0137] S304: During the process of playing the content audio, play the narration audio of the video.

[0138] The narration audio is generated based on the recognition of the plot content of the video, and the narration audio can be used to prompt the plot content of the video. The narration audio may include N audio clips, each of which corresponds to a playback time point of the content audio; any audio clip among the N audio clips is represented as the i-th audio clip, and the i-th audio clip corresponds to the i-th playback time point of the content audio; wherein i and N are both positive integers and i≤N. For the playback method of the video narration audio, please refer to Figure 4e , Figure 4e A flowchart of a method for playing narration audio provided by an exemplary embodiment of the present application is shown. Figure 4e As shown, in response to an audio playback trigger event, the content audio corresponding to the video is played; during the playback of the content audio, when the content audio is played to the i-th playback time point, the content audio can be paused and the i-th audio segment can be played; after the i-th audio segment is played, the content audio can continue to be played; continuing to play the content audio can refer to continuing to play the content audio from the i-th playback time point.

[0139] The narration audio can be used to prompt the key information of the plot content of the video; wherein the i-th audio clip can be used to prompt at least one of the following key information: the scene switching information of the video at the i-th playback time point, the character object that appears in the video at the i-th playback time point, and the action information performed by the character object that appears in the video at the i-th playback time point. The video may include a plurality of video frames arranged in sequence, and the scene switching information of the video at the i-th playback time point may refer to: the scene switching occurs from the image screen of the video frame corresponding to the i-1-th playback time point to the image screen of the video frame corresponding to the i-th playback time point, and the scene label of the video frame corresponding to the i-th playback time point may be selected as the scene switching information of the video at the i-th playback time point. The character object that appears in the video at the i-th playback time point may refer to: the character object included in the image screen of the video frame corresponding to the i-th playback time point. The action information performed by the character object that appears in the video at the i-th playback time point may refer to: the action information performed by the character object included in the image screen of the video frame corresponding to the i-th playback time point.

[0140] In one implementation, the i-th audio clip can be used to prompt the scene switching information occurring at the i-th playback time point of the video; for example, if the scene switching information occurring at the i-th playback time point is "at the bow", then the key information prompted by the i-th audio clip may be "at the bow".

[0141] In another implementation, the i-th audio clip can be used to prompt the scene switching information occurring at the i-th playback time point of the video and the character object appearing at the i-th playback time point; and the prompting order of the scene switching information occurring at the i-th playback time point and the character object appearing at the i-th playback time point of the video can be that the character object appearing at the i-th playback time point of the video comes first, and the scene switching information occurring at the i-th playback time point of the video comes later; for example, the character object appearing in the video at the i-th playback time point is "Jack", and the scene switching information occurring at the i-th playback time point of the video is "at the bow", then the key information prompted by the i-th audio clip can be "Jack is at the bow".

[0142] In another implementation, the i-th audio clip can be used to prompt the scene switching information occurring at the i-th playback time point of the video, the character object appearing at the i-th playback time point of the video, and the action information performed by the character object appearing at the i-th playback time point of the video; and the prompting order of the scene switching information occurring at the i-th playback time point, the character object appearing at the i-th playback time point of the video, and the action information performed by the character object appearing at the i-th playback time point of the video can be the character object appearing at the i-th playback time point first, followed by the scene switching information occurring at the i-th playback time point of the video, and then the action information performed by the character object appearing at the i-th playback time point of the video; for example, the character object appearing in the video at the i-th playback time point is "Jack", the scene switching information occurring at the i-th playback time point of the video is "at the bow", and the action information performed by the character object appearing at the i-th playback time point of the video is "hug", then the key information prompted by the i-th audio clip can be "Jack hugs at the bow".

[0143] The video playback interface is a service interface of the application. The application may also include a narration audio playback switch. For example, the video playback interface may include a narration audio playback switch. When the narration audio playback switch is turned on, the video narration audio can be played during the playback of the content audio. During the playback of the i-th audio clip, in response to the narration audio playback switch being triggered to be turned off, the playback of the i-th audio clip is stopped, and the content audio continues to be played. Continuing to play the content audio may mean that the content audio is played continuously from the i-th playback time point. Figure 4f A schematic diagram of a narration audio playback switch provided by an exemplary embodiment of the present application is shown. Figure 4f As shown, the video playback interface 40 may include a narration audio playback switch 407; in response to an audio playback trigger event, if the narration audio playback switch 407 is in the on state, the video narration audio can be played during the playback of the content audio. During the playback of the i-th audio clip, in response to the narration audio playback switch 407 being triggered to close, the playback of the i-th audio clip is stopped, and the content audio continues to be played. Continuing to play the content audio may mean continuing to play the content audio from the i-th playback time point. In this way, the user of the application can decide to turn on or off the narration audio playback switch according to their needs, thereby turning on or off the narration audio playback function, further improving the audio playback effect, and enhancing the user experience of the application.

[0144] In the embodiment of the present application, in response to the audio playback trigger event, the image screen of the video can be hidden, the content audio corresponding to the video can be played, and the narration audio of the video can be played during the playback of the content audio. The narration audio is generated based on the recognition of the plot content of the video, so the narration audio can prompt the plot content of the video, and playing the narration audio during the playback of the content audio can help users to have a richer and deeper understanding of the plot content, and effectively improve the audio playback effect. In addition, the user can flexibly select the applicable method to trigger the generation of the audio playback trigger event, thereby improving the audio playback efficiency, improving the audio playback effect, and meeting the user's demand for switching the video to audio playback. The user can flexibly select the applicable method to trigger the generation of the video playback trigger event, thereby conveniently switching the audio playback to the video playback, and meeting the user's demand for converting the audio to the video playback. In addition, the user of the application can decide to turn on or off the narration audio playback switch according to the needs, thereby turning on or off the narration audio playback function, further improving the audio playback effect, and improving the user experience of the application.

[0145] See also Figure 5 , Figure 5 A flowchart of a playback processing method provided by another exemplary embodiment of the present application is shown. The playback processing method can be executed by the playback processing device provided by the embodiment of the present application, and the playback processing method may include but is not limited to the following steps S501 to S503:

[0146] S501, obtaining a video to be processed.

[0147] In one implementation, the playback processing device may be Figure 1 The server 102 in the playback processing system shown, namely Figure 5 The playback processing method shown can be executed by the server 102. A video on demand application can be run in the terminal, and the server can be a data processing server of the video on demand application. When a user requests to play the audio corresponding to the video, the video on demand application can send an audio playback request to the server, and the audio playback request can carry the video identifier (such as video ID, etc.) of the requested video; the server can obtain the video to be processed and the content audio corresponding to the video according to the video identifier of the requested video, and identify the video to obtain key information of the plot content; then, the server can generate the narration audio of the video according to the key information, and send the content audio and narration audio to the video on demand application, so that the video on demand application can play the narration audio during the playback of the content audio.

[0148] In another implementation, the playback processing device may be Figure 1 The terminal 101 in the playback processing system shown in FIG. Figure 5 The playback processing method shown is executed by the terminal. A video-on-demand application may be running in the terminal. When a user requests to play the audio corresponding to the video, the video-on-demand application of the terminal can obtain the requested video and perform recognition processing on the video to obtain key information of the plot content; then, the video-on-demand application of the terminal can generate the narration audio of the video according to the key information, so that the video-on-demand application can play the narration audio during the playback of the content audio.

[0149] S502, identifying the plot content of the video to obtain key information of the plot content.

[0150] By identifying the plot content of the video, at least one key information of the plot content can be obtained. The key information may include but is not limited to at least one of the following: scene switching information, character objects, and action information performed by the character objects. The video may contain multiple video frames arranged in sequence, and identifying the plot content of the video may specifically refer to identifying each video frame contained in the video frame by frame.

[0151] In one implementation, the key information may include a character object. When the key information is a character object, the process of identifying the plot content of the video and obtaining the key information of the plot content can be found in Figure 6a The process diagram and Figure 6b The process diagram shown; Figure 6a A schematic diagram of a process of identifying a role object provided by an exemplary embodiment of the present application is shown. Figure 6b FIG. 2 is a flow chart showing a process of identifying a role object provided by another exemplary embodiment of the present application. Figure 6a As shown, the process of identifying the plot content of the video and obtaining key information of the plot content may include the following sub-steps s61 to s64:

[0152] s61, perform face recognition on the image corresponding to the j+1th frame of the video to obtain one or more character facial features and the facial coordinate area corresponding to each character facial feature.

[0153] The j+1th frame is any one of the multiple video frames contained in the video, and j is a non-negative integer. Figure 6bThe detection process on the right side of the flowchart shown, the process of performing face recognition on the image screen corresponding to the j+1th frame of the video may include: performing image preprocessing on the image screen corresponding to the j+1th frame; wherein the image preprocessing may include: performing burr removal processing, contrast stretching processing, dynamic range compression processing, etc. on the image screen corresponding to the j+1th frame; after image preprocessing, irrelevant information in the image screen can be eliminated, useful real information can be restored, and the detectability of the image screen can be effectively enhanced. Then, the face recognition model can be used to perform face recognition and cutting on the image screen corresponding to the j+1th frame after image preprocessing to obtain the face coordinate area in the image screen, and the face coordinate area obtained by cutting may include one or more; the face coordinate area refers to an area in the image screen containing a face, and the face coordinate area can be a rectangular area determined according to the upper left coordinate point and the lower right coordinate point, or the face coordinate area can also be a circular area determined according to the center and radius of the circle, or the face coordinate area can also be an elliptical area determined according to the center, major axis and minor axis, etc., and the embodiment of the present application does not limit this. Furthermore, for any face coordinate region, a face recognition model can be used to perform feature extraction processing on the image block corresponding to the face coordinate region in the image screen to obtain the character face features corresponding to the face coordinate region; feature extraction processing refers to the process of extracting features that can provide effective information and non-redundant information from the initial information contained in the image block, and the face recognition efficiency can be effectively improved through feature extraction processing. After the face recognition model performs face recognition on the image screen corresponding to the j+1 frame, one or more character face features and the face coordinate region corresponding to each character face feature can be obtained.

[0154] The training process of face recognition model and classifier can be found in Figure 6b The training process on the left side of the flowchart shown in the figure can use a sample image set to train the face recognition model and the classifier; wherein the sample image set can include a first image set and a second image set, the first image set can include multiple sample face images, and the second image set can include template face images corresponding to each sample face image in the first image set. Figure 6aAs shown in the training flow on the left, the training process of the face recognition model and the classifier is briefly described as follows: for the target sample face image in the first image set, the target sample face image is any sample face image in the first image set; the target sample face image can be image preprocessed, and the face recognition model can be used to perform face recognition processing on the target sample face image after image preprocessing to obtain the first sample face feature in the target sample face image and the face coordinate area corresponding to the first sample face feature. Then, for the target template face image corresponding to the target sample face image in the second image set, the target template face image can be image preprocessed, and the face recognition model can be used to perform face recognition processing on the target template face image after image preprocessing to obtain the second sample face feature in the target template face image and the face coordinate area corresponding to the second sample face feature. Secondly, the first sample face feature and the second sample face feature can be input into the classifier for feature matching to obtain the matching result between the first sample face feature and the second sample face feature. Then, the model parameters of the classifier can be trained and optimized according to the matching results between the first sample facial features and the second sample facial features; the model parameters of the face recognition model can be trained and optimized according to the difference between the facial coordinate area corresponding to the first sample facial features and the actual facial coordinate area in the target sample facial image; similarly, the model parameters of the face recognition model can be trained and optimized according to the difference between the facial coordinate area corresponding to the second sample facial features and the actual facial coordinate area in the target template facial image. For each sample facial image in the first image set, the face recognition model and the classifier are trained in a similar manner to the target sample facial image, so that the trained face recognition model is used for face recognition and the trained classifier is used for feature matching.

[0155] Among them, Convolution Neural Network (CNN) can be used in the feature extraction process. CNN is an artificial neural network based on deep learning theory. It mainly uses weight sharing to reduce the parameter expansion problem in ordinary neural networks, and uses convolution kernels to perform convolution operations on input data during forward calculations. The result is passed through a nonlinear function as the output of the layer. Such a layer is called a convolution layer. There will be a downsampling layer between the convolution layer and the convolution layer. The downsampling layer is mainly used to obtain the invariance of local features and reduce the scale of the feature space. Generally, after the convolution layer and the downsampling layer, there is a fully connected neural network for final recognition.

[0156] s62, obtain a face image collection.

[0157] The face image collection may include multiple character face images and a character object label corresponding to each character face image; the face image collection may be uploaded by the administrator of the application; a character face image in the face image collection may include a character object, and the included character object may be the main character in the video, and the character object label corresponding to the character face image may be the character name of the main character in the video, such as "Zhang XX", "Li XX", etc.

[0158] s63, performing face recognition processing on the character face images in the face image set to obtain a face template set.

[0159] The face template set includes multiple template face features and the character object label corresponding to each template face feature. Figure 6b The detection process on the right side of the flowchart shown, the process of performing face recognition processing on the character face images in the face image set may include: for any character face image in the face image set, performing image preprocessing on the character face image; using a face recognition model to perform face recognition and segmentation on the character face image after image preprocessing, and performing feature extraction processing, and finally obtaining a face template set including multiple template face features and a character object label corresponding to each template face feature. The face recognition process for any character face image in the face image set is similar to the face recognition process for the image picture corresponding to the j+1th frame. Please refer to the face recognition process for the image picture corresponding to the j+1th frame in the above step s61, which will not be repeated here.

[0160] s64, if the target character's facial features obtained by face recognition match the target template's facial features included in the face template set, the i-th playback time point corresponding to the j+1-th frame in the video, the character object label corresponding to the target template's facial features, and the facial coordinate area corresponding to the target character's facial features are recorded as a key information.

[0161] A classifier can be used to perform feature matching between the target character face features obtained by face recognition and the target template face features included in the face template set, and obtain a matching result between the target character face features and the target template face features; if the matching result between the target character face features and the target template face features indicates that the target character face features obtained by face recognition match the target template face features included in the face template set, then the i-th playback time point corresponding to the j+1th frame in the video, the character object label corresponding to the target template face features, and the face coordinate area corresponding to the target character face features can be recorded as a key information. Among them, the matching of the target character face features and the target template face features indicates that the character object indicated by the target character face features and the character object indicated by the target template face features are highly likely to be the same character object, so the key information recorded by this implementation method can be used to indicate a character object that appears in the video at the i-th playback time point.

[0162] In another implementation, the key information may include scene switching information. When the key information is scene switching information, the process of identifying the plot content of the video and obtaining the key information of the plot content may include the following sub-steps s71 to s73:

[0163] s71, performing scene recognition on the image picture corresponding to the j-th frame and the image picture corresponding to the j+1-th frame of the video respectively.

[0164] The jth frame is any video frame included in the video, the j+1th frame is a video frame arranged after the jth frame among multiple video frames included in the video, and j is a positive integer. A scene recognition model can be used to perform scene recognition on the image screen corresponding to the jth frame and the image screen corresponding to the j+1th frame of the video, respectively, to obtain the scene feature value of the image screen corresponding to the jth frame, and to obtain the scene feature value of the image screen corresponding to the j+1th frame.

[0165] s72, comparing the scene change values ​​between the j-th frame and the j+1-th frame.

[0166] The scene change value between the jth frame and the j+1th frame can be determined according to the scene feature value of the image picture corresponding to the jth frame and the scene feature value of the image picture corresponding to the j+1th frame. The scene change value between the jth frame and the j+1th frame can be used to indicate the scene change between the image picture corresponding to the jth frame and the image picture corresponding to the j+1th frame. The larger the scene change value between the jth frame and the j+1th frame, the larger the scene change amplitude between the image picture corresponding to the jth frame and the image picture corresponding to the j+1th frame; for example, the image picture corresponding to the jth frame includes a mountain, and the image picture corresponding to the j+1th frame includes a car, and the scene change amplitude between the image picture corresponding to the jth frame and the image picture corresponding to the j+1th frame is large. The smaller the scene change value between the jth frame and the j+1th frame, the smaller the scene change amplitude between the image picture corresponding to the jth frame and the image picture corresponding to the j+1th frame. For example, the image picture corresponding to the jth frame includes a mountain, and the image picture corresponding to the j+1th frame also includes a mountain; the scene change amplitude between the image picture corresponding to the jth frame and the image picture corresponding to the j+1th frame is relatively small.

[0167] s73, if the scene change value is greater than the change threshold, the i-th playback time point corresponding to the j+1-th frame in the video and the scene label of the j+1-th frame obtained by scene recognition are recorded as a key information.

[0168] If the scene change value is greater than the change threshold, it indicates that the scene change between the image corresponding to the jth frame and the image corresponding to the j+1th frame is large. The i-th playback time point corresponding to the j+1th frame in the video and the scene label of the j+1th frame obtained by scene recognition can be recorded as a key information. The key information recorded by this implementation method is used to indicate the scene switching information of the video at the i-th playback time point. Among them, the scene label can be the name of the real scenery in the image corresponding to the j+1th frame. For example, the scene label can be "field", "car", "sea" and so on.

[0169] For the case where the key information includes scene switching information, if the scene switching is too frequent, the recorded key information can be filtered. For example, if there is a scene switch between the first frame and the second frame of the video, between the second frame and the third frame, and between the third frame and the fourth frame, the corresponding playback time point of the second frame in the video and the scene label of the second frame can be recorded as the first key information, the corresponding playback time point of the third frame in the video and the scene label of the third frame can be recorded as the second key information, and the corresponding playback time point of the fourth frame in the video and the scene label of the fourth frame can be recorded as the third key information; the time intervals between the above first key information, the second key information and the third key information are relatively short. In this case, a time threshold can be set, and the second key information can only be retained when the time interval between the playback time point in the second key information and the playback time point in the first key information is greater than the time threshold, otherwise the second key information will be filtered.

[0170] In another implementation, the key information may include action information performed by the character object. When the key information is the action information performed by the character object, the process of identifying the plot content of the video and obtaining the key information of the plot content can be referred to in Figure 7 , Figure 7 FIG. 1 is a flow chart of an action recognition process provided by an exemplary embodiment of the present application. Figure 7As shown, an action recognition model can be used to perform action recognition on the image screens of multiple video frames contained in the video frame by frame to obtain one or more character action features, the facial coordinate area corresponding to each character action feature, and the duration range of each character action feature; the duration range refers to a time interval formed by the starting time point and the ending time point. For example, if the starting time point is 3 minutes and 20 seconds and the ending time point is 4 minutes and 30 seconds, the formed time interval [3 minutes and 20 seconds, 4 minutes and 30 seconds] is a duration range. Further, an action template set can be obtained, which can include multiple template action features and an action label corresponding to each template action feature; the action label can refer to the action information indicated by the template action feature, which can be an action name, such as "running", "hugging", etc. Then, a classifier can be used to perform feature matching between the identified target character action features and the target template action features included in the action template set, and a matching result between the target character action features and the target template action features can be obtained; if the matching result between the target character action features and the target template action features indicates that the identified target character action features match the target template action features included in the action template set, then the action label corresponding to the target template action features, the face coordinate area corresponding to the target character action features, and the duration range corresponding to the target character action features can be recorded as a key information. Among them, the match between the identified target character action features and the target template action features included in the action template set indicates that the action information indicated by the target character action features and the action information indicated by the target template action features are highly likely to be the same action information; the key information recorded by this implementation method can be used to represent the action information performed by the character object appearing in the video within a duration range.

[0171] It should be noted that the training processes of the scene recognition model, face recognition model and action recognition model are similar, except that the sample image sets used in training the scene recognition model, face recognition model and action recognition model are different; the sample image set used in training the scene recognition model mainly includes real-life landscape images (for example, images containing mountains, fields, train stations, etc.); the sample image set used in training the face recognition model mainly includes various face images; the sample image set used in training the action recognition model mainly includes action images, i.e., images containing various action forms of characters.

[0172] S503, generating a narration audio of the video according to the key information.

[0173] By identifying the plot content of the video, at least one key information of the plot content can be obtained. The key information may include but is not limited to at least one of the following: scene switching information, character objects, and action information performed by the character objects. The narration audio generated based on the key information may include any of the following:

[0174] (1) The key information may include the scene label of the video at the i-th playback time point. In this case, the scene label at the i-th playback time point may be used as a plot label, and the plot label may be converted into an audio clip corresponding to the i-th playback time point, and the narration audio may include the audio clip corresponding to the i-th playback time point. For example, if the scene label of the video at the i-th playback time point is "at the bow", then the scene label at the i-th playback time point may be used as a plot label "at the bow".

[0175] (2) The key information may include the scene label of the video at the i-th playback time point and the character object label of the video at the i-th playback time point. In this case, the scene label of the video at the i-th playback time point and the character object label of the video at the i-th playback time point may be spliced ​​into a plot label, and the plot label may be converted into an audio clip corresponding to the i-th playback time point, and the narration audio may include the audio clip corresponding to the i-th playback time point. Furthermore, the splicing format of the plot label may be "the character object label of the video at the i-th playback time point + the scene label of the video at the i-th playback time point". For example, the character object label of the video at the i-th playback time point is "Jack", and the scene label of the video at the i-th playback time point is "at the bow", then the scene label of the video at the i-th playback time point and the character object label of the video at the i-th playback time point may be spliced ​​into a plot label "Jack at the bow".

[0176] (3) The key information may include: the scene label of the video at the i-th playback time point, the character object label of the video at the i-th playback time point, the face coordinate area corresponding to the character object label at the i-th playback time point, the action label of the video in the target duration range, and the face coordinate area corresponding to the action label in the target duration range; wherein the i-th playback time point is within the target duration range, and the face coordinate area corresponding to the character object label of the i-th playback time point matches the face coordinate area corresponding to the action label in the target duration range. The matching of the face coordinate area corresponding to the character object label of the i-th playback time point and the face coordinate area corresponding to the action label in the target duration range indicates that the character object indicated by the character object label at the i-th playback time point is the same as the character object performing the action in the target duration range. In this case, the scene label of the video at the i-th playback time point, the character object label of the video at the i-th playback time point, and the action label of the video in the target duration range can be spliced ​​into a plot label; then the plot label is converted into an audio clip corresponding to the i-th playback time point, and the narration audio includes the audio clip corresponding to the i-th playback time point. Furthermore, the concatenation format of the plot labels can be "the character object label of the video at the i-th playback time point + the scene label of the video at the i-th playback time point + the action label of the video in the target duration range". For example, the character object label of the video at the i-th playback time point is "Jack", the scene label of the video at the i-th playback time point is "at the bow", and the action label of the video at the i-th playback time point is "hug", then the scene label of the video at the i-th playback time point, the character object label of the video at the i-th playback time point, and the action label of the video in the target duration range can be concatenated into a plot label "Jack hugs at the bow".

[0177] The narration audio can be used to be played during the process of playing the content audio corresponding to the video when the image screen of the video is hidden. Specifically, the i-th playback time point corresponds to the i-th audio segment in the narration audio; playing the narration audio during the process of playing the content audio corresponding to the video may include: responding to the audio playback trigger event, hiding the image screen of the video, and playing the content audio corresponding to the video; during the playback of the content audio, when the content audio is played to the i-th playback time point, the content audio can be paused and the i-th audio segment can be played; after the i-th audio segment is played, the content audio can be played continuously from the i-th playback time point.

[0178] In an embodiment of the present application, the plot content of the video can be identified to obtain key information of the plot content, and the narration audio of the video can be generated based on the key information of the plot content; the narration audio can be used to be played in the process of playing the content audio corresponding to the video when the image screen of the video is hidden. The narration audio is generated based on the key information obtained by identifying the plot content, so the narration audio can prompt the key information in the plot content of the video. When the image screen of the video is hidden, playing the narration audio in the process of playing the content audio can help users have a richer and deeper understanding of the plot content, and effectively improve the audio playback effect. In addition, in the case where the key information includes scene identification information, if the scene switching is too frequent, the recorded key information can be filtered to reduce the burden of splicing plot labels, improve the audio playback efficiency, and thus improve the audio playback effect.

[0179] The method of the embodiment of the present application is described in detail above. In order to facilitate better implementation of the above scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly.

[0180] See also Figure 8 , Figure 8 A schematic diagram of the structure of a playback processing device provided by an exemplary embodiment of the present application is shown. The playback processing device 80 can be set in a playback processing device, which can be Figure 1 The terminal 101 in the playback processing system shown; the playback processing device 80 can be used to execute Figure 3 According to the corresponding steps in the method embodiment shown, the playback processing device 80 may include the following units:

[0181] Display unit 801, used to display the video playback interface;

[0182] The processing unit 802 is used to hide the image of the video in response to the audio playback triggering event; and play the content audio corresponding to the video;

[0183] The processing unit 802 is further configured to play the narration audio of the video during the process of playing the content audio, where the narration audio is generated based on the recognition of the plot content of the video.

[0184] In one implementation, the narration audio includes N audio clips, each audio clip corresponds to a playback time point of the content audio; any audio clip among the N audio clips is represented as the i-th audio clip, and the i-th audio clip corresponds to the i-th playback time point of the content audio; wherein i and N are both positive integers and i≤N;

[0185] The processing unit 802 is used to perform the following steps when playing the narration audio of the video during the process of playing the content audio:

[0186] When the content audio is played to the i-th playback time point, the content audio is paused and the i-th audio segment is played; and after the i-th audio segment is finished playing, the content audio is continued to be played from the i-th playback time point.

[0187] In one implementation, the playback interface is a service interface of an application program, and the application program includes a narration audio playback switch; the processing unit 802 is further configured to perform the following steps:

[0188] During the playing of the i-th audio segment, in response to the narration audio playing switch being triggered to be turned off, the playing of the i-th audio segment is stopped and the playing of the content audio continues.

[0189] In one implementation, the narration audio is used to prompt key information of the plot content of the video; wherein the i-th audio clip is used to prompt at least one of the following key information: scene switching information occurring at the i-th playback time point of the video, character objects appearing at the i-th playback time point of the video, and action information performed by the character objects appearing at the i-th playback time point of the video.

[0190] In one implementation, the display unit 801 is further configured to perform the following steps:

[0191] When the content audio is played to the first progress, in response to the video playback triggering event, the image screen of the video is restored from the first progress in the playback interface.

[0192] In one implementation, the processing unit 802 is further configured to perform the following steps:

[0193] In the process of displaying the video playback interface, in response to the first gravity sensing operation, an audio playback triggering event is generated;

[0194] Alternatively, during the playback of the content audio or the narration audio, a video playback trigger event is generated in response to a second gravity sensing operation.

[0195] In one implementation, the playback interface includes an audio playback option and a video playback option; the processing unit 802 is further configured to perform the following steps:

[0196] When the audio playback option is selected, an audio playback trigger event is generated;

[0197] Alternatively, when the video play option is selected, a video play trigger event is generated.

[0198] In one implementation, the playback interface is a service interface of an application program; the processing unit 802 is further configured to perform the following steps:

[0199] If the application switches from the foreground to the background, an audio playback trigger event is generated;

[0200] Alternatively, if the application is switched from running in the background to running in the foreground, a video playback trigger event is generated.

[0201] In one implementation, the playback interface is a service interface of an application program, and the application program includes a narration audio playback switch; the processing unit 802 is further configured to perform the following steps:

[0202] When the narration audio playback switch is turned on, the narration audio of the video will be played during the playback of the content audio.

[0203] According to one embodiment of the present application, Figure 8 The various units in the playback processing device 80 shown can be individually or completely combined into one or several other units to form, or one (some) of the units can be further divided into multiple functionally smaller units to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the playback processing device 80 may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units. According to another embodiment of the present application, it can be executed by running on a general computing device of a general-purpose computer including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM). Figure 3 A computer program (including program code) for each step involved in the corresponding method shown in Figure 8 The playback processing device 80 shown in the figure can be used to implement the playback processing method of the embodiment of the present application. The computer program can be recorded on a computer-readable storage medium, for example, and loaded into the above-mentioned computing device through the computer-readable storage medium and run therein.

[0204] In the embodiment of the present application, in response to the audio playback triggering event, the image of the video can be hidden, the content audio corresponding to the video can be played, and the narration audio of the video can be played during the playback of the content audio. The narration audio is generated based on the recognition of the plot content of the video, so the narration audio can prompt the plot content of the video. Playing the narration audio during the playback of the content audio can help users have a richer and deeper understanding of the plot content, effectively improving the audio playback effect.

[0205] See also Fig. 9 , Fig. 9 A schematic diagram of the structure of a playback processing device provided by another exemplary embodiment of the present application is shown. The playback processing device 90 can be arranged in a playback processing device, which can be Figure 1 The terminal 101 or the server 102 in the playback processing system shown; the playback processing device 90 can be used to execute Figure 5 According to the corresponding steps in the method embodiment shown, the playback processing device 90 may include the following units:

[0206] An acquisition unit 901 is used to acquire a video to be processed;

[0207] The processing unit 902 is used to identify the plot content of the video and obtain key information of the plot content;

[0208] The processing unit 902 is further used to generate a narration audio of the video according to the key information. The narration audio is used to be played during the process of playing the content audio corresponding to the video when the image screen of the video is hidden.

[0209] In one implementation, the video includes a plurality of video frames arranged in sequence, and the jth frame is any video frame in the video; the processing unit 902 is used to identify the plot content of the video, and when obtaining key information of the plot content, it is specifically used to perform the following steps:

[0210] Perform scene recognition on the image picture corresponding to the jth frame and the image picture corresponding to the j+1th frame of the video respectively;

[0211] Compare the scene change values ​​between the jth frame and the j+1th frame;

[0212] If the scene change value is greater than the change threshold, the i-th playback time point corresponding to the j+1-th frame in the video and the scene label of the j+1-th frame obtained by scene recognition are recorded as a key information. The recorded key information is used to indicate the scene switching information occurring at the i-th playback time point in the video.

[0213] In one implementation, the video includes a plurality of video frames arranged in sequence, and the j+1th frame is any video frame in the video; the processing unit 902 is used to identify the plot content of the video, and when obtaining key information of the plot content, specifically to perform the following steps:

[0214] Perform face recognition on the image corresponding to the j+1th frame of the video to obtain one or more character facial features and the facial coordinate area corresponding to each character facial feature;

[0215] Obtain a face template set, the face template set including multiple template face features and a role object label corresponding to each template face feature;

[0216] If the facial features of the target character obtained by face recognition match the target template facial features included in the face template set, the i-th playback time point corresponding to the j+1-th frame in the video, the character object label corresponding to the target template facial features, and the face coordinate area corresponding to the target character facial features are recorded as a key information. The recorded key information is used to represent a character object that appears in the video at the i-th playback time point.

[0217] In one implementation, the video includes a plurality of video frames arranged in sequence; the processing unit 902 is used to identify the plot content of the video, and when obtaining key information of the plot content, it is specifically used to perform the following steps:

[0218] Performing action recognition on the image pictures of multiple video frames contained in the video frame by frame to obtain one or more character action features, the face coordinate area corresponding to each character action feature, and the duration range of each character action feature;

[0219] Acquire an action template set, wherein the action template set includes a plurality of template action features and an action label corresponding to each template action feature;

[0220] If the identified target character action feature matches the target template action feature included in the action template set, the action label corresponding to the target template action feature, the facial coordinate area corresponding to the target character action feature, and the duration range corresponding to the target character action feature are recorded as a key information. The recorded key information is used to represent the action information performed by the character object appearing in the video within a duration range.

[0221] In one implementation, the key information includes a scene tag of the video at the i-th playback time point; when the processing unit 902 is used to generate the narration audio of the video according to the key information, it is specifically used to perform the following steps:

[0222] The scene label of the i-th playback time point is used as a plot label;

[0223] The plot tag is converted into an audio segment corresponding to the i-th playback time point, and the narration audio includes the audio segment corresponding to the i-th playback time point.

[0224] In one implementation, the key information includes a scene tag of the video at the i-th playback time point and a character object tag of the video at the i-th playback time point; when the processing unit 902 is used to generate the narration audio of the video according to the key information, it is specifically used to perform the following steps:

[0225] The scene label at the i-th playback time point and the character object label at the i-th playback time point are concatenated into a plot label;

[0226] The plot label is converted into an audio clip corresponding to the i-th playback time point, and the narration audio includes the audio clip corresponding to the i-th playback time point.

[0227] In one implementation, the key information includes: a scene label of the video at the i-th playback time point, a character object label of the video at the i-th playback time point, a face coordinate region corresponding to the character object label at the i-th playback time point, an action label of the video within a target duration range, and a face coordinate region corresponding to the action label within the target duration range; wherein the i-th playback time point is within the target duration range, and the face coordinate region corresponding to the character object label at the i-th playback time point matches the face coordinate region corresponding to the action label within the target duration range;

[0228] When the processing unit 902 is used to generate the narration audio of the video according to the key information, it is specifically used to perform the following steps:

[0229] The scene label at the i-th playback time point, the character object label at the i-th playback time point, and the action label of the target duration range are concatenated into a plot label;

[0230] The plot tag is converted into an audio segment corresponding to the i-th playback time point, and the narration audio includes the audio segment corresponding to the i-th playback time point.

[0231] According to one embodiment of the present application, Fig. 9The various units in the playback processing device 90 shown can be individually or completely combined into one or several other units to form, or one (some) of the units can be further divided into multiple functionally smaller units to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the playback processing device 90 may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units. According to another embodiment of the present application, it can be executed by running on a general-purpose computing device of a general-purpose computer including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM). Figure 5 A computer program (including program code) for each step involved in the corresponding method shown in Fig. 9 The playback processing device 90 shown in the figure can be used to implement the data processing method of the embodiment of the present application. The computer program can be recorded on a computer-readable storage medium, for example, and loaded into the above-mentioned computing device through the computer-readable storage medium and run therein.

[0232] In the embodiment of the present application, the plot content of the video can be identified to obtain key information of the plot content, and the narration audio of the video can be generated according to the key information of the plot content; the narration audio can be used to be played in the process of playing the content audio corresponding to the video when the image screen of the video is hidden. The narration audio is generated according to the key information obtained by identifying the plot content, so the narration audio can prompt the key information in the plot content of the video. When the image screen of the video is hidden, playing the narration audio in the process of playing the content audio can help users have a richer and deeper understanding of the plot content, and effectively improve the audio playback effect.

[0233] See also Fig.10 , Fig.10A structural schematic diagram of a playback processing device provided by an exemplary embodiment of the present application is shown, and the playback processing device 100 at least includes a processor 1001, a computer-readable storage medium 1002 and a communication interface 1003. Among them, the processor 1001, the computer-readable storage medium 1002 and the communication interface 1003 can be connected via a bus or other means. The communication interface 1003 can be used to receive or send audio playback requests, voice conversion requests, videos, narration audio, etc. The computer-readable storage medium 1002 can be stored in a memory, and the computer-readable storage medium 1002 is used to store computer programs, and the computer programs include computer instructions. The processor 1001 is used to execute computer instructions. The processor 1001 (or CPU (Central Processing Unit)) is the computing core and control core of the playback processing device 100, which is suitable for implementing one or more computer instructions, and is specifically suitable for loading and executing one or more computer instructions to implement the corresponding method flow or corresponding function.

[0234] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in the playback processing device 100 for storing programs and data. It is understandable that the computer-readable storage medium 1002 here can include both the built-in storage medium in the playback processing device 100 and the extended storage medium supported by the playback processing device 100. The computer-readable storage medium provides a storage space, which stores the operating system of the playback processing device 100. In addition, one or more computer instructions suitable for being loaded and executed by the processor 1001 are also stored in the storage space, and these computer instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium 1002 here can be a high-speed RAM memory, or a non-volatile memory (Non-Volatile Memory), such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor 1001.

[0235] The playback processing device 100 may be Figure 1 In the data processing system shown in the terminal 101, the computer readable storage medium 1002 stores a computer program, which includes one or more computer instructions; the processor 1001 loads and executes the one or more computer instructions to implement Figure 3 Corresponding steps in the method embodiment shown; in a specific implementation, the computer instructions in the computer-readable storage medium 1002 are loaded by the processor 1001 and execute the following steps:

[0236] Display the video playback interface;

[0237] In response to an audio playback trigger event, hiding the image of the video; and playing the content audio corresponding to the video;

[0238] During the process of playing the content audio, the narration audio of the video is played, and the narration audio is generated based on the recognition of the plot content of the video.

[0239] In one implementation, the narration audio includes N audio clips, each audio clip corresponds to a playback time point of the content audio; any audio clip among the N audio clips is represented as the i-th audio clip, and the i-th audio clip corresponds to the i-th playback time point of the content audio; wherein i and N are both positive integers and i≤N;

[0240] The computer instructions in the computer-readable storage medium 1002 are loaded and executed by the processor 1001 during the process of playing the content audio, when playing the narration audio of the video, specifically for performing the following steps:

[0241] When the content audio is played to the i-th playback time point, the content audio is paused and the i-th audio segment is played; and after the i-th audio segment is finished playing, the content audio continues to be played.

[0242] In one implementation, the playback interface is a service interface of an application program, and the application program includes a narration audio playback switch; the computer instructions in the computer-readable storage medium 1002 are loaded by the processor 1001 and are also used to perform the following steps:

[0243] During the playback of the i-th audio clip, if the narration audio playback switch is triggered to be turned off, the playback of the i-th audio clip is stopped and the content audio continues to be played.

[0244] In one implementation, the narration audio is used to prompt key information of the plot content of the video; wherein the i-th audio clip is used to prompt at least one of the following key information: scene switching information occurring at the i-th playback time point of the video, character objects appearing at the i-th playback time point of the video, and action information performed by the character objects appearing at the i-th playback time point of the video.

[0245] In one implementation, the computer instructions in the computer readable storage medium 1002 are loaded by the processor 1001 and are also used to perform the following steps:

[0246] When the content audio is played to the first progress, in response to the video playback triggering event, the image screen of the video is restored from the first progress in the playback interface.

[0247] In one implementation, the computer instructions in the computer readable storage medium 1002 are loaded by the processor 1001 and are also used to perform the following steps:

[0248] In the process of displaying the video playback interface, in response to the first gravity sensing operation, an audio playback triggering event is generated;

[0249] Alternatively, during the playback of the content audio or the narration audio, a video playback trigger event is generated in response to a second gravity sensing operation.

[0250] In one implementation, the playback interface includes an audio playback option and a video playback option; the computer instructions in the computer-readable storage medium 1002 are loaded by the processor 1001 and are also used to perform the following steps:

[0251] When the audio playback option is selected, an audio playback trigger event is generated;

[0252] Alternatively, when the video play option is selected, a video play trigger event is generated.

[0253] In one implementation, the playback interface is a service interface of an application program; the computer instructions in the computer-readable storage medium 1002 are loaded by the processor 1001 and are also used to perform the following steps:

[0254] If the application switches from the foreground to the background, an audio playback trigger event is generated;

[0255] Alternatively, if the application is switched from running in the background to running in the foreground, a video playback trigger event is generated.

[0256] In one implementation, the playback interface is a service interface of an application program, and the application program includes a narration audio playback switch; the computer instructions in the computer-readable storage medium 1002 are loaded by the processor 1001 and are also used to perform the following steps:

[0257] When the narration audio playback switch is turned on, the narration audio of the video will be played during the playback of the content audio.

[0258] The playback processing device 100 may also be Figure 1 In the terminal 101 or server 102 of the data processing system shown in the figure, the computer readable storage medium 1002 stores a computer program, and the computer program includes one or more computer instructions; the processor 1001 loads and executes the one or more computer instructions to implement Figure 5 Corresponding steps in the method embodiment shown; in a specific implementation, the computer instructions in the computer-readable storage medium 1002 are loaded by the processor 1001 and execute the following steps:

[0259] Get the video to be processed;

[0260] Identify the plot content of the video and obtain key information of the plot content;

[0261] The narration audio of the video is generated according to the key information, and the narration audio is used to be played during the process of playing the content audio corresponding to the video when the image screen of the video is hidden.

[0262] In one implementation, a video includes a plurality of video frames arranged in sequence, and the jth frame is any video frame in the video; the computer instructions in the computer-readable storage medium 1002 are loaded and executed by the processor 1001 to identify the plot content of the video, and when the key information of the plot content is obtained, the computer instructions are specifically used to execute the following steps:

[0263] Perform scene recognition on the image picture corresponding to the jth frame and the image picture corresponding to the j+1th frame of the video respectively;

[0264] Compare the scene change values ​​between the jth frame and the j+1th frame;

[0265] If the scene change value is greater than the change threshold, the i-th playback time point corresponding to the j+1-th frame in the video and the scene label of the j+1-th frame obtained by scene recognition are recorded as a key information. The recorded key information is used to indicate the scene switching information occurring at the i-th playback time point in the video.

[0266] In one implementation, the video includes a plurality of video frames arranged in sequence, and the j+1th frame is any video frame in the video; the computer instructions in the computer-readable storage medium 1002 are loaded and executed by the processor 1001 to identify the plot content of the video, and when the key information of the plot content is obtained, the computer instructions are specifically used to execute the following steps:

[0267] Perform face recognition on the image corresponding to the j+1th frame of the video to obtain one or more character facial features and the facial coordinate area corresponding to each character facial feature;

[0268] Obtain a face template set, the face template set including multiple template face features and a role object label corresponding to each template face feature;

[0269] If the facial features of the target character obtained by face recognition match the target template facial features included in the face template set, the i-th playback time point corresponding to the j+1-th frame in the video, the character object label corresponding to the target template facial features, and the face coordinate area corresponding to the target character facial features are recorded as a key information. The recorded key information is used to represent a character object that appears in the video at the i-th playback time point.

[0270] In one implementation, the video includes a plurality of sequentially arranged video frames; the computer instructions in the computer-readable storage medium 1002 are loaded and executed by the processor 1001 to identify the plot content of the video, and when the key information of the plot content is obtained, the computer instructions are specifically used to perform the following steps:

[0271] Performing action recognition on the image pictures of multiple video frames contained in the video frame by frame to obtain one or more character action features, the face coordinate area corresponding to each character action feature, and the duration range of each character action feature;

[0272] Acquire an action template set, wherein the action template set includes a plurality of template action features and an action label corresponding to each template action feature;

[0273] If the identified target character action feature matches the target template action feature included in the action template set, the action label corresponding to the target template action feature, the facial coordinate area corresponding to the target character action feature, and the duration range corresponding to the target character action feature are recorded as a key information. The recorded key information is used to represent the action information performed by the character object appearing in the video within a duration range.

[0274] In one implementation, the key information includes a scene tag of the video at the i-th playback time point; when the computer instructions in the computer-readable storage medium 1002 are loaded and executed by the processor 1001 to generate the narration audio of the video according to the key information, the computer instructions are specifically used to perform the following steps:

[0275] The scene label of the i-th playback time point is used as a plot label;

[0276] The plot tag is converted into an audio segment corresponding to the i-th playback time point, and the narration audio includes the audio segment corresponding to the i-th playback time point.

[0277] In one implementation, the key information includes the scene tag of the video at the i-th playback time point and the character object tag of the video at the i-th playback time point. When the computer instructions in the computer-readable storage medium 1002 are loaded and executed by the processor 1001 to generate the narration audio of the video according to the key information, the computer instructions are specifically used to perform the following steps:

[0278] The scene label at the i-th playback time point and the character object label at the i-th playback time point are concatenated into a plot label;

[0279] The plot label is converted into an audio clip corresponding to the i-th playback time point, and the narration audio includes the audio clip corresponding to the i-th playback time point.

[0280] In one implementation, the key information includes: a scene label of the video at the i-th playback time point, a character object label of the video at the i-th playback time point, a face coordinate region corresponding to the character object label at the i-th playback time point, an action label of the video within a target duration range, and a face coordinate region corresponding to the action label within the target duration range; wherein the i-th playback time point is within the target duration range, and the face coordinate region corresponding to the character object label at the i-th playback time point matches the face coordinate region corresponding to the action label within the target duration range;

[0281] When the computer instructions in the computer-readable storage medium 1002 are loaded and executed by the processor 1001 to generate the narration audio of the video according to the key information, they are specifically used to perform the following steps:

[0282] The scene label at the i-th playback time point, the character object label at the i-th playback time point, and the action label of the target duration range are concatenated into a plot label;

[0283] The plot tag is converted into an audio segment corresponding to the i-th playback time point, and the narration audio includes the audio segment corresponding to the i-th playback time point.

[0284] In an embodiment of the present application, when the image screen of the video is hidden, the narration audio of the video can be played during the process of playing the content audio corresponding to the video; the narration audio is generated by identifying the plot content of the video, so the narration audio can prompt the plot content of the video, and playing the narration audio during the process of playing the content audio can help users have a richer and deeper understanding of the plot content, and effectively improve the audio playback effect.

[0285] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the playback processing method provided in the above-mentioned various optional modes.

[0286] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A playback processing method, characterized in that: Applied to a terminal, the method comprises: Display the video playback interface; In response to an audio playback triggering event, hiding the image screen of the video; and, Play the content audio corresponding to the video, identify the plot content of the video, obtain at least one key information of the plot content, each key information has a corresponding playback time point, splice and combine the key information to obtain one or more plot tags, each plot tag corresponds to a playback time point of the content audio, perform voice conversion on each plot tag respectively, obtain the audio clip corresponding to each plot tag, the narration audio is composed of the audio clips corresponding to each plot tag, identify the plot content of the video, and obtain at least one key information of the plot content, including: perform scene recognition on the image screen corresponding to the jth frame of the video and the image screen corresponding to the j+1th frame respectively, compare the scene change value between the jth frame and the j+1th frame, if the scene change value is greater than the change threshold, then record the qth playback time point corresponding to the j+1th frame in the video and the scene tag of the j+1th frame obtained by scene recognition as a key information, j is a non-negative integer, and q is a positive integer; During the process of playing the content audio, the narration audio of the video is played, including: when the content audio playback reaches the playback time point corresponding to the target plot tag, pausing the content audio and starting to play the audio segment corresponding to the target plot tag; after the audio segment corresponding to the target plot tag is finished playing, continuing to play the content audio from the playback time point corresponding to the target plot tag; wherein the target plot tag is any plot tag obtained by splicing and combining, the narration audio is generated based on the recognition of the plot content of the video, and the narration audio is used to prompt the scene switching information of the video at the playback time point.

2. The method according to claim 1, characterized in that The narration audio includes N audio segments, each of which corresponds to a playback time point of the content audio; any audio segment among the N audio segments is represented as the i-th audio segment, and the i-th audio segment corresponds to the i-th playback time point of the content audio; wherein i and N are both positive integers and i≤N; The step of playing the narration audio of the video during the process of playing the content audio includes: When the content audio is played to the i-th playing time point, the content audio is paused and the i-th audio segment is played; and, After the i-th audio segment is finished playing, the content audio continues to be played.

3. The method according to claim 2, characterized in that The playback interface is a service interface of an application program, and the application program includes a narration audio playback switch; the method further includes: During the playing of the i-th audio segment, in response to the narration audio playing switch being triggered to be turned off, the playing of the i-th audio segment is stopped and the playing of the content audio continues.

4. The method according to claim 2, characterized in that The i-th audio clip is also used to prompt at least one of the following key information: the character object that appears in the video at the i-th playback time point, and the action information performed by the character object that appears in the video at the i-th playback time point.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: When the content audio is played to a first progress, in response to a video play triggering event, the image screen of the video is restored to be displayed from the first progress in the play interface.

6. The method according to claim 5, characterized in that The method further comprises: In the process of displaying the video playback interface, in response to a first gravity sensing operation, generating the audio playback triggering event; or, During the playback of the content audio or the narration audio, the video playback trigger event is generated in response to a second gravity sensing operation.

7. The method according to claim 5, characterized in that The playback interface includes an audio playback option or a video playback option; the method further includes: When the audio play option is selected, the audio play trigger event is generated; or, When the video play option is selected, the video play trigger event is generated.

8. The method according to claim 5, characterized in that The playback interface is a service interface of an application program; the method further includes: If the application is switched from foreground operation to background operation, the audio playback trigger event is generated; or, If the application is switched from background operation to foreground operation, the video playback triggering event is generated.

9. The method according to claim 1, characterized in that The playback interface is a service interface of an application program, and the application program includes a narration audio playback switch; the method further includes: When the narration audio playing switch is in the on state, the narration audio of the video is played during the playing of the content audio.

10. A playback processing method, characterized in that: Applied to a server, the method comprises: Get the video to be processed; Identify the plot content of the video to obtain at least one key information of the plot content, each key information has a corresponding playback time point, splice and combine the key information to obtain one or more plot tags, each plot tag corresponds to a playback time point of the content audio, send the content audio corresponding to the video, the plot tags obtained by splicing and combining the key information, and the playback time point corresponding to each plot tag to the terminal, identify the plot content of the video to obtain at least one key information of the plot content, including: perform scene recognition on the image picture corresponding to the jth frame of the video and the image picture corresponding to the j+1th frame, compare the scene change value between the jth frame and the j+1th frame, if the scene change value is greater than the change threshold, then record the qth playback time point corresponding to the j+1th frame in the video and the scene tag of the j+1th frame obtained by scene recognition as a key information, j is a non-negative integer, and q is a positive integer; Generating the narration audio of the video according to the key information includes: receiving a voice conversion request sent by a terminal, the voice conversion request includes a target plot tag, performing voice conversion on the target plot tag to obtain an audio segment corresponding to the target plot tag, and the narration audio includes audio segments obtained by converting each plot tag; the voice conversion request is sent by the terminal to the server when the content audio is played to the playback time point corresponding to the target plot tag, pausing the content audio, the narration audio is used to be played during the playback of the content audio corresponding to the video when the image screen of the video is hidden, playing the audio segment corresponding to the target plot tag from the playback time point corresponding to the target plot tag, and continuing to play the content audio from the playback time point corresponding to the target plot tag after the audio segment corresponding to the target plot tag is played, the narration audio is generated based on the recognition of the plot content of the video, and the narration audio is used to prompt the scene switching information of the video at the playback time point.

11. The method according to claim 10, characterized in that The video includes a plurality of video frames arranged in sequence, and the j+1th frame is any video frame in the video; the identifying the plot content of the video to obtain at least one key information of the plot content also includes: Performing face recognition on the image corresponding to the j+1th frame of the video to obtain one or more character facial features and a facial coordinate area corresponding to each character facial feature; Acquire a face template set, wherein the face template set includes a plurality of template face features and a role object label corresponding to each template face feature; If the target character's facial features obtained by face recognition match the target template facial features included in the face template set, the i-th playback time point corresponding to the j+1-th frame in the video, the character object label corresponding to the target template facial features, and the face coordinate area corresponding to the target character facial features are recorded as a key information, and the recorded key information is used to represent a character object that appears in the video at the i-th playback time point.

12. The method according to claim 10, characterized in that The video includes a plurality of video frames arranged in sequence; the identifying the plot content of the video to obtain at least one key information of the plot content also includes: Performing action recognition frame by frame on the image pictures of multiple video frames contained in the video to obtain one or more character action features, a facial coordinate area corresponding to each character action feature, and a duration range of each character action feature; Acquire an action template set, wherein the action template set includes a plurality of template action features and an action label corresponding to each template action feature; If the identified target character action feature matches the target template action feature included in the action template set, the action label corresponding to the target template action feature, the facial coordinate area corresponding to the target character action feature, and the duration range corresponding to the target character action feature are recorded as a key information. The recorded key information is used to represent the action information performed by the character object appearing in the video within a duration range.

13. The method according to claim 10, characterized in that The key information includes a scene tag of the video at the i-th playback time point; and generating the narration audio of the video according to the key information includes: Taking the scene label of the i-th playback time point as a plot label; The plot tag is converted into an audio segment corresponding to the i-th playback time point, and the narration audio includes the audio segment corresponding to the i-th playback time point.

14. The method according to claim 10, characterized in that The key information includes the scene label of the video at the i-th playback time point and the character object label of the video at the i-th playback time point; The step of generating the narration audio of the video according to the key information includes: splicing the scene label at the i-th playback time point and the character object label at the i-th playback time point into a plot label; The plot tag is converted into an audio segment corresponding to the i-th playback time point, and the narration audio includes the audio segment corresponding to the i-th playback time point.

15. The method according to claim 10, characterized in that The key information includes: the scene label of the video at the ith playback time point, the character object label of the video at the ith playback time point, the face coordinate area corresponding to the character object label at the ith playback time point, the action label of the video within the target duration range, and the face coordinate area corresponding to the action label within the target duration range; wherein the ith playback time point is within the target duration range, and the face coordinate area corresponding to the character object label at the ith playback time point matches the face coordinate area corresponding to the action label within the target duration range; The step of generating the narration audio of the video according to the key information includes: splicing the scene label of the i-th playback time point, the character object label of the i-th playback time point, and the action label of the target duration range into a plot label; The plot tag is converted into an audio segment corresponding to the i-th playback time point, and the narration audio includes the audio segment corresponding to the i-th playback time point.

16. A playback processing device, characterized in that: Applied to a terminal, the playback processing device comprises: A display unit, used to display a video playback interface; A processing unit, for hiding the image of the video in response to an audio playback triggering event; and, playing the content audio corresponding to the video, identifying the plot content of the video, obtaining at least one key information of the plot content, each key information having a corresponding playback time point, splicing and combining the key information to obtain one or more plot tags, each plot tag corresponding to a playback time point of the content audio, performing voice conversion on each plot tag to obtain an audio clip corresponding to each plot tag, the narration audio is composed of the audio clips corresponding to each plot tag, identifying the plot content of the video, and obtaining at least one key information of the plot content, including: performing scene recognition on the image corresponding to the jth frame of the video and the image corresponding to the j+1th frame, comparing the scene change value between the jth frame and the j+1th frame, if the scene change value is greater than the change threshold, recording the qth playback time point corresponding to the j+1th frame in the video and the scene tag of the j+1th frame obtained by scene recognition as a key information, j is a non-negative integer, and q is a positive integer; The processing unit is also used to play the narration audio of the video during the process of playing the content audio, including: when the content audio playback reaches the playback time point corresponding to the target plot tag, pausing the content audio and starting to play the audio segment corresponding to the target plot tag; after the audio segment corresponding to the target plot tag is finished playing, continuing to play the content audio from the playback time point corresponding to the target plot tag; wherein the target plot tag is any plot tag obtained by splicing and combining, the narration audio is generated based on the recognition of the plot content of the video, and the narration audio is used to prompt the scene switching information of the video at the playback time point.

17. A playback processing device, characterized in that: Applied to a server, the playback processing device comprises: An acquisition unit, used for acquiring a video to be processed; A processing unit is used to identify the plot content of the video, obtain at least one key information of the plot content, each key information has a corresponding playback time point, and splice and combine the key information to obtain one or more plot tags, each plot tag corresponds to a playback time point of the content audio, and send the content audio corresponding to the video, the plot tags obtained by splicing and combining the key information, and the playback time point corresponding to each plot tag to the terminal, identify the plot content of the video, and obtain at least one key information of the plot content, including: performing scene recognition on the image screen corresponding to the jth frame of the video and the image screen corresponding to the j+1th frame, respectively, comparing the scene change value between the jth frame and the j+1th frame, if the scene change value is greater than the change threshold, then recording the qth playback time point corresponding to the j+1th frame in the video and the scene tag of the j+1th frame obtained by scene recognition as a key information, j is a non-negative integer, and q is a positive integer; The processing unit is further used to generate the narration audio of the video according to the key information, including: receiving a voice conversion request sent by a terminal, the voice conversion request including a target plot tag, performing voice conversion on the target plot tag to obtain an audio segment corresponding to the target plot tag, and the narration audio includes audio segments obtained by converting each plot tag; the voice conversion request is sent by the terminal to the server when the content audio is played to the playback time point corresponding to the target plot tag, pausing the content audio, the narration audio is used to be played in the process of playing the content audio corresponding to the video when the image screen of the video is hidden, playing the audio segment corresponding to the target plot tag from the playback time point corresponding to the target plot tag, and continuing to play the content audio from the playback time point corresponding to the target plot tag after the audio segment corresponding to the target plot tag is played, the narration audio is generated based on the recognition of the plot content of the video, and the narration audio is used to prompt the scene switching information of the video at the playback time point.

18. A playback processing device, characterized in that: The playback processing device comprises: a processor adapted to implement a computer program; and, A computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded by the processor and executing the playback processing method as described in any one of claims 1 to 9, or executing the playback processing method as described in any one of claims 10 to 15.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a computer program, and the computer program is suitable for being loaded by a processor and executing the playback processing method according to any one of claims 1 to 9, or executing the playback processing method according to any one of claims 10 to 15.

Images