Live interactive method, device, electronic device and readable storage medium

Abstract

The embodiment of the present application provides a live interactive method, device, electronic device and readable storage medium, which divides the first audio data input by the anchor and extracts a content feature map from the obtained multiple audio segments, then extracts the content feature vector of the content feature map and converts the content feature vector to obtain a style conversion map with a target timbre style. Then, the content feature map and the style conversion map are subjected to feature inverse conversion to obtain a second audio data with the target timbre style. Finally, an interactive video stream of the virtual image corresponding to the anchor is generated based on the second audio data, and sent to the client for playback. In this way, for any anchor, the timbre style of the virtual image during the live broadcast can be converted to the target timbre style to interact with the audience without changing the audio content, thereby improving the interactive effect during the live broadcast and mobilizing the interaction between the audience and the anchor to a greater extent.

Description

[0001] This application is a divisional application of the Chinese application with application number 2019103685107, application date May 5, 2019, and invention name “Live interactive method, device, electronic device and readable storage medium”. Technical Field

[0002] The present application relates to the field of Internet live broadcasting, and more specifically, to a live broadcasting interactive method, device, electronic device and readable storage medium. Background Art

[0003] In Internet live broadcasting, using a virtual image instead of the host's actual image to participate in the live broadcast interaction is currently a relatively popular live broadcast method.

[0004] In the current live broadcasting methods, the timbre of the virtual image mostly adopts the original timbre style of the anchor or a pre-fixed timbre style to provide live broadcast data stream, and it cannot be converted into other timbre styles to interact with the audience. This cannot meet certain specific needs of specific anchors or specific audiences, which will lead to a decrease in the interactive live broadcast effect. For example, the audience may prefer to hear the timbre style of their favorite star or the timbre style of someone they are familiar with. For another example, the anchor may not want to expose his or her timbre style to other viewers and expose privacy issues. Summary of the invention

[0005] In view of this, the purpose of the embodiments of the present application is to provide a live interactive method, device, electronic device and readable storage medium to solve the above problems.

[0006] According to one aspect of an embodiment of the present application, an electronic device is provided, which may include one or more storage media and one or more processors communicating with the storage media. The one or more storage media store machine executable instructions executable by the processor. When the electronic device is running, the processor executes the machine executable instructions to perform a live interactive method.

[0007] According to another aspect of an embodiment of the present application, a live interactive method is provided, which is applied to a host terminal, wherein at least one style conversion model is stored in the host terminal, and each style conversion model corresponds to a timbre style. The method includes:

[0008] According to the received timbre conversion request, at a preset time interval, the first audio data input by the host is segmented to obtain multiple audio segments, and an audio feature graph is extracted from the multiple audio segments, wherein the audio feature graph includes a content feature graph, and the timbre conversion request includes a target timbre style;

[0009] Inputting the content feature map into a preset feature vector extraction network to extract a content feature vector of the content feature map;

[0010] The content feature vector is converted using a style conversion model corresponding to the target timbre style to obtain a style conversion graph having the target timbre style;

[0011] Performing feature inverse conversion on the content feature map and the style conversion map to obtain second audio data having the target timbre style;

[0012] An interactive video stream of the virtual image corresponding to the anchor is generated according to the second audio data, and sent to the client for playback.

[0013] According to another aspect of an embodiment of the present application, a live interactive device is provided, which is applied to a host terminal, wherein at least one style conversion model is stored in the host terminal, and each style conversion model corresponds to a timbre style. The device includes:

[0014] an extraction module, configured to segment the first audio data input by the host at preset intervals according to the received timbre conversion request to obtain a plurality of audio segments, and extract an audio feature graph from the plurality of audio segments, wherein the audio feature graph includes a content feature graph, and the timbre conversion request includes a target timbre style;

[0015] An input module, used for inputting the content feature map into a preset feature vector extraction network to extract a content feature vector of the content feature map;

[0016] A conversion module, configured to convert the content feature vector using a style conversion model corresponding to the target timbre style to obtain a style conversion graph having the target timbre style;

[0017] A reverse conversion module, used for performing feature reverse conversion on the content feature map and the style conversion map to obtain second audio data having the target timbre style;

[0018] A sending module is generated, which is used to generate an interactive video stream of the virtual image corresponding to the anchor according to the second audio data, and send it to the client for playback.

[0019] According to another aspect of an embodiment of the present application, a readable storage medium is provided, on which machine executable instructions are stored. When the computer program is executed by a processor, the steps of the above-mentioned live interactive method can be executed.

[0020] Based on any of the above aspects, compared with the prior art, the embodiment of the present application divides the first audio data input by the anchor, extracts a content feature graph from the obtained multiple audio segments, and then extracts the content feature vector through a preset feature vector extraction network, and then uses the style conversion model corresponding to the target timbre style to convert the content feature vector to obtain a style conversion graph with the target timbre style. Then, the content feature graph and the style conversion graph are subjected to feature inverse conversion to obtain the second audio data with the target timbre style. Finally, an interactive video stream of the virtual image corresponding to the anchor is generated according to the second audio data, and sent to the client for playback. In this way, for the audio content provided by any anchor, the timbre style of the virtual image during the live broadcast can be converted to the target timbre style to interact with the audience without changing the audio content, thereby improving the interactive effect during the live broadcast and mobilizing the interaction between the audience and the anchor to a greater extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A schematic diagram of a live broadcast system provided in an embodiment of the present application is shown;

[0023] Figure 2 One of the flow diagrams of the live interactive method provided in the embodiment of the present application is shown;

[0024] Figure 3 A schematic diagram of an interface for selecting a target timbre style in a live Internet application provided by an embodiment of the present application is shown;

[0025] Figure 4 A schematic diagram of the live broadcast interface of the anchor terminal provided in an embodiment of the present application is shown;

[0026] Figure 5 The second flowchart of the live interactive method provided in the embodiment of the present application is shown;

[0027] Figure 6 The embodiment of the present application provides Figure 5 The schematic diagram of the flow chart of each sub-step included in step S101 shown in FIG.

[0028] Figure 7 A schematic diagram of the training process of the style conversion model provided in an embodiment of the present application is shown;

[0029] Figure 8 A schematic diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0031] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0032] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0033] Reference Figure 1 As shown, Figure 1 1 is a schematic diagram of the architecture of the live broadcast system 10 provided in an embodiment of the present application. For example, the live broadcast system 10 may be a service platform for Internet live broadcasting. The live broadcast system 10 may include a live broadcast server 200, an anchor terminal 100, and a client 300. The live broadcast server 200 is respectively connected to the anchor terminal 100 and the client 300 for providing live broadcast services to the anchor terminal 100 and the client 300. For example, the live broadcast server 200 may store the correspondence between the anchor terminal 100 and each live broadcast channel. When the client 300 selects a live broadcast channel, the live broadcast server 200 may send a live video stream to the client 300 belonging to the same live broadcast channel according to the correspondence between each live broadcast channel and the anchor terminal 100.

[0034] In some implementation scenarios, the anchor terminal 100 and the client 300 can be used interchangeably. For example, the anchor of the anchor terminal 100 can use the anchor terminal 100 to provide live video services to the audience, or view the live videos provided by other anchors as an audience. For another example, the audience of the client 300 can also use the client 300 to watch the live video provided by the anchor of interest, or provide live video services to other audiences as an anchor. In this embodiment, the anchor terminal 100 and the client 300 can include but are not limited to any handheld electronic product based on an intelligent operating system, which can interact with the user through input devices such as keyboards, virtual keyboards, touch pads, touch screens, and voice control devices, such as smart phones, tablet computers, personal computers, etc. Among them, the intelligent operating system includes but is not limited to any operating system that enriches the device functions by providing various mobile applications to mobile devices, such as Android, iOS, Windows Phone, etc. Among them, the anchor terminal 100 and the client 300 can be installed with Internet products for providing Internet live broadcast services, for example, Internet products can be application APPs, Web pages, applets, etc. related to Internet live broadcast services used in computers or smart phones.

[0035] In this embodiment, the live broadcast system 10 may further include a video acquisition device 400 for acquiring the anchor's anchor video frames. The video acquisition device 400 is directly installed or integrated in the anchor terminal 100, or may be independent of the anchor terminal 100 and connected to the anchor terminal 100.

[0036] Reference Figure 2 As shown, Figure 2 A schematic diagram showing a flow chart of a live interactive method provided by an embodiment of the present application is shown. The live interactive method can be performed by Figure 1 It should be understood that in other embodiments, the order of some steps in the live interactive method of this embodiment can be interchanged according to actual needs, or some steps can be omitted or deleted. The detailed steps of the live interactive method are described as follows.

[0037] Step S110, according to the received timbre conversion request, the first audio data input by the host is segmented at every preset time interval to obtain multiple audio segments, and audio feature graphs are extracted from the multiple audio segments.

[0038] Step S120: input the content feature map into a preset feature vector extraction network to extract a content feature vector of the content feature map.

[0039] Step S130 , converting the content feature vector using a style conversion model corresponding to the target timbre style to obtain a style conversion graph having the target timbre style.

[0040] Step S140: perform feature inverse conversion on the content feature map and the style conversion map to obtain second audio data having the target timbre style.

[0041] Step S150, generating an interactive video stream of the virtual image corresponding to the anchor according to the second audio data, and sending it to the client 300 for playback.

[0042] In this embodiment, with respect to step S110, any anchor can start the live broadcast by clicking on the live broadcast Internet application installed on the anchor terminal 100 and entering the live broadcast interface. During the live broadcast, data such as live video streams, live pictures, live audio, and text barrages will be generated.

[0043] Optionally, the timbre conversion request may include a target timbre style selected by the host or the audience entering the host's live broadcast room, and the target timbre style can be understood as the timbre style that the host or the audience entering the host's live broadcast room want to hear when listening to the aforementioned live audio. For example, the host may hope that the audio data output by him / her sounds like the timbre style of his / her favorite idol star, or the timbre style of a friend he / she is familiar with, or the timbre style of his / her favorite speaking accent. For another example, for some viewers, they may also hope that the audio data output by the host they hear sounds similar to the timbre style of their favorite idol star, or the timbre style of a friend they are familiar with. Based on this, the timbre conversion request can be issued by the host's corresponding host terminal 100, or by the client 300 of the audience entering the host's live broadcast room.

[0044] For example, a selection interface for the target timbre style may be provided in the interface of the live broadcast Internet application installed on the host terminal 100 or the client terminal 300. The selection interface displays multiple options for different timbre styles. The host or the audience entering the host's live broadcast room may select the option corresponding to the target timbre style they need from the various options displayed in the selection interface, and then generate a corresponding timbre conversion request through the host terminal 100 or the client terminal 300.

[0045] For example only, see Figure 3, shows a schematic diagram of the interface of the live broadcast Internet application installed on the anchor terminal 100 or the client 300, in which options of different timbre styles are displayed, including timbre style A, timbre style B, timbre style C, timbre style D, etc. The anchor or the audience entering the anchor's live broadcast room can select the option corresponding to the target timbre style they need from the selection interface. For example, the anchor likes the timbre style of a familiar friend A, and timbre style A is the timbre style of friend A, then the anchor can select timbre style A, and then generate a corresponding timbre conversion request through the anchor terminal 100. For another example, the audience in the anchor's live broadcast room likes the timbre style of singer Xiao Zhang, and timbre style B is the timbre style of singer Xiao Zhang, then the audience can select timbre style B, and then generate a corresponding timbre conversion request through the client 300.

[0046] The first audio data may be audio data pre-recorded by the host, or may be audio data output in real time during a live broadcast, and this embodiment does not impose any specific restrictions on this.

[0047] The inventor of the present application has found that any segment of audio data can be represented by a series of waveform graphs. Based on this, an exemplary method of executing step S110 to extract an audio feature graph can be: the first audio data is segmented at a preset time interval (for example, every 10 seconds) to obtain multiple audio segments, and then the sound wave graph, spectrum graph or spectrogram of each audio segment is extracted or the image after image processing and transformation of the sound wave graph, spectrum graph or spectrogram of each audio segment is used as the audio feature graph. Therefore, this embodiment can avoid the jamming of the host terminal 100 caused by the excessive amount of audio data processed at one time by segmenting the first audio data. On the other hand, the time length of each audio segment obtained by segmentation is consistent, which can facilitate subsequent processing.

[0048] The audio feature graph may include a content feature graph and a style feature graph. The style feature graph may be used to represent the style features of the first audio data, such as timbre style, etc.; the content feature graph may be used to represent the content features of the first audio data, such as volume, speech content, etc.

[0049] For step S120, the preset feature vector extraction network can use a convolutional neural network, which is a feedforward neural network whose artificial neurons can respond to surrounding units within a part of the coverage range, and has excellent performance in image processing. The convolutional neural network can complete object recognition by extracting the abstract features of the object through multi-layer convolution. Based on this, the content feature vector of the content feature map can be extracted by a convolutional neural network. Optionally, the preset feature vector extraction network can use a model such as a visual graphics generator (Visual Graphics Generator, VGG) model, a deep residual network (Deep Residual Network, ResNet) model, etc. for extracting vector features of images.

[0050] In this embodiment, at least one style transfer model for step S130 is pre-stored in the host terminal 100. Each style transfer model corresponds to a timbre style. Each style transfer model can be used to convert a content feature map of any host into a style transfer map with a target timbre style.

[0051] With respect to step S140, since the style conversion graph replaces the style feature graph in the original audio feature graph, the content feature graph in this step and the converted style conversion graph can be understood as an audio feature graph having the target timbre style. In order to generate audio data that the audience can hear, this embodiment also needs to perform feature inverse conversion on the content feature graph and the converted style conversion graph to obtain second audio data having the target timbre style. In this way, the second audio data combines the style features of the content feature graph corresponding to the first audio data and the converted style conversion graph, thereby achieving the auditory effect corresponding to the target timbre style without changing the content of the first audio data.

[0052] It is worth noting that, although some voice changing functions (such as the voice of an old person or a child) are used in the prior art to change the voice of a speaker, the sound effect of the conversion in this solution is not ideal, and a good realistic effect cannot be achieved, and the desired timbre style cannot be converted. Through the technical solution provided in this embodiment, the converted timbre is the timbre of the desired target timbre style, which has a very strong realistic effect.

[0053] It is also worth noting that since the style transfer model provided by this solution can learn the style feature vector of the corresponding timbre style, any content output by any anchor can be converted into a style transfer graph with the corresponding timbre style, without the need to train a style transfer model for each anchor separately, which greatly reduces the training workload. The specific training process for the style transfer model will be described in detail later.

[0054] With respect to step S140, in order to improve the fun of the live interactive process, a virtual image can be used in the display interface of the live broadcast room to replace the actual image of the anchor to interact with the audience. For example, the virtual image can imitate the anchor's expression, action and other characteristic attributes in real time, so as to interact with the audience on behalf of the anchor, that is, the audience can interact with the anchor through the virtual image, and the audience can be any one of the anchor's many subscribed fans. In addition, the first virtual image can imitate the anchor to perform operations or actions related to the anchor's content, such as holding a product, introducing a product, etc.

[0055] After the second audio data is generated, an interactive video frame of the virtual image corresponding to each audio frame in the second audio data can be generated in real time. For example, the emotional content or specific keywords in each audio frame in the second audio data can be identified, and then the virtual image can be controlled to perform interactive actions in the corresponding emotional expression form according to the emotional content, or the interactive expression form corresponding to the corresponding keyword can be searched according to the specific keyword to perform interactive actions, and the interactive video frame when the virtual image performs the interactive action can be recorded.

[0056] Next, each audio frame and its corresponding interactive video frame are synthesized to obtain an interactive video stream of the virtual image. For example, the text content contained in each audio frame can be parsed, and then the audio frame, the text content contained in the audio frame, and the interactive video frame corresponding to the audio frame are synthesized to obtain an interactive video stream of the virtual image corresponding to the anchor. On this basis, the interactive video stream of the virtual image can be sent to the client 300 through the live broadcast server 200 for playback.

[0057] For example, see Figure 4 , shows an example diagram of a live broadcast interface of the anchor terminal 100, in which the live broadcast interface may include a live broadcast interface display frame, an anchor video frame display frame, a bullet screen area, a virtual image area, and the text content of each audio frame of the anchor XXXXX. Among them, the live broadcast interface display frame is used to display the video stream currently being broadcast on the live broadcast platform or the complete video stream formed after the live broadcast is completed, the anchor video frame display frame is used to display the anchor video frame collected by the video acquisition device in real time, the virtual image area is used to display the anchor's virtual image and the interactive video frame of the virtual image, and the bullet screen area is used to display the interactive content between the audience and the anchor (such as AAAAA, BBBBB, CCCCC, DDDDD, EEEEE).

[0058] Understandably, Figure 4The live broadcast interface shown is for illustration only. During the actual live broadcast process, the live broadcast interface may also include a live broadcast information area. The live broadcast information area may include at least one of the following information: the name of the live broadcast room, the host user account, the host avatar, the audience user account, the audience avatar, the number of times the host has been followed, the host's popularity index, and a ranking of gifts received by the host.

[0059] In this way, this embodiment can convert the timbre style of the virtual image during the live broadcast into the target timbre style to interact with the audience without changing the audio content, thereby improving the interactive effect during the live broadcast and mobilizing the interaction between the audience and the anchor to a greater extent.

[0060] As a possible implementation, see Figure 5 Before the aforementioned step S110, the live broadcast method provided in this embodiment may further include the following steps:

[0061] Step S101: Pre-train a style transfer model corresponding to the target timbre style based on training samples. Figure 6 , step S101 may include the following sub-steps:

[0062] Sub-step S1011, obtaining training samples, where the training samples include a first audio sample and a second audio sample of any host.

[0063] In this embodiment, the first audio sample may be any audio sample having a target timbre style. For example, if the target timbre style is the timbre style of an actor A, a large amount of audio data of the actor A may be collected as the first audio sample.

[0064] In this embodiment, the second audio sample is not specifically limited, and can be audio data of any anchor or any other user, and can be collected as the second audio sample.

[0065] Please refer to Figure 7 The training process of this embodiment involves a feature extraction network, a feature vector extraction network, and an initial conversion network. Figure 7 The training process of the style transfer model in step S101 is exemplified.

[0066] Sub-step S1012, extracting a reference style feature map of the first audio sample and a content feature map of the second audio sample respectively.

[0067] See also Figure 7 As shown, the reference style feature map of the first audio sample and the content feature map of the second audio sample can be extracted through the feature extraction network in the manner of extracting the audio feature map from the first audio data input by the host.

[0068] Sub-step S1013, extracting the reference style feature vector corresponding to the reference style feature map and the content feature vector corresponding to the content feature map respectively through the feature vector extraction network.

[0069] Sub-step S1014, training an initial conversion model based on the content feature vector and the reference style feature vector, obtaining a style conversion model corresponding to the target timbre style, and storing it in the anchor terminal 100.

[0070] The following is based on Figure 7 The detailed training process of this sub-step S1014 is exemplified.

[0071] First, the content feature vector is input into the initial conversion model to generate a reference style transfer graph of the content feature vector.

[0072] Second, the reference style transfer feature vector corresponding to the reference style transfer image is extracted through the feature vector extraction network.

[0073] Third, the network parameters of the initial conversion model are adjusted according to the content feature vector, the reference style feature vector and the reference style conversion feature vector.

[0074] In detail, the present embodiment may calculate a first vector difference value between a reference style feature vector and a reference style conversion feature vector, and a second vector difference value between a reference style conversion feature vector and a content feature vector. Optionally, the calculation method of the first vector difference value and the second vector difference value may be: generating a content feature grayscale image corresponding to the content feature vector, a reference style grayscale image corresponding to the reference style feature vector, and a reference style conversion feature grayscale image corresponding to the reference style conversion feature vector.

[0075] Next, the pixel difference value between the reference style grayscale image and the reference style conversion feature grayscale image is calculated as the first vector difference value. For example, the grayscale difference between the grayscale pixel value of the pixel point in the reference style grayscale image and the grayscale pixel value of the pixel point at the corresponding position of the reference style conversion feature grayscale image can be calculated, and the square value of the difference between each pixel point in the reference style grayscale image and the corresponding position in the reference style conversion feature grayscale image is calculated. Then, the square values ​​of the difference corresponding to all the pixels are summed to obtain the pixel difference value between the reference style grayscale image and the reference style conversion feature grayscale image as the first vector difference value.

[0076] At the same time, the pixel difference value between the reference style conversion feature grayscale image and the content feature grayscale image is calculated as the second vector difference value. For example, the grayscale difference between the grayscale pixel value of the pixel point in the reference style conversion feature grayscale image and the grayscale pixel value of the pixel point at the corresponding position in the content feature grayscale image can be calculated, and the square value of the difference between each pixel point in the reference style conversion feature grayscale image and the pixel point at the corresponding position in the content feature grayscale image is calculated. Then, the square values ​​of the difference corresponding to all the pixels are summed to obtain the pixel difference value between the reference style conversion feature grayscale image and the content feature grayscale image as the second vector difference value.

[0077] It is worth noting that, in the actual training stage, those skilled in the art may also add other loss functions in addition to the aforementioned first vector difference value and second vector difference value, and this application does not impose any detailed restrictions on this.

[0078] Based on the above, back propagation training can be performed according to the first vector difference value and the second vector difference value, and the gradient of the network parameters of the initial conversion model can be calculated. Then, according to the calculated gradient, the network parameters of the initial conversion model are updated by the stochastic gradient descent method and the training is continued until the initial conversion model meets the training termination condition, and the style conversion model corresponding to the target timbre style obtained by training is output.

[0079] Among them, the calculation process of the stochastic gradient descent method is to solve the minimum value along the direction of gradient descent (it can also solve the maximum value along the direction of gradient ascent). The direction of gradient descent can be obtained by differentiating the function. In general, when the gradient vector is 0, it indicates that an extreme point has been reached, and the amplitude of the gradient is also 0. When the gradient descent algorithm is used for optimization, the termination condition of the algorithm iteration is that the amplitude of the gradient vector is close to 0, and a very small constant threshold can be set.

[0080] The above training termination condition may include at least one of the following three conditions:

[0081] 1) The number of iterative trainings reaches the set number; 2) The difference value of the first vector and the difference value of the second vector are lower than the set threshold; 3) The difference value of the first vector and the difference value of the second vector no longer decrease.

[0082] In addition, in the actual implementation process, it is not limited to using the above examples as training termination conditions. Those skilled in the art can design training termination conditions different from the above examples according to actual needs.

[0083] The style conversion model corresponding to the target timbre style obtained based on the above steps can be used to convert the content feature map corresponding to the audio data of any anchor into a style feature conversion map with the target timbre style. Without changing the audio content of the audio data of any anchor, the timbre style of the virtual image during the live broadcast is converted to the target timbre style to interact with the audience, thereby improving the interactive effect during the live broadcast and mobilizing the interaction between the audience and the anchor to a greater extent. In addition, the style conversion model corresponding to the target timbre style can be used for any audio content output by any anchor, and there is no need to train a style conversion model for each anchor separately, which greatly reduces the training workload.

[0084] It is worth noting that the above is only the training process of the style transfer model corresponding to the aforementioned target timbre style. For the training of style transfer models corresponding to other timbre styles, reference can be made to the relevant description of the above embodiment, which will not be repeated here.

[0085] Figure 8 A schematic diagram of an electronic device provided in an embodiment of the present application is shown. In this embodiment, the electronic device may refer to Figure 1 The host terminal 100 shown includes a storage medium 110, a processor 120, and a live interactive device 500. In this embodiment, the storage medium 110 and the processor 120 are both located in the host terminal 100 and are separately arranged. However, it should be understood that the storage medium 110 can also be independent of the host terminal 100 and can be accessed by the processor 120 through a bus interface. Alternatively, the storage medium 110 can also be integrated into the processor 120, for example, it can be a cache and / or a general register.

[0086] The storage medium 110, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the live interactive method described in any embodiment of the present application (for example, the extraction module 510, input module 520, conversion module 530, reverse conversion module 540 and generation module 550 included in the live interactive device 500). The storage medium 110 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application required for at least one function; the storage data area may store data created according to the use of the device, etc. In addition, the storage medium 110 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the storage medium 110 may further include a memory remotely arranged relative to the processor 120, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0087] The functions of each functional module of the live interactive device 500 are described in detail below.

[0088] The extraction module 510 is used to segment the first audio data input by the host at preset intervals according to the received timbre conversion request to obtain multiple audio segments, and extract audio feature graphs from the multiple audio segments, wherein the audio feature graphs include content feature graphs, and the timbre conversion request includes the target timbre style to which the timbre style of the first audio data needs to be converted. It can be understood that the extraction module 510 can be used to execute the above step S110, and the detailed implementation of the extraction module 510 can refer to the above content related to step S110.

[0089] The input module 520 is used to input the content feature map into a preset feature vector extraction network to extract the content feature vector of the content feature map. It can be understood that the input module 520 can be used to perform the above step S120. The detailed implementation of the input module 520 can refer to the above content related to step S120.

[0090] The conversion module 530 is used to convert the content feature vector using the style conversion model corresponding to the target timbre style to obtain a style conversion graph having the target timbre style. It can be understood that the conversion module 530 can be used to perform the above step S130, and the detailed implementation of the conversion module 530 can refer to the above content related to step S130.

[0091] The inverse conversion module 540 is used to perform feature inverse conversion on the content feature map and the style conversion map to obtain the second audio data having the target timbre style. It can be understood that the inverse conversion module 540 can be used to perform the above step S140, and the detailed implementation of the inverse conversion module 540 can refer to the above content related to step S140.

[0092] The generating and sending module 550 is used to generate an interactive video stream of the virtual image corresponding to the anchor according to the second audio data, and send it to the client 300 for playback. It can be understood that the generating and sending module 550 can be used to perform the above step S150, and the detailed implementation of the generating and sending module 550 can refer to the above content related to step S150.

[0093] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are executed, the live broadcast interactive method provided in the above embodiment is implemented.

[0094] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A live interactive method, characterized in that: Applied to a host terminal, the host terminal stores at least one style conversion model, each style conversion model corresponds to a timbre style, and the method includes: According to the received timbre conversion request, the first audio data input by the host is segmented at a preset time interval to obtain multiple audio segments, and an audio feature graph is extracted from the multiple audio segments, wherein the audio feature graph includes a content feature graph, and the content feature graph is used to represent the content feature of the first audio data, wherein the content feature includes volume and speech content, and the timbre conversion request includes a target timbre style; Inputting the content feature map into a preset feature vector extraction network to extract a content feature vector of the content feature map; The content feature vector is converted using a style conversion model corresponding to the target timbre style to obtain a style conversion graph having the target timbre style; Performing feature inverse conversion on the content feature map and the style conversion map to obtain second audio data having the target timbre style; An interactive video stream of the virtual image corresponding to the anchor is generated according to the second audio data, and sent to the client for playback.

2. The live interactive method according to claim 1, characterized in that: The style transfer model is obtained by training a neural network based on deep learning using a first audio sample and a second audio sample of any host, wherein the first audio sample has the target timbre style.

3. The live interactive method according to claim 1, characterized in that: Before the method of segmenting the first audio data input by the host at preset intervals according to the received timbre conversion request to obtain a plurality of audio segments and extracting the audio feature graph from the plurality of audio segments, the method further includes: Pre-training a style conversion model corresponding to the target timbre style based on training samples specifically includes: Acquire a training sample, wherein the training sample includes a first audio sample and a second audio sample of any host, wherein the first audio sample has the target timbre style; Respectively extracting a reference style feature map of the first audio sample and a content feature map of the second audio sample; Extracting the reference style feature vector corresponding to the reference style feature map and the content feature vector corresponding to the content feature map respectively through the feature vector extraction network; An initial conversion model is trained according to the content feature vector and the reference style feature vector to obtain a style conversion model corresponding to the target timbre style, and the style conversion model is stored in the anchor terminal.

4. The live interactive method according to claim 3, characterized in that: The step of training an initial conversion model according to the content feature vector and the reference style feature vector to obtain a style conversion model corresponding to the target timbre style comprises: Inputting the content feature vector into an initial conversion model to generate a reference style conversion graph of the content feature vector; Extracting a reference style conversion feature vector corresponding to the reference style conversion image through the feature vector extraction network; The network parameters of the initial conversion model are adjusted according to the content feature vector, the reference style feature vector, and the reference style conversion feature vector.

5. The live interactive method according to claim 4, characterized in that: The step of adjusting the network parameters of the initial conversion model according to the content feature vector, the reference style feature vector and the reference style conversion feature vector comprises: Calculating a first vector difference value between the reference style feature vector and the reference style conversion feature vector and a second vector difference value between the reference style conversion feature vector and the content feature vector; Performing back propagation training according to the first vector difference value and the second vector difference value, and calculating the gradient of the network parameters of the initial conversion model; According to the calculated gradient, the network parameters of the initial conversion model are updated by the stochastic gradient descent method and the training is continued until the initial conversion model meets the training termination condition, and the style conversion model corresponding to the target timbre style obtained by training is output.

6. The live interactive method according to claim 5, characterized in that: The step of calculating a first vector difference value between the reference style feature vector and the reference style conversion feature vector and a second vector difference value between the reference style conversion feature vector and the content feature vector comprises: Generate a content feature grayscale image corresponding to the content feature vector, a reference style grayscale image corresponding to the reference style feature vector, and a reference style conversion feature grayscale image corresponding to the reference style conversion feature vector; A pixel difference value between the reference style grayscale image and the reference style conversion feature grayscale image is calculated as the first vector difference value, and a pixel difference value between the reference style conversion feature grayscale image and the content feature grayscale image is calculated as the second vector difference value.

7. The live interactive method according to claim 6, characterized in that: The step of calculating the pixel difference value between the reference style grayscale image and the reference style conversion feature grayscale image as the first vector difference value comprises: Calculate the grayscale difference between the grayscale pixel value of the pixel point in the reference style grayscale image and the grayscale pixel value of the pixel point at the corresponding position of the reference style conversion feature grayscale image, and calculate the square value of the difference between each pixel point in the reference style grayscale image and the corresponding position in the reference style conversion feature grayscale image; The squared difference values ​​corresponding to all pixel points are summed to obtain a pixel difference value between the reference style grayscale image and the reference style conversion feature grayscale image; The step of calculating a pixel difference value between the reference style conversion feature grayscale image and the content feature grayscale image as the second vector difference value comprises: Calculate the grayscale difference between the grayscale pixel values ​​of the pixels in the reference style conversion feature grayscale image and the grayscale pixel values ​​of the pixels at the corresponding positions in the content feature grayscale image, and calculate the square value of the difference between each pixel in the reference style conversion feature grayscale image and the pixel at the corresponding position in the content feature grayscale image; The squared difference values ​​corresponding to all pixel points are summed to obtain the pixel difference value between the reference style conversion feature grayscale image and the content feature grayscale image.

8. The live interactive method according to any one of claims 1 to 7, characterized in that: The step of generating an interactive video stream of the virtual image corresponding to the anchor according to the second audio data, and sending the interactive video stream to the client for playing includes: For each audio frame in the second audio data, generating an interactive video frame of the virtual image corresponding to the audio frame; Each audio frame and its corresponding interactive video frame are synthesized to obtain an interactive video stream of the virtual image, and the interactive video stream of the virtual image is sent to a client for playback.

9. The live interactive method according to claim 8, characterized in that: The step of synthesizing each audio frame and its corresponding interactive video frame to obtain the interactive video stream of the virtual image includes: For each audio frame, parse out the text content contained in the audio frame; The audio frame, the text content contained in the audio frame and the interactive video frame corresponding to the audio frame are synthesized to obtain an interactive video stream of the virtual image corresponding to the anchor.

10. A live interactive device, characterized in that: Applied to a host terminal, the host terminal stores at least one style conversion model, each style conversion model corresponds to a timbre style, and the device includes: an extraction module, configured to segment the first audio data input by the host at preset intervals according to the received timbre conversion request to obtain a plurality of audio segments, and extract an audio feature graph from the plurality of audio segments, wherein the audio feature graph includes a content feature graph, and the content feature graph is used to represent content features of the first audio data, wherein the content features include volume and speech content, and the timbre conversion request includes a target timbre style to which the timbre style of the first audio data needs to be converted; An input module, used for inputting the content feature map into a preset feature vector extraction network to extract a content feature vector of the content feature map; A conversion module, configured to convert the content feature vector using a style conversion model corresponding to the target timbre style to obtain a style conversion graph having the target timbre style; A reverse conversion module, used for performing feature reverse conversion on the content feature map and the style conversion map to obtain second audio data having the target timbre style; A sending module is generated, which is used to generate an interactive video stream of the virtual image corresponding to the anchor according to the second audio data, and send it to the client for playback.

11. An electronic device, characterized in that: The electronic device includes one or more storage media and one or more processors communicating with the storage media, and the one or more storage media store machine executable instructions executable by the processor. When the electronic device is running, the processor executes the machine executable instructions to implement the live interactive method described in any one of claims 1-9.

12. A readable storage medium, characterized in that: The readable storage medium stores machine executable instructions, and when the machine executable instructions are executed, the live broadcast interactive method described in any one of claims 1-9 is implemented.

Images