Interaction information positioning method

Abstract

The invention discloses an interaction information positioning method. The method is applied to first equipment, and comprises the following steps: under the condition that input including audio tag information is received, sending an audio acquisition request to second equipment, the audio acquisition request including the audio tag information; audio data corresponding to the audio tag information sent by the second device and target association information of mute frames and interaction information in the audio data are received, and the target association information is association information correspondingly generated by a third device according to the sequence of the interaction information and the sequence of the mute frames; obtaining the position of a mute frame in an audio frame corresponding to the audio data; and determining the position of the interaction information according to the target associated information and the position of the mute frame. According to the embodiment of the invention, the user can perform the interaction action in the mute time, the flexibility is improved, and through the target association information of the mute frame and the interaction information, the manual operation steps are reduced, the labor cost is reduced, and the accuracy is improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular, relates to a method for locating interactive information. Background Art

[0002] In recent years, audio applications have been widely used, and how to interact with users in audio has become one of the research hotspots. Currently, during audio playback, there are interactive actions such as video pop-ups, displaying pictures or texts. If you want to achieve interactive actions, you need to insert interactive information into the audio data to display the interactive information.

[0003] In the prior art, interactive information is generally displayed at a specific position of an audio frame, which has poor flexibility and requires manual confirmation of the position of the interactive information before display to check whether the position of the interactive information is correct. The manual operation cost is high and the accuracy is low. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, device and computer storage medium for locating interactive information, which inserts interactive information into a silent frame, and determines the position of the interactive information based on target association information between the silent frame and the interactive information, so that the user can perform interactive actions during the silent time. For different audio data, the position of the interactive information can be different and changes based on changes in the silent frame, thereby improving flexibility. In addition, by establishing target association information between the silent frame and the interactive information, the manual operation steps are reduced, the labor cost is reduced, and the accuracy is improved.

[0005] In a first aspect, an embodiment of the present application provides a method for locating interaction information, which is applied to a first device and includes:

[0006] When receiving an input including audio tag information, sending an audio acquisition request to a second device, where the audio acquisition request includes the audio tag information;

[0007] Receive audio data corresponding to the audio tag information sent by the second device, and target association information between silent frames and interactive information in the audio data, where the target association information is that after the third device converts the acquired audio data into an audio signal, it performs fast Fourier transform on sub-audio signals of multiple audio frames obtained by framing the audio signal according to a preset time length to obtain a short-time energy value and a spectrum amplitude of each audio frame, identifies silent frames in the multiple audio frames based on the short-time energy value and the spectrum amplitude, determines the position of the interactive information in the audio frame corresponding to the audio data according to the order of the interactive information and the order of the silent frames, and generates the association information accordingly;

[0008] Get the position of the silent frame in the audio frame corresponding to the audio data;

[0009] The position of the interactive information is determined according to the target association information and the position of the silent frame.

[0010] In a possible implementation, after determining the position of the interaction information according to the target association information and the position of the silence frame, the method further includes:

[0011] receiving an interaction instruction for first target interaction information sent by the second device, where the first target interaction information is any one of the interaction information, and the interaction instruction is an instruction sent from the second device when the interaction requirement information of the interaction information indicates that the first target interaction information needs to interact with the user;

[0012] The first target interaction information is displayed at the position of the first target interaction information.

[0013] In a possible implementation, after determining the position of the interaction information according to the target association information and the position of the silence frame, the method further includes:

[0014] Play the audio data and display the interactive information at the location of the interactive information;

[0015] In a case where it is detected that the second target interaction information in the interaction information includes verification request information, displaying the verification request information at the location of the second target interaction information;

[0016] When receiving the verification response information input by the user, the verification response information is sent to the second device, so that the second device verifies the verification response information. When the verification response information satisfies the preset condition, the key for encrypting the audio data is sent to the first device, where the encrypted audio data is the audio data corresponding to the audio frame after the position of the second target interaction information, and is obtained by the third device encrypting the target audio data in the audio data using the key;

[0017] receiving a key sent by the second device, and decrypting the encrypted audio data using the key to obtain decrypted audio data;

[0018] Play the decrypted audio data.

[0019] In a second aspect, an embodiment of the present application provides a method for locating interaction information, which is applied to a second device, including:

[0020] receiving an audio acquisition request sent by the first device, where the audio acquisition request includes audio tag information;

[0021] Acquire audio data corresponding to the audio tag information, and target association information between silent frames and interactive information in the audio data, where the target association information is that after the third device converts the acquired audio data into an audio signal, it performs fast Fourier transform on sub-audio signals of multiple audio frames obtained by framing the audio signal according to a preset time length to obtain a short-time energy value and a spectrum amplitude of each audio frame, identifies silent frames in the multiple audio frames based on the short-time energy value and the spectrum amplitude, determines the position of the interactive information in the audio frame corresponding to the audio data according to the order of the interactive information and the order of the silent frames, and generates the corresponding association information;

[0022] The audio data and the target association information are sent to the first device so that the first device can obtain the position of the silent frame in the audio frame corresponding to the audio data, and determine the position of the interactive information according to the target association information and the position of the silent frame.

[0023] In one possible implementation, after sending the audio data and the target association information to the first device, the method further includes:

[0024] Acquire interaction requirement information of interaction information;

[0025] When the interaction demand information represents that the first target interaction information needs to interact with the user, an interaction instruction for the first target interaction information is sent to the first device, and the interaction instruction is used to instruct the first device to display the first target interaction information at the position of the first target interaction information, and the first target interaction information is any one of the interaction information.

[0026] In one possible implementation, after sending the audio data and the target association information to the first device, the method further includes:

[0027] Receiving verification response information sent by the first device, where the verification response information is verification response information of the verification request information input by the user received by the first device when the first device detects that the second target interaction information in the interaction information includes the verification request information;

[0028] The verification response information is verified. When the verification response information meets the preset conditions, the key for the encrypted audio data is sent to the first device, so that the first device uses the key to decrypt the encrypted audio data, obtains the decrypted audio data, and plays the decrypted audio data. The encrypted audio data is the audio data corresponding to the audio frame after the position of the second target interaction information, which is obtained by the third device using the key to encrypt the target audio data in the audio data.

[0029] In a third aspect, an embodiment of the present application provides a method for locating interaction information, which is applied to a third device, including:

[0030] Acquire audio data, audio tag information and interaction information of the audio data, and convert the audio data into an audio signal;

[0031] Divide the audio signal into frames according to a preset time length to obtain sub-audio signals of multiple audio frames;

[0032] Performing a fast Fourier transform on the sub-audio signal of each audio frame to obtain spectrum feature information of each audio frame, where the spectrum feature information includes a short-time energy value and a spectrum amplitude;

[0033] Based on the short-time energy value and the spectrum amplitude, a silent frame is identified among the multiple audio frames;

[0034] According to the order of the interactive information and the order of the silent frames, determine the position of the interactive information in the audio frame corresponding to the audio data, and generate target association information of the silent frame and the interactive information accordingly;

[0035] Audio data, audio tag information of the audio data, and target association information are sent to the second device, so that when the second device receives an audio acquisition request sent by the first device, the second device sends the audio data and the target association information to the first device, so that the first device can obtain the position of the silent frame in the audio frame corresponding to the audio data, and determine the position of the interactive information according to the target association information and the position of the silent frame, wherein the audio acquisition request includes the audio tag information.

[0036] In a possible implementation, before performing a fast Fourier transform on the sub-audio signal corresponding to each audio frame to obtain the frequency spectrum feature information of each audio frame, the method further includes:

[0037] Performing windowing processing on the sub-audio signal corresponding to each audio frame to obtain a target sub-audio signal corresponding to each audio frame;

[0038] Perform fast Fourier transform on the sub-audio signal corresponding to each audio frame to obtain the spectrum feature information of each audio frame, including:

[0039] Perform fast Fourier transform on the target sub-audio signal corresponding to each audio frame to obtain the spectrum feature information of each audio frame.

[0040] In a possible implementation, based on the short-time energy value and the spectrum amplitude, identifying the silent frame in the plurality of audio frames includes:

[0041] For a target audio frame among the multiple audio frames, sliding windows are respectively set before and after the target audio frame, and the target audio frame is any one of the multiple audio frames;

[0042] When the average short-time energy value of the audio frames in the sliding window is less than the first preset value and the short-time energy value of the target audio frame is not less than the first preset value, updating the short-time energy value of the target audio frame to the target short-time energy value to obtain a plurality of first smoothed audio frames;

[0043] Divide each audio frame into multiple frequency levels based on the Mel scale;

[0044] Smoothing the frequency spectrum amplitude corresponding to each audio frame of each frequency level to obtain a plurality of second smoothed audio frames corresponding to a plurality of frequency levels;

[0045] Based on the short-time energy values ​​of the plurality of first smooth audio frames, identifying a first sub-silent frame in the plurality of first smooth audio frames, and based on the spectrum amplitudes of the plurality of second smooth audio frames, identifying a second sub-silent frame in the plurality of second smooth audio frames;

[0046] A silence frame among the plurality of audio frames is determined according to the first sub silence frame and the second sub silence frame.

[0047] In a possible implementation, the audio data includes encrypted audio data, the interaction information includes second target interaction information, and the second target interaction information includes verification request information; before sending the audio data, the audio tag information of the audio data, and the target association information to the second device, the method further includes:

[0048] The target audio data in the audio data is encrypted using a key to obtain encrypted audio data, and second target interaction information is generated, where the encrypted audio data is audio data corresponding to an audio frame after the position of the second target interaction information.

[0049] In a fourth aspect, an embodiment of the present application provides a positioning device for interaction information, which is applied to a first device, including:

[0050] A sending module, configured to send an audio acquisition request to a second device when receiving an input including audio tag information, wherein the audio acquisition request includes the audio tag information;

[0051] a receiving module, configured to receive audio data corresponding to the audio tag information sent by the second device, and target association information between silent frames and interactive information in the audio data, wherein the target association information is that after the third device converts the acquired audio data into an audio signal, it performs fast Fourier transform on sub-audio signals of multiple audio frames obtained by framing the audio signal according to a preset time length, obtains a short-time energy value and a spectrum amplitude of each audio frame, identifies silent frames in the multiple audio frames based on the short-time energy value and the spectrum amplitude, determines the position of the interactive information in the audio frame corresponding to the audio data according to the order of the interactive information and the order of the silent frames, and generates the association information accordingly;

[0052] An acquisition module, used to acquire the position of a silent frame in an audio frame corresponding to the audio data;

[0053] The determination module is used to determine the position of the interaction information according to the target association information and the position of the silence frame.

[0054] In a fifth aspect, an embodiment of the present application provides a positioning device for interaction information, which is applied to a second device, including:

[0055] A receiving module, configured to receive an audio acquisition request sent by a first device, where the audio acquisition request includes audio tag information;

[0056] an acquisition module, used to acquire audio data corresponding to the audio tag information, and target association information between silent frames and interactive information in the audio data, wherein the target association information is that after the third device converts the acquired audio data into an audio signal, a fast Fourier transform is performed on sub-audio signals of multiple audio frames obtained by framing the audio signal according to a preset time length to obtain a short-time energy value and a spectrum amplitude of each audio frame, and based on the short-time energy value and the spectrum amplitude, silent frames in the multiple audio frames are identified, and the position of the interactive information in the audio frame corresponding to the audio data is determined according to the order of the interactive information and the order of the silent frames, and the corresponding association information is generated;

[0057] The sending module is used to send audio data and target association information to the first device, so that the first device obtains the position of the silent frame in the audio frame corresponding to the audio data, and determines the position of the interactive information according to the target association information and the position of the silent frame.

[0058] In a sixth aspect, an embodiment of the present application provides a positioning device for interaction information, which is applied to a third device, including:

[0059] An acquisition module, used to acquire audio data, audio tag information and interaction information of the audio data, and convert the audio data into an audio signal;

[0060] A framing module, used to frame the audio signal according to a preset duration to obtain sub-audio signals of multiple audio frames;

[0061] A transformation module, used for performing a fast Fourier transform on the sub-audio signal of each audio frame to obtain the spectrum feature information of each audio frame, where the spectrum feature information includes a short-time energy value and a spectrum amplitude;

[0062] An identification module, used for identifying a silent frame among a plurality of audio frames based on a short-time energy value and a spectrum amplitude;

[0063] A determination module, used to determine the position of the interaction information in the audio frame corresponding to the audio data according to the order of the interaction information and the order of the silence frame, and to generate target association information of the silence frame and the interaction information accordingly;

[0064] A sending module is used to send audio data, audio tag information of the audio data and target association information to a second device, so that when the second device receives an audio acquisition request sent by the first device, the second device sends the audio data and target association information to the first device, so that the first device obtains the position of a silent frame in an audio frame corresponding to the audio data, and determines the position of the interactive information according to the target association information and the position of the silent frame, wherein the audio acquisition request includes the audio tag information.

[0065] In a seventh aspect, an embodiment of the present application provides an electronic device, the device comprising:

[0066] a processor and a memory storing computer program instructions;

[0067] When the processor executes the computer program instructions, any one of the above-mentioned methods for locating the interactive information is implemented.

[0068] In an eighth aspect, an embodiment of the present application provides a computer storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, any one of the above-mentioned methods for locating interactive information is implemented.

[0069] In a ninth aspect, an embodiment of the present application provides a computer program product, characterized in that when the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to execute any one of the above-mentioned methods for locating interactive information.

[0070] The method, apparatus, device and computer storage medium for locating interactive information of an embodiment of the present application are applied to a first device, comprising: upon receiving an input including audio tag information, sending an audio acquisition request to a second device, the audio acquisition request including audio tag information; receiving audio data corresponding to the audio tag information sent by the second device, and target association information between silent frames and interactive information in the audio data, the target association information being that after a third device converts the acquired audio data into an audio signal, a fast Fourier transform is performed on sub-audio signals of multiple audio frames obtained by framing the audio signal according to a preset time length to obtain a short-time energy value and a spectrum amplitude of each audio frame, based on the short-time energy value and the spectrum amplitude, silent frames in multiple audio frames are identified, and the position of the interactive information in the audio frame corresponding to the audio data is determined according to the order of the interactive information and the order of the silent frames, and the corresponding association information is generated; the position of the silent frame in the audio frame corresponding to the audio data is obtained; and the position of the interactive information is determined according to the target association information and the position of the silent frame.

[0071] In this way, the interactive information is inserted into the silent frame, and the position of the interactive information is determined based on the target association information between the silent frame and the interactive information, so that the user can perform interactive actions during the silent time. For different audio data, the position of the interactive information can be different, and it changes based on the change of the silent frame, which improves flexibility. Moreover, by establishing the target association information between the silent frame and the interactive information, the manual operation steps are reduced, the labor cost is reduced, and the accuracy is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0073] Figure 1 It is a flowchart of a method for locating interactive information provided by an embodiment of the present application;

[0074] Figure 2 is a schematic diagram of a time-frequency spectrum provided by another embodiment of the present application;

[0075] Figure 3 is a schematic diagram of spectrum amplitude provided by yet another embodiment of the present application;

[0076] Figure 4 is a schematic diagram of the short-time energy value of each audio frame provided by yet another embodiment of the present application;

[0077] Figure 5 is a flowchart of a method for locating interactive information provided by yet another embodiment of the present application;

[0078] Figure 6 is a flowchart of a method for locating interactive information provided by yet another embodiment of the present application;

[0079] Figure 7 is a flowchart of a method for locating interactive information provided by yet another embodiment of the present application;

[0080] Figure 8 is a flowchart of a method for locating interactive information provided by yet another embodiment of the present application;

[0081] Fig. 9 is a schematic diagram of a sliding window construction provided by yet another embodiment of the present application;

[0082] Fig.10 is a schematic diagram of spectrum amplitude smoothing processing provided by yet another embodiment of the present application;

[0083] Fig.11is a structural diagram of a positioning device for interactive information of a first device provided in yet another embodiment of the present application;

[0084] Fig.12 is a structural diagram of a positioning device for interactive information of a second device provided in yet another embodiment of the present application;

[0085] Fig.13 is a structural diagram of a positioning device for interactive information of a third device provided in yet another embodiment of the present application;

[0086] Fig.14 It is a structural diagram of an electronic device provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0087] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0088] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0089] In recent years, audio applications have been widely used, and how to interact with users in audio has become one of the research hotspots. Currently, during audio playback, there are interactive actions such as video pop-ups, displaying pictures or texts. If you want to achieve interactive actions, you need to insert interactive information into the audio data to display the interactive information.

[0090] In the prior art, interactive information is generally displayed at a specific position of an audio frame, which has poor flexibility and requires manual confirmation of the position of the interactive information before display to check whether the position of the interactive information is correct. The manual operation cost is high and the accuracy is low.

[0091] In order to solve the problems of the prior art, the embodiments of the present application provide a method, device, equipment and computer storage medium for locating interactive information. The method for locating interactive information provided in the embodiments of the application can be applied to audio and video playback and cloud note module applications in a cloud disk client. The interactive information is inserted into a silent frame, and the location of the interactive information is determined based on the target association information between the silent frame and the interactive information, so that the user can perform interactive actions during the silent time. For different audio data, the location of the interactive information can be different, and it changes based on the change of the silent frame, which improves flexibility. In addition, by establishing the target association information between the silent frame and the interactive information, the manual operation steps are reduced, the labor cost is reduced, and the accuracy is improved.

[0092] The following is an introduction to the method for locating the interactive information provided in the embodiments of the present application. Figure 1 FIG. 1 is a flow chart showing a method for locating interactive information provided by an embodiment of the present application. Figure 1 As shown, before the first device receives input including audio tag information, a method for locating interaction information provided by an embodiment of the present application includes the following steps.

[0093] S110. The third device obtains audio data, audio tag information and interaction information of the audio data, and converts the audio data into an audio signal.

[0094] Here, audio data is collected by a recording device. Audio is recorded by the audio recording hardware and recording application of the recording device.

[0095] In some embodiments, a recording device can be set in a third device, and audio is collected through the recording device. The audio data can also be recorded by a separate recording device, sent to the third device in advance, and saved by the third device. When the recording application is recording, the recording application performs audio sampling, and returns the audio data of a short period of time in the past through the audio callback function. The audio data is generally a linear pulse code (line-Pulse Code Modulation, line-PCM) file, which includes equal-sized PCM segments, and each equal-sized PCM segment has the same recording time. The PCM segment is converted into an audio signal, and the audio signal is an audio time domain signal.

[0096] Specifically, the audio tag information of the audio data may be a simple description of the audio data, such as type and name, etc. The interactive information may include, but is not limited to, interactive information corresponding to interactive actions such as video title, displayed pictures or texts.

[0097] S120: The third device divides the audio signal into frames according to a preset time length to obtain sub-audio signals of multiple audio frames.

[0098] Here, the preset duration may be set in advance.

[0099] In some embodiments, for an audio signal, framing is performed according to a custom framing rule F1. The custom framing rule F1 here presets the duration of framing. For example, the duration of each frame is determined by 20 milliseconds, and the specific duration is manually determined before framing. The framing time of an audio time domain signal is usually between 20 and 50 milliseconds.

[0100] S130: The third device performs fast Fourier transform on the sub-audio signal of each audio frame to obtain spectrum feature information of each audio frame, where the spectrum feature information includes a short-time energy value and a spectrum amplitude.

[0101] In some embodiments, after the third device performs fast Fourier transform on the sub-audio signal of each audio frame, a time-frequency spectrum with time as the horizontal axis and frequency as the vertical axis can be obtained (the time-frequency spectrum is as follows: Figure 2 Based on the frequency of Mel scale 0-16k, the frequency band is divided according to the division rule F2, and the spectrum amplitude of different frequency bands in each frame can be obtained. The spectrum amplitude diagram is shown in Figure 3 As shown, 11 frequency bands are divided, where the upper frequency point of the frequency band is f(n) times the lower frequency point. The purpose of frequency band division is to better quantify the audio frequency and perform feature extraction. The division rule F2 here is set in advance based on the Mel scale.

[0102] It is understandable that, according to psychoacoustics, dividing equal frequency intervals into logarithmic intervals is more in line with human hearing. Mel scale is one of the audio frequency band division methods.

[0103] S140. The third device identifies silent frames among the multiple audio frames based on the short-time energy value and the spectrum amplitude.

[0104] In some embodiments, the third device identifies a silent frame in the plurality of audio frames based on the short-time energy value and the spectrum amplitude, which may specifically include:

[0105] In some embodiments, the silence frames determined based on the short-time energy value and the spectrum amplitude are combined to finally obtain the silence frames in the multiple audio frames. Here, the combination can be a union or an intersection.

[0106] Specifically, for the short-time energy value, the short-time energy value of each audio frame is compared with the first silent frame threshold, the absolute value of the difference between the short-time energy value of each audio frame and the first silent frame threshold is calculated (Euclidean distance can be used), and the audio frame whose absolute value is less than the first silent frame threshold is determined to be a silent frame. Here, the first silent frame threshold can be set in advance. As an example, the first silent frame threshold is a preset multiple of the average short-time energy value of multiple audio frames, such as 80%. The short-time energy value of each audio frame is as follows: Figure 4 shown.

[0107] For the spectrum amplitude, since each audio frame is divided into N (such as 11) frequency bands (N frequency bands belong to different frequency ranges), the spectrum amplitude of the N frequency bands corresponding to each frame time can correspond to an N-dimensional vector. Therefore, the silent frame is identified by calculating the cosine similarity of two adjacent frames. If the cosine similarity is less than the second silent frame threshold, it is determined that the previous frame is a silent frame, or the next frame is a silent frame, or it is determined that both frames are silent frames. Here, the second silent frame threshold can be set in advance.

[0108] In some embodiments, the locations of silence frames are recorded.

[0109] S150. The third device determines the position of the interactive information in the audio frame corresponding to the audio data according to the order of the interactive information and the order of the silent frame, and generates target association information of the silent frame and the interactive information accordingly.

[0110] Here, the order of interactive information is set in advance.

[0111] Specifically, before audio collection, the user determines the interactive information that needs to be inserted into the silent frame, such as pop-up click interaction, voice recording feedback, and designated information displayed to the audience. After the silent frame is detected, the interactive information is automatically associated with the silent frame interval in sequence to generate target-related information.

[0112] S160. The third device sends the audio data, the audio tag information of the audio data, and the target association information to the second device.

[0113] Here, the second device may be a server or a voice platform. The second device stores the audio data, the audio tag information of the audio data, and the target association information sent by the third device.

[0114] It should be noted that, during subsequent audio playback, the target association information is matched to the silent frame interval of the audio through the client requesting the server interface, and the corresponding interactive action is implemented on the client.

[0115] Specifically, after the audio is collected, if the audio is confirmed to be valid, the audio tag information, audio interaction requirement information, silent frame position and interaction information, etc. can be reported to the server in real time through the interface. At this point, the audio collection process is completed. Here, the audio is valid means that there are no abnormalities in the collection process, or the audio data does not contain preset information, etc.

[0116] Then, if Figure 5 As shown, a method for locating interactive information provided by another embodiment of the present application includes the following steps.

[0117] S210: When receiving an input including audio tag information, the first device sends an audio acquisition request to the second device, where the audio acquisition request includes the audio tag information.

[0118] Here, the first device may be a client device. The input including the audio tag information may be a user input. The first device may obtain and play the audio data from the second device via a socket or an http long link. Playing includes online playing and offline playing.

[0119] S220: The second device obtains audio data corresponding to the audio tag information, and target association information between silent frames and interactive information in the audio data.

[0120] It is understandable that the second device not only stores an audio data packet, but also searches for audio data corresponding to the audio tag information after receiving the audio tag information sent by the first device. It is understandable that the audio data corresponding to the audio tag information is the audio data that the user wants to play.

[0121] S230: The second device sends audio data and target association information to the first device.

[0122] In some embodiments, during online playback or offline playback, the second device sends the interaction information and target association information of the silent frame to the first device.

[0123] S240. The first device obtains a position of a silent frame in an audio frame corresponding to the audio data.

[0124] In some embodiments, the first device may obtain the position of the silence frame from the second device.

[0125] In some embodiments, the first device may also identify a silence frame by utilizing the step of determining a silence frame by the third device.

[0126] S250. The first device determines a position of the interaction information according to the target association information and a position of the silence frame.

[0127] It can be understood that the position of the silent frame is the display position of the interactive information.

[0128] In this way, the interactive information is inserted into the silent frame, and the position of the interactive information is determined based on the target association information between the silent frame and the interactive information, so that the user can perform interactive actions during the silent time. For different audio data, the position of the interactive information can be different, and it changes based on the change of the silent frame, which improves flexibility. In addition, by establishing the target association information between the silent frame and the interactive information, the manual operation steps are reduced, the labor cost is reduced, and the accuracy is improved. In addition, the accuracy of audio interaction is improved, and the coverage scenario of audio interaction can be expanded.

[0129] Based on this, in some embodiments, such as Figure 6 As shown, after the above S250, the method may further include:

[0130] S251, the second device obtains interaction requirement information of the interaction information;

[0131] S252: When the interaction requirement information indicates that the first target interaction information needs to interact with the user, the second device sends an interaction instruction for the first target interaction information to the first device, where the first target interaction information is any one of the interaction information;

[0132] S253: The first device displays the first target interaction information at the location of the first target interaction information.

[0133] Here, the interaction requirement information may be information input by the user through a third device and is set in advance.

[0134] In some embodiments, the interaction requirement information may be used as audio interaction trigger entry information, which may be understood as the position where the interaction information starts to be displayed or the range where the interaction information is displayed.

[0135] In some embodiments, when the interaction demand information indicates that the first target interaction information needs to interact with the user, the second device sends an interaction instruction for the first target interaction information to the first device, instructing the first device to display the first target interaction information at the location of the first target interaction information.

[0136] It is understandable that if the interaction requirement information of all the interaction information indicates that there is no need to interact with the user, then during the audio playback, there is no need to display the interaction information.

[0137] In this way, only when the interaction demand information indicates the need to interact with the user, the second device will send an interaction instruction for the interaction information to the first device to display the interaction information, thereby making the display of the interaction information more flexible and improving the user experience.

[0138] Based on this, in some embodiments, such as Figure 7As shown, after the above S250, the method may further include:

[0139] S310: The first device plays audio data and displays interaction information at the location of the interaction information;

[0140] S320: When the first device detects that the second target interaction information in the interaction information includes the verification request information, the first device displays the verification request information at the location of the second target interaction information;

[0141] S330: When receiving the verification response information input by the user, the first device sends the verification response information to the second device;

[0142] S340, the second device verifies the verification response information, and sends the key for encrypting the audio data to the first device if the verification response information meets the preset condition, where the encrypted audio data is the audio data corresponding to the audio frame after the position of the second target interaction information, and is obtained by the third device encrypting the target audio data in the audio data using the key;

[0143] S350, the first device decrypts the encrypted audio data using the key to obtain decrypted audio data;

[0144] S360: The first device plays the decrypted audio data.

[0145] In some embodiments, the interaction instruction of the second target interaction information includes an authentication instruction, or the data has been encrypted and requires authentication before decryption. After the first device receives the authentication instruction, the second target interaction information is displayed at the location of the second target interaction information while performing authentication.

[0146] In some embodiments, the verification request information in the second target interaction information is displayed to the user so that the user can input the corresponding verification response information. When the first device receives the verification response information input by the user, it sends the verification response information to the second device. The second device verifies the verification response information, and when the verification response information meets the preset conditions, it sends the key for encrypting the audio data to the first device, and the first device decrypts the encrypted audio data using the key to obtain the decrypted audio data.

[0147] In some embodiments, the verification method may include: number authentication or SIM authentication. The verification response information may be a token representing the user identity, and the token is a single-point authentication token. The second device verifies the verification response information, and when the verification response information meets the preset conditions, sends the key for encrypting the audio data to the first device, including:

[0148] The second device verifies the token. If the token is in the trust relationship list, the verification is successful and the key for encrypting the audio data is sent to the first device. If the token is not in the trust relationship list, the verification fails. At this time, the audio playback in the first device will be abnormal and the verification request information will be displayed again. The verification response information corresponding to the verification request information needs to wait for the audio author or administrator to allow it to return the audio data normally and proceed to the next step of interaction.

[0149] In some embodiments, the verification response information may also carry encrypted audio data. If the second device passes the verification response information, the second device decrypts the encrypted audio data using the key and returns the decrypted audio data to the first device. The first device plays the decrypted audio data.

[0150] As an example, scenarios that require authentication may include but are not limited to:

[0151] 1. When the user plays audio online on the first device, after the first device detects the silent frame of the online audio clip, it determines the location of the corresponding interactive information based on the interactive information synchronously carried when the second device returns the audio clip. If the interactive information requires authentication of subsequent audio data (including verification request information), the audio stops playing at this time, and requests SIM authentication or number authentication to the second device, and the second device uses the authentication system for authentication. If it is SIM authentication, it is necessary to apply for authentication to the audio creator or specific user, and the device used by the audio creator or specific user will pop up the SIM authentication confirmation; if it is number authentication, the corresponding number is transmitted to the second device for user trust relationship list authentication.

[0152] Taking SIM authentication as an example, if the SIM authentication is successful, the authentication system notifies the second device through the authentication success information to decrypt the encrypted audio data and return the playable audio clip to the first device for continued playback; if the SIM authentication fails, the first device does not continue to play the audio. The user manually pulls the progress to the latter part of the audio. If the second device cannot detect the authentication success information, it will not return the correct audio data to the first device. Send an error field to the first device, and the first device pops up a prompt.

[0153] 2. When the user plays offline audio on the first device, he requests the target association information of audio data and silent frames and interactive information from the second device in advance. After the first device detects the silent frame in the offline audio, it determines the location of the corresponding interactive information according to the target association information. If the interactive information requires authentication of subsequent audio data, the audio stops playing and requests SIM authentication or number authentication from the second device. The second device uses the authentication system for authentication. If it is SIM authentication, it is necessary to apply for authentication from the audio creator or specific user, and the device used by the audio creator or specific user will pop up the SIM authentication confirmation; if it is number authentication, the corresponding number will be transmitted to the second device for user trust relationship list authentication.

[0154] Taking SIM authentication as an example, if the SIM authentication is passed, there are two ways to decrypt:

[0155] a. The authentication system notifies the second device of the successful authentication result, the second device sends the key to the first device, and the first device decrypts the offline audio file and plays it; if the next interactive information is found after decryption and requires subsequent audio data authentication access, the authentication process is triggered again.

[0156] b. The first device uploads an offline audio data segment, and the second device decrypts the audio data segment using a key and returns the decrypted audio data segment to the first device for playback; if the next interactive information is found after decryption and requires subsequent audio data authentication access, the authentication process is triggered again.

[0157] If the SIM authentication fails, the first device will not continue to play the audio. If the user manually scrolls to the later part of the offline audio, the offline audio cannot be played normally.

[0158] It should be noted that the authentication system can be a system in the same network as the second device. The authentication system and the second device can interact through the intranet, which can improve the notification speed and success rate.

[0159] As an example, there is a scenario in the audio data where a password or verification code needs to be manually entered, and the user needs to enter the password or verification code during the silent frame. If the password or verification code is entered and verified incorrectly, the audio data that should be played later cannot be played. At the same time, this part of the audio data needs to be encrypted in advance to prevent the user from directly locating the encrypted audio data and playing it directly. This encrypted audio data is generally the sound content between two silent frames. If the password or verification code is entered and verified correctly, the first device can interact with the second device, obtain the key to decrypt the encrypted audio data, and then play it.

[0160] In some embodiments, if the first device does not detect that the interaction information includes verification request information, no authentication is required. In this case, the audio is played directly and the corresponding interaction information is displayed according to the interaction requirement information.

[0161] It is understandable that if the author of the audio uses a third device to set the interactive information that requires authentication, then even if the audio data is downloaded to the first device locally, the audio data within the authentication range is still in an encrypted state. It cannot be played normally when played by other ordinary audio players. If the audio data does not require authentication, other ordinary audio players can play it normally.

[0162] As an example, scenarios where authentication is not required may include but are not limited to:

[0163] a. Audio and video pop-up questions: When the user plays audio, after the voice ends and the silence begins, a specified question will pop up for the user to answer. After the answer is completed, the data packet can be uploaded to the second device, and the second device returns the correct answer and displays it in the first device application. Settings such as question display format and whether the audio needs to be paused can be configured when setting the interactive information.

[0164] b. Display of online teaching content: When the teaching audio and video are muted and the user needs to read the pictures or text carefully, the reading content will pop up for the user to read. When voice appears in the teaching audio, it will automatically disappear or disappear after a delay.

[0165] In some embodiments, process data of user interaction through interactive information can be sent to the second device through the background interface. The second device confirms the result according to the process data and performs corresponding processing to achieve the effect of audio interaction.

[0166] In this way, part or all of the audio data can be encrypted to protect the privacy of the audio fragments, further enriching the user's interactive scenarios in audio. During the interactive process, number authentication and SIM authentication methods are supported. After the user is successfully authenticated, the latter part of the audio content can be decrypted and played and interactive actions can be performed. If the authentication fails, the correct audio content and the subsequent inserted interactive information cannot be read, thus realizing the privacy protection of the audio.

[0167] Based on this, in some embodiments, before the above S130, the method may further include:

[0168] Performing windowing processing on the sub-audio signal corresponding to each audio frame to obtain a target sub-audio signal corresponding to each audio frame;

[0169] The above S130 may specifically include:

[0170] Perform fast Fourier transform on the target sub-audio signal corresponding to each audio frame to obtain the spectrum feature information of each audio frame.

[0171] In some embodiments, after framing, it is necessary to perform a fast Fourier transform for feature extraction. However, after framing, the signal has a high degree of discontinuity at the start and end of the frame. If a fast Fourier transform is performed directly, high-frequency components may be generated at the discontinuous points, resulting in local peaks in the spectrum after the Fourier transform, and the Gibbs phenomenon occurs. At the same time, the spectrum has a tailing phenomenon in the entire frequency band, causing spectrum leakage and distortion. Therefore, the sub-audio signal corresponding to each audio frame is windowed, and the windowing function can be selected from the Hanning window or the Hamming window.

[0172] In this way, the sub-audio signal corresponding to each audio frame after the frame division is subjected to windowing processing, so as to reduce the influence caused by the frame division and avoid the occurrence of distortion.

[0173] In some embodiments, voice activity detection (VAD) is used to pre-process high-energy sharp noise by sliding window before judging the short-term energy value, and convolution smoothing is used before judging the spectrum amplitude. The first silent frame threshold is judged by dynamic threshold, and the cosine similarity is used for judging based on the spectrum amplitude.

[0174] Based on this, in some embodiments, such as Figure 8 As shown, the above S140 may specifically include:

[0175] S141. The third device sets sliding windows before and after a target audio frame among the multiple audio frames, respectively, where the target audio frame is any one of the multiple audio frames.

[0176] S142: When the average short-time energy value of the audio frames in the sliding window is less than the first preset value and the short-time energy value of the target audio frame is not less than the first preset value, the third device updates the short-time energy value of the target audio frame to the target short-time energy value to obtain multiple first smoothed audio frames;

[0177] S143, the third device divides each audio frame into multiple frequency levels based on the Mel scale;

[0178] S144: The third device smoothes the frequency spectrum amplitude corresponding to each audio frame of each frequency level to obtain multiple second smoothed audio frames corresponding to multiple frequency levels;

[0179] S145. The third device identifies a first sub-silent frame in the plurality of first smooth audio frames based on the short-time energy values ​​of the plurality of first smooth audio frames, and identifies a second sub-silent frame in the plurality of second smooth audio frames based on the spectrum amplitudes of the plurality of second smooth audio frames;

[0180] S146. The third device determines a silent frame among the multiple audio frames according to the first sub-silent frame and the second sub-silent frame.

[0181] In some embodiments, after obtaining the short-time energy value of each audio frame, the short-time energy value is quantized and classified into several levels. For example, the difference between the highest value and the lowest value is divided into N energy levels, forming a histogram with time as the x-axis and the short-time energy value as the y-axis. Considering that high-energy sharp noise may be introduced to affect the detection of silent frames, it is necessary to introduce a left and right sliding window comparison method to reduce the impact of high-energy sharp noise. The specific process is as follows:

[0182] like Fig. 9 As shown, the above S142 may specifically include: starting from the left side, for each audio frame, select two left and right sliding windows of the same size (the left window of the first audio frame is regarded as a silent window, and the right window of the tail audio frame is regarded as a silent window), and then compare whether the two sliding windows are low-energy windows (that is, the average short-time energy value of all audio frames in the window is less than the first preset value, which is a low-energy window. The low-energy window can be considered a silent window, and the window may include multiple audio frames). If both sliding windows are low-energy windows, but the short-time energy value of the current audio frame is not less than the first preset value, the current audio frame can be considered to be high-energy sharp noise, marked as a silent frame, and the short-time energy value of this audio frame is reduced to the target short-time energy value, the high-energy sharp noise is eliminated, and smoothing is performed. For example, the target short-time energy value is the average of the short-time energy values ​​of the left and right windows.

[0183] After completing the above operations, check the right window. If the right window is a low-energy window (silent window), slide to the right to the first audio frame on the right side of the right window and do the next round of comparison again. If the right window is a high-energy window, start the next round of comparison again from the first frame in the right window. Finally, the short-term energy values ​​of multiple audio frames are smoothed.

[0184] It should be noted that one audio frame corresponds to one energy level, that is, one short-time energy value is one energy level. High energy and low energy can be determined by the short-time energy value itself or by the energy level.

[0185] In some embodiments, a certain audio frame may correspond to a Mel-scale spectrum histogram. The horizontal axis of the spectrum histogram is divided into N frequency levels (corresponding to the N frequency bands mentioned above). Then, according to the horizontal axis frequency in the spectrum histogram, with time as the horizontal axis and spectrum amplitude as the vertical axis, a feature sequence of N frequency ranges can be sorted out. The spectrum amplitude corresponding to each audio frame of each frequency level is smoothed, and the specific process is as follows:

[0186] like Fig.10As shown, the above S144 may specifically include: performing weighted average calculation on the spectrum amplitude of each sub-frame according to the custom convolution kernel for the N feature sequences, and generating new N feature sequences. It can be understood that the purpose of convolution weighted average calculation (convolution: in image and audio processing, the target can be numerically averaged by convolution operation to achieve the effect of smoothing) is to perform local smoothing and reduce the impact of burr signals. Here, the custom convolution kernel is set in advance, that is, when performing weighted average calculation on each sub-frame, the number of frames of the sub-frame is set in advance, and the weight is also set in advance.

[0187] As an example, in Fig.10 In the , the same time corresponds to N spectrum amplitudes (the same frame is divided into 0-16 frequencies, 16-32 frequencies, 32-64 frequencies, 64-128 frequencies...), so an N-dimensional vector is used to calculate the cosine similarity when identifying silent frames.

[0188] Specifically, for the short-time energy value, the short-time energy value of each first smooth audio frame is compared with the first silent frame threshold, the absolute value of the difference between the short-time energy value of each first smooth audio frame and the first silent frame threshold is calculated (Euclidean distance can be used), and the first smooth audio frame whose absolute value is less than the first silent frame threshold is determined to be the first sub-silent frame.

[0189] For the spectrum amplitude, since each second smooth audio frame is divided into N (such as 11) frequency bands (N frequency bands belong to different frequency ranges), the spectrum amplitude of the N frequency bands corresponding to each frame time can correspond to an N-dimensional vector. Therefore, the silent frame is identified by calculating the cosine similarity of two adjacent frames. If the cosine similarity is less than the second silent frame threshold, it is determined that the previous frame is the second sub-silent frame, or the next frame is the second sub-silent frame, or it is determined that both frames are second sub-silent frames.

[0190] In some embodiments, the first sub-silent frame and the second sub-silent frame determined based on the short-time energy value and the spectrum amplitude are combined to finally obtain the silent frame in the multiple audio frames. Here, the combination can be a union or an intersection.

[0191] In this way, we optimize the conventional VAD algorithm, use a sliding window to pre-process high-energy sharp noise before judging the short-term energy value, use convolution smoothing before judging the spectrum amplitude, and perform multiple rounds of parameter tuning. The first silent frame threshold uses a dynamic threshold, and the spectrum amplitude is judged using cosine similarity, which improves the robustness of the VAD algorithm. In abnormal scenarios (adding burr noise, high-energy sharp noise, and non-stationary noise), more accurate results can be obtained without a large amount of training data. The optimized algorithm has high operating efficiency and is easy to implement, and can achieve better detection results.

[0192] Based on this, in some embodiments, the audio data includes encrypted audio data, the interaction information includes second target interaction information, and the second target interaction information includes verification request information; before the above S160, the method may further include:

[0193] The target audio data in the audio data is encrypted using a key to obtain encrypted audio data, and second target interaction information is generated, where the encrypted audio data is audio data corresponding to an audio frame after the position of the second target interaction information.

[0194] In some embodiments, for scenarios that require authentication, when inserting interactive information, you can choose whether subsequent audio data needs authentication and determine the authentication interval. Generally, the interval that requires authentication is the non-silent frame interval. To determine the interval that requires authentication, you need to select a suitable encryption algorithm for processing based on the encryption scenario, such as AES encryption, elliptical encryption, etc.

[0195] In some embodiments, authentication of other interactive information itself may also be required. For example, if there are two silent frame intervals, and there are two interactive information A and B in succession, A can set the subsequent content to require authentication, or it can set the interactive information B to be displayed only after successful authentication.

[0196] In this way, part or all of the audio data can be encrypted to protect the privacy of the audio fragments and further enrich the user's audio interaction scenarios.

[0197] In the embodiments provided in the present application, silence detection is introduced. When audio is collected in real time or when an audio file is detected, user interaction information is automatically inserted into the silence frame, and the user can interact in the silence frame, such as answering questions. When recording audio, the user plans the silence timing and duration, and selects the interaction information. When recording, the silence can be automatically allocated and inserted into the silence start time according to the number of silence frames, thereby reducing the user operation steps, the difficulty of operation and the probability of error. When inserting interaction information, specific authentication methods can be introduced, such as automatic number determination (number authentication) and SIM authentication. Reading rights of certain audio intervals are opened to specific users, and others need to be decrypted before they can be played; for non-target customers, only encrypted audio can be read, but it cannot be played correctly, which improves security.

[0198] Based on the interactive information positioning method provided in the above embodiment, the present application also provides a specific implementation of the interactive information positioning device. Please refer to the following embodiment.

[0199] See also Fig.11 The interactive information positioning device 400 provided in the embodiment of the present application is applied to a first device and includes:

[0200] The sending module 410 is configured to send an audio acquisition request to the second device when receiving an input including the audio tag information, where the audio acquisition request includes the audio tag information;

[0201] A receiving module 420 is used to receive audio data corresponding to the audio tag information sent by the second device, and target association information between silent frames and interactive information in the audio data, where the target association information is that after the third device converts the acquired audio data into an audio signal, it performs fast Fourier transform on sub-audio signals of multiple audio frames obtained by framing the audio signal according to a preset time length to obtain a short-time energy value and a spectrum amplitude of each audio frame, identifies silent frames in the multiple audio frames based on the short-time energy value and the spectrum amplitude, determines the position of the interactive information in the audio frame corresponding to the audio data according to the order of the interactive information and the order of the silent frames, and generates the association information accordingly;

[0202] An acquisition module 430 is used to acquire a position of a silent frame in an audio frame corresponding to the audio data;

[0203] The determination module 440 is configured to determine the position of the interaction information according to the target association information and the position of the silence frame.

[0204] Based on this, in some embodiments, the apparatus 400 may further include:

[0205] The receiving module 420 is further configured to receive an interaction instruction for first target interaction information sent by the second device after determining the position of the interaction information according to the target association information and the position of the silent frame, where the first target interaction information is any one of the interaction information, and the interaction instruction is an instruction sent from the second device when the interaction requirement information of the interaction information indicates that the first target interaction information needs to interact with the user;

[0206] The display module is configured to display the first target interaction information at the location of the first target interaction information.

[0207] Based on this, in some embodiments, the apparatus 400 may further include:

[0208] A playing module, configured to play the audio data and display the interactive information at the position of the interactive information after determining the position of the interactive information according to the target association information and the position of the silent frame;

[0209] The display module is further configured to display the verification request information at the location of the second target interaction information when it is detected that the second target interaction information in the interaction information includes the verification request information;

[0210] The sending module 410 is further configured to, upon receiving the verification response information input by the user, send the verification response information to the second device, so that the second device verifies the verification response information, and, upon satisfying a preset condition, send the key for encrypting the audio data to the first device, wherein the encrypted audio data is audio data corresponding to the audio frame after the position of the second target interaction information, obtained by encrypting the target audio data in the audio data by the third device using the key;

[0211] The receiving module 420 is further configured to receive a key sent by the second device, and decrypt the encrypted audio data using the key to obtain decrypted audio data;

[0212] The playing module is also used to play the decrypted audio data.

[0213] See also Fig.12 The interactive information positioning device 500 provided in the embodiment of the present application is applied to a second device, including:

[0214] The receiving module 510 is configured to receive an audio acquisition request sent by the first device, where the audio acquisition request includes audio tag information;

[0215] an acquisition module 520, configured to acquire audio data corresponding to the audio tag information, and target association information between silent frames and interactive information in the audio data, wherein the target association information is that after the third device converts the acquired audio data into an audio signal, a fast Fourier transform is performed on a sub-audio signal of a plurality of audio frames obtained by framing the audio signal according to a preset time length, so as to obtain a short-time energy value and a spectrum amplitude of each audio frame, identify silent frames in the plurality of audio frames based on the short-time energy value and the spectrum amplitude, determine a position of the interactive information in the audio frame corresponding to the audio data according to an order of the interactive information and an order of the silent frames, and generate the association information accordingly;

[0216] The sending module 530 is used to send audio data and target association information to the first device, so that the first device obtains the position of the silent frame in the audio frame corresponding to the audio data, and determines the position of the interactive information according to the target association information and the position of the silent frame.

[0217] Based on this, in some embodiments, the apparatus 500 may further include:

[0218] The acquisition module 520 is further configured to acquire interaction requirement information of the interaction information after sending the audio data and the target association information to the first device;

[0219] The sending module 530 is also used to send an interaction instruction for the first target interaction information to the first device when the interaction demand information indicates that the first target interaction information needs to interact with the user. The interaction instruction is used to instruct the first device to display the first target interaction information at the position of the first target interaction information. The first target interaction information is any one of the interaction information.

[0220] Based on this, in some embodiments, the apparatus 500 may further include:

[0221] The receiving module 510 is further configured to receive verification response information sent by the first device after sending the audio data and the target association information to the first device, where the verification response information is verification response information of the verification request information input by the user received by the first device when the first device detects that the second target interaction information in the interaction information includes the verification request information;

[0222] The sending module 530 is also used to verify the verification response information. When the verification response information meets the preset conditions, the key for the encrypted audio data is sent to the first device, so that the first device uses the key to decrypt the encrypted audio data, obtains the decrypted audio data, and plays the decrypted audio data. The encrypted audio data is the audio data corresponding to the audio frame after the position of the second target interaction information, which is obtained by the third device using the key to encrypt the target audio data in the audio data.

[0223] See also Fig.13 The interactive information positioning device 600 provided in the embodiment of the present application is applied to a third device, including:

[0224] An acquisition module 610, configured to acquire audio data, audio tag information and interaction information of the audio data, and convert the audio data into an audio signal;

[0225] A framing module 620, configured to frame the audio signal according to a preset duration to obtain sub-audio signals of multiple audio frames;

[0226] The transformation module 630 is used to perform a fast Fourier transform on the sub-audio signal of each audio frame to obtain the spectrum feature information of each audio frame, where the spectrum feature information includes a short-time energy value and a spectrum amplitude;

[0227] An identification module 640, configured to identify a silent frame in a plurality of audio frames based on a short-time energy value and a spectrum amplitude;

[0228] A determination module 650 is used to determine the position of the interaction information in the audio frame corresponding to the audio data according to the order of the interaction information and the order of the silence frame, and to generate target association information of the silence frame and the interaction information accordingly;

[0229] A sending module 660 is used to send audio data, audio tag information of the audio data, and target association information to a second device, so that when the second device receives an audio acquisition request sent by the first device, the second device sends the audio data and the target association information to the first device, so that the first device obtains the position of a silent frame in an audio frame corresponding to the audio data, and determines the position of the interactive information according to the target association information and the position of the silent frame, and the audio acquisition request includes the audio tag information.

[0230] Based on this, in some embodiments, the device 660 may further include:

[0231] A processing module, configured to perform windowing processing on the sub-audio signal corresponding to each audio frame to obtain a target sub-audio signal corresponding to each audio frame before performing fast Fourier transform on the sub-audio signal corresponding to each audio frame to obtain frequency spectrum feature information of each audio frame;

[0232] The transformation module 630 may be specifically used for:

[0233] Perform fast Fourier transform on the target sub-audio signal corresponding to each audio frame to obtain the spectrum feature information of each audio frame.

[0234] Based on this, in some embodiments, the identification module 640 may be specifically used to:

[0235] For a target audio frame among the multiple audio frames, sliding windows are respectively set before and after the target audio frame, and the target audio frame is any one of the multiple audio frames;

[0236] When the average short-time energy value of the audio frames in the sliding window is less than the first preset value and the short-time energy value of the target audio frame is not less than the first preset value, updating the short-time energy value of the target audio frame to the target short-time energy value to obtain a plurality of first smoothed audio frames;

[0237] Divide each audio frame into multiple frequency levels based on the Mel scale;

[0238] Smoothing the frequency spectrum amplitude corresponding to each audio frame of each frequency level to obtain a plurality of second smoothed audio frames corresponding to a plurality of frequency levels;

[0239] Based on the short-time energy values ​​of the plurality of first smooth audio frames, identifying a first sub-silent frame in the plurality of first smooth audio frames, and based on the spectrum amplitudes of the plurality of second smooth audio frames, identifying a second sub-silent frame in the plurality of second smooth audio frames;

[0240] A silence frame among the plurality of audio frames is determined according to the first sub silence frame and the second sub silence frame.

[0241] Based on this, in some embodiments, the audio data includes encrypted audio data, the interaction information includes second target interaction information, and the second target interaction information includes verification request information; the apparatus 600 may further include:

[0242] The encryption module is used to encrypt the target audio data in the audio data using a key before sending the audio data, the audio tag information of the audio data and the target association information to the second device to obtain the encrypted audio data and generate the second target interaction information, wherein the encrypted audio data is the audio data corresponding to the audio frame after the position of the second target interaction information.

[0243] The various modules of the interactive information positioning device provided in the embodiment of the present application can realize the functions of the various steps of the interactive information positioning method provided above and achieve the corresponding technical effects, which will not be repeated here for the sake of brevity.

[0244] Based on the same inventive concept, an embodiment of the present application also provides an electronic device.

[0245] Fig.14 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0246] The electronic device may include a processor 701 and a memory 702 storing computer program instructions.

[0247] Specifically, the processor 701 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0248] The memory 702 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 702 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 702 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 702 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 702 is a non-volatile solid-state memory.

[0249] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of the present disclosure.

[0250] The processor 701 implements any one of the interaction information positioning methods in the above embodiments by reading and executing the computer program instructions stored in the memory 702 .

[0251] In one example, the electronic device may further include a communication interface 703 and a bus 710. Fig.14 As shown, the processor 701, the memory 702, and the communication interface 703 are connected via a bus 710 and communicate with each other.

[0252] The communication interface 703 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0253] The bus 710 includes hardware, software or both, coupling the components of the electronic device to each other. For example, but not limitation, the bus may include an accelerated graphics port (Accelerated Graphics Port, AGP) or other graphics bus, an enhanced industry standard architecture (Extended Industry Standard Architecture, EISA) bus, a front side bus (Front Side Bus, FSB), a hypertransport (Hyper Transport, HT) interconnection, an industry standard architecture (Industry Standard Architecture, ISA) bus, an infinite bandwidth interconnection, a low pin count (Linear Predictive Coding, LPC) bus, a memory bus, a micro channel architecture (MicroChannel Architecture, MCA) bus, a peripheral component interconnect (Peripheral Component Interconnect, PCI) bus, a PCI-Express (Peripheral Component Interconnect-X, PCI-X) bus, a serial advanced technology attachment (Serial Advanced Technology Attachment, SATA) bus, a video electronics standard association local (VESA Local Bus, VLB) bus or other suitable bus or a combination of two or more of these. Where appropriate, the bus 710 may include one or more buses. Although the embodiments of the present application describe and illustrate a specific bus, the present application considers any suitable bus or interconnection. The electronic device can execute the method for locating the interactive information in the embodiments of the present invention, thereby realizing the above method for locating the interactive information.

[0254] In addition, in combination with the interactive information positioning method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any interactive information positioning method in the above embodiment is implemented.

[0255] The present application also provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes each process of implementing any one of the above-mentioned positioning method embodiments for interactive information.

[0256] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0257] The functional blocks shown in the structural block diagram described above can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), appropriate firmware, plug-in, function card, etc. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or communication link by a data signal carried in a carrier. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (Read-Only Memory, ROM), flash memory, erasable read-only memory (Erasable ReadOnly Memory, EROM), floppy disks, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical discs, hard disks, optical fiber media, radio frequency (Radio Frequency, RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0258] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0259] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0260] The above are only specific implementation methods of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A method for locating interactive information, characterized in that: Applied to a first device, comprising: In case of receiving an input including audio tag information, sending an audio acquisition request to a second device, wherein the audio acquisition request includes the audio tag information; Receive audio data corresponding to the audio tag information sent by the second device, and target association information of silent frames and interactive information in the audio data, wherein the target association information is that after the third device converts the acquired audio data into an audio signal, it performs fast Fourier transform on sub-audio signals of multiple audio frames obtained by framing the audio signal according to a preset time length to obtain a short-time energy value and a spectrum amplitude of each of the audio frames, identifies silent frames in the multiple audio frames based on the short-time energy value and the spectrum amplitude, determines the position of the interactive information in the audio frame corresponding to the audio data according to the order of the interactive information and the order of the silent frames, and generates the association information accordingly; Obtaining a position of the silent frame in the audio frame corresponding to the audio data; The position of the interaction information is determined according to the target association information and the position of the silence frame.

2. The method for locating interactive information according to claim 1, characterized in that: After determining the position of the interaction information according to the target association information and the position of the silence frame, the method further includes: receiving an interaction instruction for first target interaction information sent by the second device, where the first target interaction information is any one of the interaction information, and the interaction instruction is an instruction sent from the second device when interaction requirement information of the interaction information indicates that the first target interaction information needs to interact with a user; The first target interaction information is displayed at the position of the first target interaction information.

3. The method for locating interactive information according to claim 1 or 2, characterized in that: After determining the position of the interaction information according to the target association information and the position of the silence frame, the method further includes: Playing the audio data and displaying the interaction information at the location of the interaction information; In a case where it is detected that the second target interaction information in the interaction information includes verification request information, displaying the verification request information at the location of the second target interaction information; When receiving verification response information input by the user, sending the verification response information to the second device, so that the second device verifies the verification response information, and when the verification response information satisfies a preset condition, sending a key for encrypting audio data to the first device, where the encrypted audio data is audio data corresponding to an audio frame after the position of the second target interaction information, obtained by the third device encrypting target audio data in the audio data using the key; receiving the key sent by the second device, and decrypting the encrypted audio data using the key to obtain decrypted audio data; The decrypted audio data is played.

4. A method for locating interactive information, characterized in that: Applied to the second device, comprising: receiving an audio acquisition request sent by a first device, wherein the audio acquisition request includes audio tag information; Acquire audio data corresponding to the audio tag information, and target association information between silent frames and interactive information in the audio data, wherein the target association information is that after the third device converts the acquired audio data into an audio signal, a sub-audio signal of a plurality of audio frames obtained by framing the audio signal according to a preset time length is subjected to fast Fourier transform to obtain a short-time energy value and a spectrum amplitude of each of the audio frames, and based on the short-time energy value and the spectrum amplitude, a silent frame in the plurality of audio frames is identified, and a position of the interactive information in the audio frame corresponding to the audio data is determined according to an order of the interactive information and an order of the silent frames, and the corresponding association information is generated; The audio data and the target association information are sent to the first device so that the first device obtains the position of the silent frame in the audio frame corresponding to the audio data, and determines the position of the interactive information according to the target association information and the position of the silent frame.

5. The method for locating interactive information according to claim 4, characterized in that: After sending the audio data and the target association information to the first device, the method further includes: Acquire interaction requirement information of the interaction information; In a case where the interaction demand information represents that the first target interaction information needs to interact with the user, an interaction instruction for the first target interaction information is sent to the first device, and the interaction instruction is used to instruct the first device to display the first target interaction information at the position of the first target interaction information, and the first target interaction information is any one of the interaction information.

6. The method for locating interactive information according to claim 4, characterized in that: After sending the audio data and the target association information to the first device, the method further includes: receiving verification response information sent by the first device, where the verification response information is verification response information of the verification request information input by the user and received by the first device when the first device detects that the second target interaction information in the interaction information includes the verification request information; The verification response information is verified, and if the verification response information meets a preset condition, a key for the encrypted audio data is sent to the first device, so that the first device uses the key to decrypt the encrypted audio data, obtains the decrypted audio data, and plays the decrypted audio data, wherein the encrypted audio data is the audio data corresponding to the audio frame after the position of the second target interaction information, and is obtained by the third device using the key to encrypt the target audio data in the audio data.

7. A method for locating interactive information, characterized in that: Applicable to third-party devices, including: Acquire audio data, audio tag information and interaction information of the audio data, and convert the audio data into an audio signal; Divide the audio signal into frames according to a preset time length to obtain sub-audio signals of multiple audio frames; Performing a fast Fourier transform on the sub-audio signal of each of the audio frames to obtain frequency spectrum feature information of each of the audio frames, wherein the frequency spectrum feature information includes a short-time energy value and a frequency spectrum amplitude; Based on the short-time energy value and the spectrum amplitude, identifying a silent frame among the plurality of audio frames; According to the order of the interaction information and the order of the silence frames, determine the position of the interaction information in the audio frame corresponding to the audio data, and correspondingly generate target association information between the silence frame and the interaction information; The audio data, the audio tag information of the audio data, and the target association information are sent to the second device, so that when the second device receives the audio acquisition request sent by the first device, the audio data and the target association information are sent to the first device, so that the first device can obtain the position of the silent frame in the audio frame corresponding to the audio data, and determine the position of the interactive information according to the target association information and the position of the silent frame, wherein the audio acquisition request includes the audio tag information.

8. The method for locating interactive information according to claim 7, characterized in that: Before performing fast Fourier transform on the sub-audio signal corresponding to each of the audio frames to obtain the frequency spectrum feature information of each of the audio frames, the method further includes: Performing windowing processing on the sub-audio signal corresponding to each of the audio frames to obtain a target sub-audio signal corresponding to each of the audio frames; The performing fast Fourier transform on the sub-audio signal corresponding to each of the audio frames to obtain the frequency spectrum feature information of each of the audio frames includes: Performing a fast Fourier transform on the target sub-audio signal corresponding to each of the audio frames to obtain frequency spectrum feature information of each of the audio frames.

9. The method for locating interactive information according to claim 7, characterized in that: The step of identifying a silent frame among the plurality of audio frames based on the short-time energy value and the spectrum amplitude comprises: For a target audio frame among the multiple audio frames, respectively set sliding windows before and after the target audio frame, the target audio frame being any one of the multiple audio frames; When the average short-time energy value of the audio frames in the sliding window is less than a first preset value and the short-time energy value of the target audio frame is not less than the first preset value, updating the short-time energy value of the target audio frame to a target short-time energy value to obtain a plurality of first smoothed audio frames; Dividing each of the audio frames into a plurality of frequency levels based on the Mel scale; Smoothing the frequency spectrum amplitude corresponding to each of the audio frames at each of the frequency levels to obtain a plurality of second smoothed audio frames corresponding to the plurality of frequency levels; Identify a first sub-silent frame in the plurality of first smooth audio frames based on short-time energy values ​​of the plurality of first smooth audio frames, and identify a second sub-silent frame in the plurality of second smooth audio frames based on spectrum amplitudes of the plurality of second smooth audio frames; A silence frame among the multiple audio frames is determined according to the first sub-silence frame and the second sub-silence frame.

10. The method for locating interactive information according to claim 7, characterized in that: The audio data includes encrypted audio data, the interaction information includes second target interaction information, and the second target interaction information includes verification request information; Before sending the audio data, the audio tag information of the audio data, and the target association information to the second device, the method further includes: The target audio data in the audio data is encrypted using a key to obtain encrypted audio data, and the second target interaction information is generated, wherein the encrypted audio data is audio data corresponding to an audio frame after a position of the second target interaction information.

Images