Method, device and equipment for determining sound event start and end time, and storage medium

Abstract

The application discloses a method, device and equipment for determining the start and end time of a sound event and a storage medium. The method comprises the following steps: determining the acoustic timing characteristics of to-be-detected audio data based on a convolutional neural network and a recurrent convolutional neural network; determining a sound event recognition result of the to-be-detected audio data according to the acoustic timing characteristics and a pre-determined sound event relationship characteristic; wherein the sound event relationship characteristic is determined based on a pre-constructed sound event relationship graph; the sound event relationship graph is determined according to the statistical result of sound events in an audio data set; determining a target frame in the to-be-detected audio data according to the sound event recognition result; and determining the start and end time of a sound event in the to-be-detected audio data according to the number of audio frames between adjacent target frames. The technical scheme solves the problem of low accuracy of sound event start and end time positioning, and can greatly improve the accuracy of sound event start and end time positioning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of voice detection, and in particular to a method and device for determining the start and end time of a sound event, an apparatus, and a storage medium. BACKGROUND

[0002] With the development of intelligent and networked vehicles, there are higher requirements for perception capabilities. In addition to visual technology, sound signals can also provide a lot of useful information and have the advantage of not requiring light and not being affected by rain, clouds, and darkness. Audio detection devices with sound event start and end time determination technology can be widely used in the cabin of intelligent vehicles due to their low cost, small size, easy installation, high reliability, resistance to damage, and simple maintenance.

[0003] Currently, the positioning of sound event start and end time is mainly based on a deep learning model to determine the type of sound event, and then determine the start and end time of the sound event according to the audio frames involved in the sound event. Convolutional neural networks are good at capturing local features of sound signals and can classify sound signals well. Combining CNN and RNN, a convolutional recurrent neural network can achieve more ideal detection performance and has become the mainstream model for determining the start and end time of sound events.

[0004] However, the existing scheme does not consider the relationship between sound events in a sound scene, so there is still a gap in the positioning accuracy of sound event start and end time in the context of polyphonic sound events, especially when multiple sound events overlap in time. SUMMARY

[0005] The present application provides a method and device for determining the start and end time of a sound event, an apparatus, and a storage medium to solve the problem of low accuracy in positioning the start and end time of a sound event, which can greatly improve the accuracy of start and end time positioning.

[0006] According to an aspect of the present application, a method for determining the start and end time of a sound event is provided, the method comprising:

[0007] determining the acoustic timing features of the audio data to be detected based on a convolutional neural network and a recurrent convolutional neural network;

[0008] determining the sound event recognition result of the audio data to be detected according to the acoustic timing features and the pre-determined sound event relationship features; wherein the sound event relationship features are determined based on a pre-constructed sound event relationship graph; and the sound event relationship graph is determined according to the statistical results of sound events in an audio data set;

[0009] determining the target frame in the audio data to be detected according to the sound event recognition result;

[0010] The start and end times of the sound event in the audio data to be detected are determined according to the number of audio frames between adjacent target frames.

[0011] According to another aspect of the present application, there is provided a device for determining start and end times of a sound event, comprising:

[0012] an acoustic timing feature determination module configured to determine acoustic timing features of the audio data to be detected based on a convolutional neural network and a recurrent convolutional neural network;

[0013] a recognition result determination module configured to determine a sound event recognition result of the audio data to be detected according to the acoustic timing features and pre-determined sound event relationship features, wherein the sound event relationship features are determined based on a pre-constructed sound event relationship graph, and the sound event relationship graph is determined according to statistical results of sound events in an audio data set;

[0014] a target frame determination module configured to determine target frames in the audio data to be detected according to the sound event recognition result;

[0015] a start and end time determination module configured to determine the start and end times of the sound event in the audio data to be detected according to the number of audio frames between adjacent target frames.

[0016] According to another aspect of the present application, there is provided an electronic device, comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein

[0019] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the method for determining start and end times of a sound event according to any one of the embodiments of the present application.

[0020] According to another aspect of the present application, there is provided a computer readable storage medium storing computer instructions for enabling a processor to perform the method for determining start and end times of a sound event according to any one of the embodiments of the present application when executed by the processor.

[0021] The technical solution of the embodiment of the present application determines the sound event recognition result of the to-be-detected audio data through the acoustic timing feature and the prior sound event relationship feature of the to-be-detected audio data, determines the target frame in the to-be-detected audio data according to the sound event recognition result, and further determines the start and end time of the sound event in the to-be-detected audio data according to the number of audio frames between adjacent target frames. The technical solution solves the problem of low accuracy of sound event start and end time positioning, and can greatly improve the accuracy of sound event start and end time positioning.

[0022] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0024] Figure 1A is a flowchart of a sound event start and end time determination method according to an embodiment of the present application;

[0025] Figure 1B is a sound event relationship diagram according to an embodiment of the present application;

[0026] Figure 2A is a flowchart of a sound event start and end time determination method according to an embodiment of the present application;

[0027] Figure 2B is a sound event start and end time positioning diagram according to an embodiment of the present application;

[0028] Figure 2C is a sound event start and end time positioning result diagram according to an embodiment of the present application;

[0029] Figure 3 is a structural diagram of a sound event start and end time determination device according to an embodiment of the present application;

[0030] Figure 4 is a structural diagram of an electronic device for implementing the sound event start and end time determination method according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative work should belong to the protection scope of the present application.

[0032] It should be noted that the terms "first", "second" and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in other than the order illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to include only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device. The acquisition, storage, use, processing and the like of data in the technical solutions of the present application comply with the relevant provisions of national laws and regulations.

[0033] Embodiment one

[0034] Figure 1A A flowchart of a method for determining the start and end time of a sound event is provided for the first embodiment of the present application. The present embodiment can be applied to the scenario of determining the start and end time of a sound event in a car cabin. The method can be executed by a device for determining the start and end time of a sound event. The device can be realized in the form of hardware and / or software, and can be configured in an electronic device. As shown in the figure, the method comprises: Figure 1A

[0035] S110, determining the acoustic timing features of the to-be-detected audio data based on the convolutional neural network and the recurrent convolutional neural network.

[0036] The present scheme can be executed by a vehicle-mounted system. The vehicle-mounted system can obtain the to-be-detected audio data in the vehicle through a voice pickup device such as a microphone. The vehicle-mounted system can perform frame division, windowing and discrete Fourier transform on the to-be-detected audio data, and extract the amplitude, phase and other acoustic features of the to-be-detected audio data. The vehicle-mounted system can process the acoustic features obtained by processing the voice signal based on the convolutional neural network and the recurrent convolutional neural network, to obtain the acoustic timing features.

[0037] ​The convolutional neural network can be sequentially connected with the recurrent neural network, the acoustic feature obtained by processing the speech signal is input into the convolutional neural network, and the deep acoustic feature extracted by the convolutional neural network is taken as the input of the recurrent convolutional neural network to obtain the acoustic time sequence feature. It is easy to understand that the convolutional neural network can further extract the deep acoustic feature on the basis of the acoustic feature output by the speech signal processing, and the recurrent convolutional neural network can extract the time sequence feature in the acoustic feature. The convolutional neural network can be constructed based on the network structure of AlexNet, VGG-Net or ResNet, or the network structure can be customized according to actual application needs. The recurrent convolutional neural network can be built based on the Gated Recurrent Unit network (GRU), the Bidirectional RNN (Bi-RNN) or the Long Short-Term Memory network (LSTM).

[0038] In S120, the sound event recognition result of the to-be-detected audio data is determined according to the acoustic time sequence feature and the pre-determined sound event relationship feature.

[0039] It is easy to understand that the sound event can be a sound carrying physical event information, such as a siren sound, a brake sound and a speaking sound. The sound event relationship feature can be pre-stored in the vehicle-mounted system. The sound event relationship feature can be determined based on a pre-constructed sound event relationship graph. The sound event relationship graph can be determined according to the statistical result of the sound events in the audio data set. The audio data set can be a public audio data set or an audio data set collected and made by the researchers according to needs. The audio data set can include audio data and label information associated with the audio data, wherein the audio data in the audio data set can be collected in the car cabin, and the label information of the audio data can include the type of the sound event in the audio data, the start and end time of each sound event and the like. Using a computer, a server or other electronic equipment, the sound events existing in the audio data set can be counted to obtain the occurrence frequency of each type of sound event, the simultaneous occurrence frequency of each type of sound event and the sequential occurrence frequency of each type of sound event and the like. According to the statistical result of the sound events in the audio data set, the electronic equipment can construct a sound event relationship graph for representing the association relationship between each type of sound event.

[0040] Figure 1B is a sound event relationship diagram provided according to an embodiment of the present application. After obtaining the sound event relationship, the electronic equipment can draw a diagram as Figure 1BThe voice event relationship graph is shown. The electronic device can extract the voice event relationship features implied by the voice event relationship graph to assist the acoustic features of the to-be-detected audio data, so as to realize voice event recognition in the to-be-detected audio data.

[0041] The vehicle-mounted system can fuse the extracted acoustic features and the pre-determined voice event relationship features. According to the fused features, the vehicle-mounted system can determine a voice event recognition result of the to-be-detected audio data based on a voice event classification model. The voice event recognition result can include the types of voice events existing in the to-be-detected audio data and the occurrence probabilities of the types of voice events.

[0042] In the present scheme, optionally, the determination process of the voice event relationship features includes:

[0043] Obtaining label information of each audio data in the audio data set;

[0044] According to the label information of each audio data, determining the temporal relationship between each type of voice event, determining the occurrence probability matched with the temporal relationship, and determining the commonality of each type of voice event;

[0045] According to the commonality of each type of voice event, the temporal relationship between each type of voice event, and the occurrence probability matched with the temporal relationship, constructing a voice event relationship graph;

[0046] According to the voice event relationship graph, determining the voice event relationship features based on a graph convolutional neural network.

[0047] The label information can include the distribution information of voice events in the audio data, i.e., the start and end time of the audio data, the temporal relationship of the voice events, and the start and end time of the voice events. According to the distribution information of the voice events, the electronic device can statistically obtain the temporal relationship between each type of voice event in the audio data set, the occurrence probability of each temporal relationship, and the commonality of each type of voice event. The commonality can be used to evaluate the commonality of the voice events, which can be determined according to the occurrence frequency and / or duration of the voice events.

[0048] In one specific example, there are 10,000 pieces of audio data in the audio data set, among which 500 pieces of audio data each contain type I voice events and type II voice events. Among the 500 pieces of audio data, 300 pieces of type I voice events occur before type II voice events, and 200 pieces of type II voice events occur before type I voice events. The electronic device can statistically obtain that the occurrence probability of type I voice events occurring before type II voice events is 0.6, and the occurrence probability of type II voice events occurring before type I voice events is 0.4.

[0049] According to the start and end time of each piece of audio data, the duration of each piece of audio data is determined to be 30s, and the total duration of 10000 pieces of audio data is 300000s. According to the start and end time of the I-type sound event in 500 pieces of audio data, the duration of the I-type sound event is determined to be 5000s, and the commonality of the I-type sound event can be 5000 / 300000≈0.017.

[0050] According to the type of sound event existing in 10000 pieces of audio data, the electronic device can determine the total number of occurrences of each type of sound event. There are 7 types of sound events in 10000 pieces of audio data, and 7 types of sound events occur 15000 times, of which II type sound event occurs 500 times, and the commonality of II type sound event can be expressed as 500 / 15000≈0.033.

[0051] According to the commonality of each type of sound event, the electronic device can correct the time sequence relationship between each type of sound event and the occurrence probability matched with the time sequence relationship to construct a reliable sound event relationship graph. After correcting the time sequence relationship and the occurrence probability, the electronic device can construct a sound event relationship graph according to the corrected time sequence relationship and the occurrence probability.

[0052] The electronic device can construct edges between nodes according to the time sequence relationship between sound events, and the occurrence probability matched with the time sequence relationship can be used as the weight of the edges between nodes, so as to obtain a sound event relationship graph. The time sequence relationship between sound events can be directed or undirected, so the sound event relationship graph can be a directed graph or an undirected graph.

[0053] After obtaining the sound event relationship graph, the electronic device can determine the adjacency matrix, the degree matrix, and the feature vector matrix of the node according to the sound event relationship graph. According to the adjacency matrix, the degree matrix, and the feature vector matrix of the node, the relationship feature is extracted based on the graph convolutional neural network, and the electronic device can obtain the sound event relationship feature.

[0054] On the basis of the above-mentioned scheme, the label information includes the type of sound event existing in the audio data and the start and end time of the sound event.

[0055] The label information of each audio data is determined, the time sequence relationship between each type of sound event is determined, the occurrence probability matched with the time sequence relationship is determined, and the commonality of each type of sound event is determined, including:

[0056] According to the type of sound events existing in each audio data, the occurrence frequency of each type of sound event is determined; and according to the type of sound events existing in each audio data and the start and end time of the sound events in each audio data, the time sequence relationship between each type of sound event is determined;

[0057] According to the time sequence relationship between each type of sound event and the occurrence frequency of each type of sound event, the occurrence probability matching the time sequence relationship is determined;

[0058] According to the occurrence frequency of each type of sound event, the occurrence frequency of each type of sound event is determined, and according to the start and end time of the sound event, the time proportion of each type of sound event to all audio data is determined;

[0059] According to the occurrence frequency and time proportion, the commonness of each type of sound event is determined.

[0060] The electronic device such as computer and server can obtain the label information of audio data from the audio data set. According to the label information, the occurrence frequency of each type of sound event, the total number of sound events, the number of simultaneous occurrence of multiple types of sound events in each audio data and the number of occurrence are counted. Specifically, the electronic device can record each type of sound event as L i , i = 0, 1, 2, 3,.., i represents the index of sound event type. The case that two types of sound events exist in an audio data is recorded as L ij , i, j = 0, 1, 2, 3,.. and i ≠ j, i and j both represent the index of sound event type. L ij represents that the sound event L i and the sound event L j exist in the same audio data, that is, both the simultaneous occurrence and the sequential occurrence of two types of sound events are considered. In order to ensure that the sequential nature of the sound events in the same audio data calculated conforms to the real rule, the duration of a single audio data can be set within a certain time range, for example, 30s-60s, to ensure the integrity and relevance of the audio data information. According to the type of sound events existing in each audio data, the occurrence frequency of each type of sound event is counted, which is recorded as X i , which represents the number of times the sound event L i occurs. By adding the occurrence frequency of each type of sound event, the total number of sound events can be obtained, X represents the total number of sound events. The number of times when two sound events occur is recorded as X ij , which represents the number of times when the sound event L i and the sound event L j occur in the same audio data.

[0061] The electronic device can calculate the occurrence probability of each type of sound event according to the occurrence number of each type of sound event, and then calculate the probability of simultaneous occurrence and sequential occurrence of various sound events through a prior probability formula. Specifically, the occurrence frequency of each type of sound event is calculated According to the occurrence number X of the sound event L i , the occurrence probability of the sound event L i is calculated i The probability P of simultaneous occurrence and sequential occurrence of each type of sound event is calculated ij According to the occurrence number X of each type of sound event i and the occurrence number X of both types of sound events ij , the probability P of occurrence of the sound event L i under the premise that the sound event L j occurs is calculated ij = P (L j |L i ) = X ij / X i . It should be noted that P ij and P ji are not necessarily equal, and the meanings expressed are also different, P ij represents the probability of occurrence of the sound event L i under the premise that the sound event L j occurs, and P ji represents the probability of occurrence of the sound event L j under the premise that the sound event L i occurs.

[0062] The electronic device can calculate the proportion of the occurrence duration of each type of sound event in the total duration of all sound events in the audio data set Specifically, the electronic device can calculate the occurrence duration of each type of sound event according to the start and end time of the sound event and calculate the total duration X of all sound events time , calculate the time proportion of the sound event L i

[0063] Since some types of sound events occur multiple times in the audio data and almost exist in the entire audio data, for example, engine running sound, and some types of sound events occur rarely, for example, the sound of a violent collision in a car accident. Therefore, it is not comprehensive to determine whether a sound event is common only by the occurrence frequency of the sound event. In order to perfect the prior information of the relationship of the sound event, the electronic device can calculate the occurrence frequency Y fre of the sound event and the duration proportion Y time ​​The two proportions are adjusted by setting an adjustment parameter σ.

[0064] In one possible implementation, the commonality calculation formula is:

[0065]

[0066] wherein, represents the occurrence frequency, represents the time proportion, σ represents the adjustment parameter, σ ∈ [0, 1], and i represents the sound event type index.

[0067] The above scheme can quantify the commonality of the sound event, and the sound event relationship graph is adjusted according to the commonality of the sound event type, which is beneficial to adjust the prior information of the sound event relationship and ensure the reliability of the prior information.

[0068] In one preferred implementation, the sound event relationship graph is constructed according to the commonality of each type of sound event, the time sequence relationship between each type of sound event, and the occurrence probability matched with the time sequence relationship, and includes the following steps.

[0069] If the commonality of each type of sound event is greater than a preset commonality threshold, the sound event relationship graph is constructed according to the time sequence relationship between each type of sound event and the occurrence probability matched with the time sequence relationship.

[0070] If the commonality of the target type of sound event is less than or equal to the preset commonality threshold, and the target type of sound event is in the pre-set important uncommon event set, the occurrence probability matched with the time sequence relationship of the target type of sound event is corrected according to a preset probability correction principle, and the sound event relationship graph is constructed according to the time sequence relationship between each type of sound event and the corrected occurrence probability matched with the time sequence relationship.

[0071] If the commonality of each type of sound event is greater than the commonality threshold, it indicates that each type of sound event is a common event, and the current time sequence relationship and occurrence probability can be used as the basis for constructing the sound event relationship graph. The electronic device can take each type of sound event as a node, determine the directed edges between the nodes according to the time sequence relationship between each type of sound event, and take the occurrence probability matched with the time sequence relationship as the weight of each directed edge, thereby realizing the construction of the sound event relationship graph.

[0072] If at least one target type of sound event has a commonity less than or equal to a preset commonity threshold, and this target type of sound event is included in a pre-defined set of important but infrequent events, then the target type of sound event must be retained. To reinforce the temporal relationships of important but infrequent events, the electronic device can directly set a large probability of occurrence, such as 1, for the temporal relationship matching associated with the target type of sound event. Alternatively, it can gradually increase the probability of occurrence with a preset correction step size until the probability of occurrence of the temporal relationship matching associated with the target type of sound event is greater than or equal to a preset probability threshold. After correction, the electronic device can construct a sound event relationship graph based on the temporal relationships between different types of sound events and the corrected probability of occurrence matching the temporal relationships.

[0073] If at least one target type of sound event has a commonity less than or equal to a preset commonity threshold, and the target type of sound event is not in a pre-set set of important uncommon events, then the target type of sound event is considered a rare event. Rare events have weak correlations. To ensure the accuracy of sound event relationships, electronic devices can directly delete the temporal relationships and probabilities associated with the target type of sound event, and construct a sound event relationship graph based on the corrected temporal relationships and probabilities.

[0074] The above scheme can address the deficiencies in audio datasets by correcting the sound event relationship graph, which helps to ensure the accuracy of prior information on sound event relationships and thus improves the accuracy of sound event recognition.

[0075] S130. Based on the sound event recognition results, determine the target frame in the audio data to be detected.

[0076] The audio data to be detected may include at least two audio frames. The sound event recognition result may include the sound event type recognition result of each audio frame in the audio data to be detected. Each audio frame may contain one type of sound event or multiple types of sound events. The vehicle system can use the audio frame that includes the target type sound event in the sound event type recognition result as the target frame that matches the target type sound event, that is, the audio frame contains the target type sound event.

[0077] S140. Determine the start and end times of sound events in the audio data to be detected based on the number of audio frames between adjacent target frames.

[0078] It can be understood that the frequency range of normal human hearing is about 20Hz-20kHz, therefore, the sampling frequency of the audio data to be detected is usually set to 44.1KHZ, and the frame length of the audio frame is 512 sampling points. Humans are difficult to perceive sounds with an interval of less than 0.1s, and it is difficult to present complete and smooth sound events directly according to the target frame to determine the start and end time of the sound event. At the same time, there is a certain probability of misjudgment in the sound event recognition result, and directly determining the start and end time of the sound event according to the target frame cannot avoid the positioning error caused by the recognition error. Therefore, the vehicle-mounted system can remove the misjudged target frame according to the number of audio frames between adjacent target frames to ensure the completeness and smoothness of the sound event within the start and end time.

[0079] Specifically, the vehicle-mounted system can determine the target frame associated with the target type sound event in the audio data to be detected, and take each adjacent two target frames and the video frames between the two target frames as an analysis group. If the number of audio frames between the adjacent target frames in the current group is greater than the preset interval threshold, the termination time of the current sound event can be determined according to the earlier target frame in the adjacent target frames, and the start time of the next sound event can be determined according to the later target frame in the adjacent target frames. If the number of audio frames between the adjacent target frames in the current group is less than the preset interval number, it can be determined that the sound event continues to occur in the current group of audio frames, and the number of audio frames between the adjacent target frames in the next group is compared with the preset interval number to determine the termination time of the current sound event.

[0080] The technical solution determines the sound event recognition result of the audio data to be detected through the acoustic timing feature and the prior sound event relationship feature of the audio data to be detected, and then determines the target frame in the audio data to be detected according to the sound event recognition result, and further determines the start and end time of the sound event in the audio data to be detected according to the number of audio frames between adjacent target frames. The technical solution solves the problem of low positioning accuracy of the start and end time of the sound event, and can greatly improve the positioning accuracy of the start and end time.

[0081] Embodiment two

[0082] Figure 2A A flowchart of a method for determining the start and end time of a sound event provided by the second embodiment of the present application is based on the above-mentioned embodiment. As shown in the figure, the method comprises: Figure 2A

[0083] S210, determining the acoustic timing feature of the audio data to be detected based on the convolutional neural network and the recurrent convolutional neural network.

[0084] S220, fusing the acoustic timing feature and the sound event relationship feature to obtain a joint feature.

[0085] ​The vehicle-mounted system can fuse the acoustic time sequence feature output by the convolutional neural network with the sound event relationship feature. The vehicle-mounted system can directly splice and combine the acoustic time sequence feature and the sound event relationship feature to obtain a joint feature. The vehicle-mounted system can also perform matrix calculation on the acoustic time sequence feature and the sound event relationship feature to obtain a joint feature. In a specific example, the sound event relationship feature can be represented by a matrix as N x M, where N represents the number of sound event types, and M represents the feature dimension. In order to sufficiently fuse the sound event relationship feature, the acoustic time sequence feature can be a matrix of M x N. Based on the above scheme, the feature fusion manner can include the following three kinds:

[0086] (1) Adjust the sound event relationship feature and the acoustic time sequence feature to the same dimension by matrix transposition, and then add the corresponding elements in the sound event relationship feature and the acoustic time sequence feature of the same dimension to obtain a joint feature;

[0087] (2) Adjust the sound event relationship feature and the acoustic time sequence feature to the same dimension by matrix transposition, and then multiply the corresponding elements in the sound event relationship feature and the acoustic time sequence feature of the same dimension to obtain a joint feature;

[0088] (3) Directly multiply the sound event relationship feature and the acoustic time sequence feature to obtain a joint feature.

[0089] This scheme can fuse the prior sound event relationship feature and the acoustic time sequence feature to enrich the features of the sound event, thereby improving the detection probability of the sound event.

[0090] S230, determining the recognition probability of each type of sound event in each audio frame based on the joint feature based on the sound event recognition network.

[0091] It is easy to understand that the sound event recognition result can include the recognition probability of each type of sound event; and the audio data to be detected can include at least two audio frames. The vehicle-mounted system can take the joint feature as the input of the sound event recognition network, and determine the sound event recognition result of the audio data to be detected according to the output result of the sound event recognition network. The sound event recognition network can be pre-trained and used to realize the recognition of the sound event. The sound event recognition network can be based on a neural network, or can be based on a convolutional neural network. The output result of the sound event recognition network can include the recognition probability of each type of sound event.

[0092] S240, determining the target frame of each type of sound event according to the recognition probability of each type of sound event in each audio frame.

[0093] If the recognition probability of the target type sound event in the current audio frame is greater than a preset probability threshold, for example, 0.5, the current audio frame is determined as a target frame of the target type sound event.

[0094] S250, determining the start and end time of the sound event in the audio data to be detected according to the number of audio frames between adjacent target frames.

[0095] In one possible implementation, the step of determining the start and end time of the sound event in the audio data to be detected according to the number of audio frames between adjacent target frames comprises:

[0096] If the number of audio frames between the current group of adjacent target frames is greater than or equal to a preset interval threshold, the end time of the current sound event is determined according to the earlier target frame among the adjacent target frames, and the start time of the next sound event is determined according to the later target frame among the adjacent target frames.

[0097] If the number of audio frames between the current group of adjacent target frames is less than the preset interval number, the end time of the current sound event is determined according to the number of audio frames between at least one group of adjacent target frames after the current group.

[0098] The vehicle-mounted system can determine the target frames associated with the target type sound event in the audio data to be detected, and take each two adjacent target frames and the video frames between the two target frames as an analysis group. If the number of audio frames between the current group of adjacent target frames is greater than a preset interval threshold, the end time of the current sound event can be determined according to the earlier target frame among the adjacent target frames, and the start time of the next sound event can be determined according to the later target frame among the adjacent target frames. If the number of audio frames between the current group of adjacent target frames is less than the preset interval number, it can be determined that the sound event continues to occur in the current group of audio frames, and the number of audio frames between the next group of adjacent target frames is compared with the preset interval number to determine the end time of the current sound event.

[0099] Figure 2B is a schematic diagram for positioning the start and end time of a sound event according to an embodiment of the present application, Figure 2B In the example shown in A, the number of audio frames between the second and third target frames is 4, which is less than the interval threshold 5, so it is determined that the target type sound event continues between the second and third target frames. As shown in B, the number of audio frames between the second and third target frames is 5, which is equal to the interval threshold, so it is determined that the target type sound event ends at the third target frame. Figure 2B In the example shown in A, the number of audio frames between the second and third target frames is 4, which is less than the interval threshold 5, so it is determined that the target type sound event continues between the second and third target frames. As shown in B, the number of audio frames between the second and third target frames is 5, which is equal to the interval threshold, so it is determined that the target type sound event ends at the third target frame. Figure 2BIn the target frame distribution shown in B, the interval between the second and third target frames is 6 video frames, which is greater than the interval threshold 5, so it is determined that the target type sound event is interrupted between the second and third target frames, the end time of the current sound event is determined according to the second target frame, and the start time of the next sound event is determined according to the third target frame.

[0100] It should be noted that, for the above-mentioned determination of the start time or end time of the sound event according to the target frame, the vehicle-mounted system can directly use the time point associated with the target frame as the start time or end time of the sound event, or can determine the start time or end time of the sound event according to the time point associated with the target frame according to a preset time compensation rule. For example, a preset time length is added or subtracted from the time point associated with the target frame as the start time or end time of the sound event.

[0101] This scheme can adapt to the perceptual time interval of human hearing, and is beneficial to the completeness and fluency of the presentation of the sound event.

[0102] In a preferred scheme, after determining the start and end times of the sound events in the to-be-detected audio data according to the number of audio frames between adjacent target frames, the method further comprises:

[0103] If the number of audio frames associated with each sound event is greater than a preset continuous number, the start and end times of each sound event are corrected according to the number of target frames in each continuous number of audio frames in the audio frames associated with each sound event.

[0104] On the basis of the above-mentioned scheme, the vehicle-mounted system can correct the start and end times of each sound event. The vehicle-mounted system can determine the audio frames covered by each sound event and calculate the number of audio frames. If the number of audio frames of the sound event is greater than a preset continuous number, it means that the duration of the sound event is too long, and there may be a misjudgment of the target frame. The vehicle-mounted system can determine whether the start and end times of the sound event need to be corrected according to the number of target frames in each continuous number of audio frames in the audio frames associated with the sound event.

[0105] Taking a specific example, the continuous threshold can be set to 10, for example, if the number of audio frames associated with a sound event is 15, the vehicle-mounted system can determine that the start and end times of the sound event need to be corrected. Figure 2B In the target frame distribution shown in C, according to the aforementioned positioning method of the start and end times of the sound event, the sound event occurs continuously between the second target frame and the fourth target frame. Only two of the 10 consecutive audio frames starting from the second target frame are target frames, which means that Figure 2B The third target frame in C is misjudged, and the sound event needs to be divided. The vehicle-mounted system can determine the end time of the sound event before the second target frame and the start time of the sound event after the fourth target frame.

[0106] Figure 2CIt is a voice event start and end time positioning result schematic diagram provided according to an embodiment of the present application. Based on the above scheme, each bar grid can represent an audio frame, and the start and end time can be determined according to the audio frame covered by the voice event. Figure 2C

[0107] The present scheme can realize accurate positioning of the start and end time of the voice event, and is beneficial to avoiding the influence of the target frame misjudgment on the integrity and fluency of the voice event.

[0108] The technical scheme determines the voice event recognition result of the to-be-detected audio data through the acoustic timing feature of the to-be-detected audio data and the prior voice event relationship feature, determines the target frame in the to-be-detected audio data according to the voice event recognition result, and further determines the start and end time of the voice event in the to-be-detected audio data according to the number of audio frames between adjacent target frames. The technical scheme solves the problem of low accuracy of voice event start and end time positioning, and can greatly improve the accuracy of voice event start and end time positioning.

[0109] Embodiment three

[0110] Figure 3 A structure schematic diagram of a voice event start and end time determination device provided for an embodiment of the present application is shown in FIG. 3. Figure 3 As shown in the figure, the device comprises:

[0111] An acoustic timing feature determination module 310 is configured to determine the acoustic timing feature of the to-be-detected audio data based on a convolutional neural network and a recurrent convolutional neural network.

[0112] An identification result determination module 320 is configured to determine the voice event recognition result of the to-be-detected audio data according to the acoustic timing feature and a pre-determined voice event relationship feature. The voice event relationship feature is determined based on a pre-constructed voice event relationship graph. The voice event relationship graph is determined according to the statistical result of the voice event in the audio data set.

[0113] A target frame determination module 330 is configured to determine the target frame in the to-be-detected audio data according to the voice event recognition result.

[0114] A start and end time determination module 340 is configured to determine the start and end time of the voice event in the to-be-detected audio data according to the number of audio frames between adjacent target frames.

[0115] In the present scheme, optionally, the device further comprises a relationship feature determination module, which comprises:

[0116] A label information acquisition unit is configured to acquire the label information of each audio data in the audio data set.

[0117] ​The commonness determination unit is configured to determine a time sequence relationship between the sound events of different types according to the label information of the audio data, determine an occurrence probability matching the time sequence relationship, and determine the commonness of the sound events of different types;

[0118] The relationship graph determination unit is configured to construct a sound event relationship graph according to the commonness of the sound events of different types, the time sequence relationship between the sound events of different types, and the occurrence probability matching the time sequence relationship.

[0119] The relationship feature determination unit is configured to determine a sound event relationship feature based on a graph convolutional neural network according to the sound event relationship graph.

[0120] In the above scheme, the label information includes the types of sound events present in the audio data and the start and end times of the sound events.

[0121] The commonness determination unit includes:

[0122] The time sequence relationship determination subunit is configured to determine the occurrence frequencies of the sound events of different types according to the types of sound events present in the audio data, and determine the time sequence relationship between the sound events of different types according to the types of sound events present in the audio data and the start and end times of the sound events in the audio data.

[0123] The occurrence probability determination subunit is configured to determine the occurrence probability matching the time sequence relationship according to the time sequence relationship between the sound events of different types and the occurrence frequencies of the sound events of different types.

[0124] The frequency and time proportion determination subunit is configured to determine the occurrence frequencies of the sound events of different types according to the occurrence frequencies of the sound events of different types, and determine the time proportions of the sound events of different types in all audio data according to the start and end times of the sound events.

[0125] The commonness determination subunit is configured to determine the commonness of the sound events of different types according to the occurrence frequencies and the time proportions.

[0126] In the above scheme, the commonness calculation formula is:

[0127]

[0128] wherein, represents the occurrence frequency, represents the time proportion, σ represents an adjustment parameter, σ ∈ [0, 1], and i represents a sound event type index.

[0129] In the above scheme, the relationship graph determination unit includes:

[0130] The first relationship graph determining sub-unit is configured to, if the commonness of each type of sound event is greater than a preset commonness threshold, construct a sound event relationship graph according to the time sequence relationship between each type of sound event and the occurrence probability matched with the time sequence relationship.

[0131] The second relationship graph determining sub-unit is configured to, if the commonness of the target type of sound event is less than or equal to the preset commonness threshold and the target type of sound event is in the pre-set important uncommon event set, correct the occurrence probability matched with the time sequence relationship associated with the target type of sound event according to a preset probability correction principle, and construct a sound event relationship graph according to the time sequence relationship between each type of sound event and the corrected occurrence probability matched with the time sequence relationship.

[0132] Optionally, the sound event recognition result includes a recognition probability of each type of sound event; and the to-be-detected audio data includes at least two audio frames.

[0133] The recognition result determining module 320 includes:

[0134] The joint feature determining unit is configured to fuse the acoustic time sequence feature and the sound event relationship feature to obtain a joint feature.

[0135] The recognition probability determining unit is configured to determine, according to the joint feature, a recognition probability of each type of sound event in each audio frame based on the sound event recognition network.

[0136] The target frame determining module 330 is specifically configured to:

[0137] Determine a target frame of each type of sound event according to the recognition probability of each type of sound event in each audio frame.

[0138] On the basis of the above scheme, the start and end time determining module 340 is specifically configured to:

[0139] If the number of audio frames between the adjacent target frames in the current group is greater than or equal to a preset interval threshold, determine the end time of the current sound event according to the target frame in the front of the adjacent target frames and determine the start time of the next sound event according to the target frame in the back of the adjacent target frames.

[0140] If the number of audio frames between the adjacent target frames in the current group is less than the preset interval number, determine the end time of the current sound event according to the number of audio frames between the adjacent target frames of at least one group after the current group.

[0141] In one preferred scheme, the apparatus further includes:

[0142] The time correction module is configured to correct the start and end time of each sound event according to a target frame number in each continuous number of audio frames associated with the sound event, if the number of audio frames associated with each sound event is greater than the preset continuous number.

[0143] The sound event start and end time determination apparatus provided by the embodiments of the present application can perform the sound event start and end time determination method provided by any of the embodiments of the present application, and has the corresponding function modules and advantages of performing the method.

[0144] Embodiment four

[0145] Figure 4 A structural schematic diagram of an electronic device 410 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0146] As shown in Figure 4 The electronic device 410 includes at least one processor 411, and a memory, such as a read-only memory (ROM) 412, a random access memory (RAM) 413, etc., connected to the at least one processor 411 in communication, wherein the memory stores a computer program executable by the at least one processor 411, and the processor 411 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 412 or loaded into the random access memory (RAM) 413 from the storage unit 418. In the RAM 413, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 411, the ROM 412, and the RAM 413 are connected to each other through a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0147] The various components in the electronic device 410 are connected to the I / O interface 415, including an input unit 416, such as a keyboard, a mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a magnetic disk, an optical disk, etc.; and a communication unit 419, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 419 allows the electronic device 410 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunications networks.

[0148] The processor 411 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, and the like. The processor 411 performs various methods and processes described above, such as the determination method of sound event start and end times.

[0149] In some embodiments, the determination method of sound event start and end times can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 418. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 410 via the ROM 412 and / or the communication unit 419. When the computer program is loaded onto the RAM 413 and executed by the processor 411, one or more steps of the determination method of sound event start and end times described above can be performed. Alternatively, in other embodiments, the processor 411 can be configured to perform the determination method of sound event start and end times by any other suitable means, such as by means of firmware.

[0150] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0151] Computer programs used to implement the methods of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, and partially on a machine or a remote machine or a server.

[0152] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0153] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0154] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), blockchain network, and the Internet.

[0155] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0156] It should be understood that the various forms of flow shown above can be reordered, added to, or have steps deleted. For example, the steps described in the present application can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, which are not limited herein.

[0157] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method of determining the start and end times of a sound event, characterized by, The method comprises: determining acoustic timing features of the audio data to be detected based on a convolutional neural network and a recurrent convolutional neural network; determining a sound event recognition result of the audio data to be detected according to the acoustic timing features and predetermined sound event relationship features; wherein the sound event relationship features are determined based on a pre-constructed sound event relationship graph; and the sound event relationship graph is determined according to statistical results of sound events in an audio data set; determining target frames in the audio data to be detected according to the sound event recognition result; determining the start and end times of sound events in the audio data to be detected according to the number of audio frames between adjacent target frames; wherein the determination process of the sound event relationship features comprises: obtaining label information of each audio data in the audio data set; determining the timing relationship between each type of sound event, determining the occurrence probability matching the timing relationship, and determining the commonness of each type of sound event according to the label information of each audio data; constructing a sound event relationship graph according to the commonness of each type of sound event, the timing relationship between each type of sound event, and the occurrence probability matching the timing relationship; determining the sound event relationship features based on a graph convolutional neural network according to the sound event relationship graph.

2. The method of claim 1, wherein, The label information comprises the type of sound event present in the audio data and the start and end times of the sound event; The determination of the timing relationship between each type of sound event, the determination of the occurrence probability matching the timing relationship, and the determination of the commonness of each type of sound event according to the label information of each audio data comprises: determining the number of occurrences of each type of sound event according to the type of sound event present in each audio data; and determining the timing relationship between each type of sound event according to the type of sound event present in each audio data and the start and end times of the sound event in each audio data; determining the occurrence probability matching the timing relationship according to the timing relationship between each type of sound event and the number of occurrences of each type of sound event; determining the occurrence frequency of each type of sound event according to the number of occurrences of each type of sound event, and determining the time proportion of each type of sound event with respect to all audio data according to the start and end times of the sound event; determining the commonness of each type of sound event according to the occurrence frequency and the time proportion.

3. The method of claim 1, wherein, The construction of the sound event relationship graph according to the commonness of each type of sound event, the timing relationship between each type of sound event, and the occurrence probability matching the timing relationship comprises: if the commonness of each type of sound event is greater than a preset commonness threshold, constructing the sound event relationship graph according to the timing relationship between each type of sound event and the occurrence probability matching the timing relationship; if the commonness of a target type of sound event is less than or equal to the preset commonness threshold, and the target type of sound event is in a pre-set important uncommon event set, then the occurrence probability matching the timing relationship associated with the target type of sound event is corrected according to a preset probability correction principle, and the sound event relationship graph is constructed according to the timing relationship between each type of sound event and the corrected occurrence probability matching the timing relationship.

4. The method of claim 1, wherein, The sound event recognition result comprises recognition probabilities of each type of sound event; and the audio data to be detected comprises at least two audio frames; The method comprises the following steps: fusing the acoustic time sequence feature and the sound event relationship feature to obtain a joint feature; determining, according to the joint feature, recognition probabilities of each type of sound event in each audio frame based on a sound event recognition network; The method comprises the following steps: determining, according to the recognition probabilities of each type of sound event in each audio frame, target frames of each type of sound event.

5. The method of claim 1, wherein, The method comprises the following steps: if the number of audio frames between the adjacent target frames in the current group is greater than or equal to the preset interval threshold, determining the end time of the current sound event according to the earlier target frame among the adjacent target frames and determining the start time of the next sound event according to the later target frame among the adjacent target frames; if the number of audio frames between the adjacent target frames in the current group is less than the preset interval number, determining the end time of the current sound event according to the number of audio frames between the adjacent target frames in at least one group of target frames after the current group.

6. The method of claim 2, wherein, After determining the start and end times of the sound events in the audio data to be detected according to the number of audio frames between the adjacent target frames, the method further comprises the following steps: if the number of audio frames associated with each sound event is greater than the preset continuous number, correcting the start and end times of each sound event according to the number of target frames in each continuous number of audio frames associated with each sound event.

7. An apparatus for determining the start and end times of a sound event, characterized by The method comprises the following steps: an acoustic time sequence feature determination module configured to determine acoustic time sequence features of the audio data to be detected based on a convolutional neural network and a recurrent convolutional neural network; an identification result determination module configured to determine sound event recognition results of the audio data to be detected according to the acoustic time sequence features and predetermined sound event relationship features; wherein the sound event relationship features are determined based on a pre-constructed sound event relationship graph; and the sound event relationship graph is determined according to statistical results of sound events in an audio data set; a target frame determination module configured to determine target frames in the audio data to be detected according to the sound event recognition results; a start and end time determination module configured to determine start and end times of sound events in the audio data to be detected according to the number of audio frames between the adjacent target frames; The device further comprises a relationship feature determination module, which comprises the following steps: a label information acquisition unit configured to acquire label information of each audio data in an audio data set; a commonality determination unit configured to determine time sequence relationships between each type of sound event according to the label information of each audio data, determine occurrence probabilities matching the time sequence relationships, and determine commonalities of each type of sound event; a relationship graph determination unit configured to construct a sound event relationship graph according to the commonalities of each type of sound event, the time sequence relationships between each type of sound event, and the occurrence probabilities matching the time sequence relationships. The relationship feature determination unit is configured to determine, based on a graph convolutional neural network, a sound event relationship feature according to the sound event relationship graph.

8. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the method for determining the start and end times of a sound event according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to implement the method for determining the start and end times of a sound event according to any one of claims 1-6 when executed.

Images