Data processing method and device, equipment, storage medium and computer program product

Abstract

The embodiment of the application provides a data processing method, device, equipment, storage medium and computer program product, which can be applied to various fields or scenes such as artificial intelligence, block chain, cloud technology, intelligent transportation, smart home, vehicle-mounted and the like, wherein the method comprises: obtaining audio to be detected of an audio device to be detected; processing the audio to be detected to determine an audio feature set of the audio to be detected, the audio feature set comprising one or a combination of both of a first modal feature set and a second modal feature set; the first modal feature set is obtained by processing a frequency spectrum diagram of the audio to be detected, and the second modal feature set is obtained by processing at least two audio segments obtained by segmenting the audio to be detected; and determining a state detection result of the audio device to be detected according to the audio feature set. Through the embodiment of the application, the state detection result of the audio device can be quickly and accurately determined, and the method is simple and efficient.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular, to a data processing method, a data processing apparatus, a computer device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] Audio devices are ubiquitous in people's daily life. For example, terminal devices such as smart phones, tablet computers, notebook computers, smart voice interaction devices (such as smart speakers), and smart home appliances (such as smart televisions) are all equipped with audio devices. At present, in order to detect whether the audio device is normal, the audio device is usually sent to an audio laboratory for acoustic verification. Although this method can obtain accurate judgment results, it is not only high in cost, but also complex and low in efficiency. SUMMARY

[0003] The embodiments of the present application provide a data processing method, apparatus, device, storage medium and computer program product, which can quickly and accurately determine the state detection result of the audio device, and the method is simple and efficient.

[0004] In one aspect, the present application provides a data processing method, which comprises:

[0005] obtaining audio to be detected of an audio device to be detected;

[0006] processing the audio to be detected to determine an audio feature set of the audio to be detected, the audio feature set comprising a combination of one or both of a first modal feature set and a second modal feature set; wherein the first modal feature set is obtained by processing a frequency spectrum diagram of the audio to be detected, and the second modal feature set is obtained by processing at least two audio segments obtained by segmenting the audio to be detected;

[0007] determining a state detection result of the audio device to be detected according to the audio feature set.

[0008] In one aspect, the present application provides a data processing apparatus, which comprises:

[0009] an obtaining unit configured to obtain audio to be detected of an audio device to be detected;

[0010] a processing unit configured to process the audio to be detected to determine an audio feature set of the audio to be detected, the audio feature set comprising a combination of one or both of a first modal feature set and a second modal feature set; wherein the first modal feature set is obtained by processing a frequency spectrum diagram of the audio to be detected, and the second modal feature set is obtained by processing at least two audio segments obtained by segmenting the audio to be detected;

[0011] The processing unit is further configured to determine a state detection result of the audio device to be detected according to the audio feature set.

[0012] In an embodiment, when determining the state detection result of the audio device to be detected according to the audio feature set, the processing unit is specifically configured to:

[0013] determine a first state detection result of the audio device to be detected according to a target modality feature set included in the audio feature set, the target modality feature set including the first modality feature set or the second modality feature set; and determine the first state detection result as the state detection result of the audio device to be detected if the first state detection result indicates that the audio device to be detected is abnormal.

[0014] In an embodiment, the audio feature set includes the first modality feature set and the second modality feature set, and the processing unit is further configured to:

[0015] determine a second state detection result of the audio device to be detected according to the second modality feature set if the first state detection result indicates that the audio device to be detected is normal and the first state detection result is determined according to the first modality feature set, and determine the state detection result of the audio device to be detected according to the second state detection result; or determine a second state detection result of the audio device to be detected according to the first modality feature set if the first state detection result indicates that the audio device to be detected is normal and the first state detection result is determined according to the second modality feature set, and determine the state detection result of the audio device to be detected according to the second state detection result.

[0016] In an embodiment, the processing unit is further configured to: generate a spectrum graph of the audio to be detected; perform block processing on the spectrum graph to obtain at least two image blocks; for any image block in the at least two image blocks, determine a gradient graph of the any image block, and determine a first modality feature of the any image block according to the gradient graph; and determine the first modality feature set according to the first modality feature of each image block in the at least two image blocks.

[0017] In an embodiment, the processing unit is further configured to: perform segmentation processing on the audio to be detected to obtain at least two audio segments; for any audio segment in the at least two audio segments, determine a mel spectrum of the any audio segment, and determine a second modality feature of the any audio segment according to the mel spectrum; and determine the second modality feature set according to the second modality feature of each audio segment in the at least two audio segments.

[0018] In an embodiment, the processing unit is further configured to: for any first modality feature in the first modality feature set, determine a predicted state detection result of the audio device to be detected according to the any first modality feature; determine a first number of first modality features whose corresponding predicted state detection results are normal states and a second number of first modality features whose corresponding predicted state detection results are abnormal states according to the predicted state detection results corresponding to each first modality feature in the first modality feature set; compare the first number and the second number, and determine a state detection result of the audio device to be detected according to the first modality feature set according to a comparison result.

[0019] In an embodiment, when the acquisition unit acquires the audio to be detected of the audio device to be detected, the acquisition unit is specifically configured to: acquire test audio, send the test audio to the audio device to be detected, so that the audio device to be detected plays the test audio; acquire playing audio of the audio device to be detected through a pickup device in a process in which the audio device to be detected plays the test audio; and determine the playing audio acquired by the pickup device as the audio to be detected of the audio device to be detected.

[0020] In an embodiment, the audio device to be detected and the pickup device are placed in the same closed cavity.

[0021] In an aspect, an embodiment of the present application provides a computer device, including a processor, a communication interface, and a memory, the processor, the communication interface, and the memory are connected to each other, wherein the memory stores executable program codes, and the processor is configured to invoke the executable program codes to execute a data processing method provided by an embodiment of the present application.

[0022] Correspondingly, an embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium stores instructions, when the instructions are executed on a computer, the computer executes a data processing method provided by an embodiment of the present application.

[0023] Correspondingly, an embodiment of the present application further provides a computer program product, the computer program product includes a computer program or computer instructions, when the computer program or computer instructions are executed by a processor, steps of a data processing method provided by an embodiment of the present application are implemented.

[0024] Correspondingly, an embodiment of the present application further provides a computer program, the computer program includes computer instructions, the computer instructions are stored in a computer readable storage medium, a processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes a data processing method provided by an embodiment of the present application.

[0025] By adopting the embodiment of the present application, the audio feature set of the to-be-detected audio of the to-be-detected audio device is first determined, the audio feature set includes a first modal feature set obtained by processing a spectrum graph of the to-be-detected audio, and / or a second modal feature set obtained by processing at least two audio segments obtained by segmenting the to-be-detected audio, and then the state detection result of the to-be-detected audio device can be quickly and accurately determined according to the audio feature set, which is simple and efficient. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0027] Figure 1 The architecture of a data processing system to which the data processing method provided by the present application is applicable is shown;

[0028] Figure 2 A placement scene of a to-be-detected audio device and a sound pickup device is shown;

[0029] Figure 3 It is a flowchart of a data processing method provided by an embodiment of the present application;

[0030] Figure 4 A way of block processing of an audio spectrum graph is shown;

[0031] Figure 5 It is a flowchart of a prediction model training method provided by an embodiment of the present application;

[0032] Figure 6 It is a network structure diagram of a prediction model provided by an embodiment of the present application;

[0033] Figure 7 It is a flowchart of another prediction model training method provided by an embodiment of the present application;

[0034] Figure 8 It is a network structure diagram of another prediction model provided by an embodiment of the present application;

[0035] Figure 9 A training process of a prediction model is shown;

[0036] Figure 10 A reasoning process of a prediction model is shown;

[0037] Figure 11A flowchart illustrating an application of the prediction model is shown.

[0038] Figure 12 is a structural schematic diagram of a data processing device provided by an embodiment of the present application.

[0039] Figure 13 is a structural schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0041] The embodiments of the present application provide a data processing method to quickly and accurately determine a state detection result of a to-be-detected audio device according to an audio feature set of to-be-detected audio of the to-be-detected audio device. The data processing method provided by the embodiments of the present application can be implemented based on AI (Artificial Intelligence) technology. AI refers to the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. AI technology is a comprehensive discipline, which involves a wide range of fields; and the data processing method provided by the embodiments of the present application mainly involves machine learning (Machine Learning, ML) technology in AI technology. Machine learning usually includes artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and inductive learning.

[0042] In possible embodiments, the data processing method provided by the embodiments of the present application can also be implemented based on cloud technology and / or blockchain technology. Specifically, it can involve one or more of cloud storage and cloud database in cloud technology. For example, data (such as to-be-detected audio) required for executing the data processing method is obtained from a cloud database. For another example, the data required for executing the data processing method can be stored in the form of blocks on a blockchain; the data (such as an audio feature set and a state detection result) generated by executing the data processing method can be stored in the form of blocks on a blockchain; in addition, the data processing device executing the data processing method can be a node device in a blockchain network.

[0043] The data processing method provided by the embodiments of the present application can be applied to Figure 1 As shown in the data processing system, the data processing system comprises a data processing device 10, a to-be-detected audio device 11 and a sound pickup device 12. The to-be-detected audio device 11 can be a separate device or can be arranged on a first terminal device. In this case, the to-be-detected audio device 11 can be a sound output device such as a loudspeaker (or speaker) on the first terminal device. The sound pickup device 12 can be a separate device or can be arranged on a second terminal device. The second terminal device can be the same as or different from the first terminal device. In this case, the sound pickup device 12 can be a sound collection device such as a microphone module (which can comprise one or more microphone sensors) on the second terminal device.

[0044] The data processing device 10 can be a server or a terminal device with a data (such as audio data, image data) processing function. The data processing device can be the same as or different from the above-mentioned first terminal device and / or second terminal device. The server can be a separate physical server, a server cluster or a distributed system comprising a plurality of physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms. The terminal device can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, etc., but is not limited thereto.

[0045] The data processing method provided by the embodiments of the present application comprises but is not limited to the following steps: the to-be-detected audio device 11 acquires test audio, which can be sent by the data processing device 10 to the to-be-detected audio device 11; the to-be-detected audio device 11 plays the test audio, and the sound pickup device 12 collects the playing audio of the to-be-detected audio device 11 in the process of playing the test audio by the to-be-detected audio device 11; the sound pickup device 12 sends the collected playing audio to the data processing device 10, and the data processing device 10 determines the playing audio as the to-be-detected audio of the to-be-detected audio device 11; the data processing device 10 first determines an audio feature set of the to-be-detected audio, which comprises a first modal feature set obtained by processing a spectrum graph of the to-be-detected audio, and / or a second modal feature set obtained by processing at least two audio segments obtained by segmenting the to-be-detected audio, and then determines a state detection result of the to-be-detected audio device according to the audio feature set. In this way, the state detection result of the to-be-detected audio device can be quickly and accurately determined according to the audio feature set of the to-be-detected audio of the to-be-detected audio device. The simple and efficient method is also conducive to reducing costs.

[0046] In an implementation, to ensure the accuracy of the state detection result of the audio device 11 to be detected, the test audio needs to be ensured to be lossless, so as to avoid misjudgment caused by the damaged test audio. Figure 2 In an implementation, to reduce the loss of energy transmission and avoid the influence of external sound on the playing audio of the audio device 11 collected by the pickup device 12, as shown in

[0047] The above briefly introduces the data processing method provided by the embodiments of the present application, and the specific implementation of the data processing method is described in detail below.

[0048] Please refer to Figure 3 , Figure 3 the flowchart of the data processing method provided by the embodiments of the present application. The data processing method described in the embodiments of the present application can be applied to the data processing system as shown in Figure 1 , and can be executed by the data processing device 10 as shown in Figure 1 . The data processing method includes but is not limited to the following steps:

[0049] S301, obtaining the audio to be detected of the audio device to be detected.

[0050] In the embodiments of the present application, the audio device to be detected obtains the test audio, which can be sent by the data processing device to the audio device to be detected. The audio device to be detected plays the test audio, and the pickup device collects the playing audio of the audio device to be detected in the process of playing the test audio by the audio device to be detected. The pickup device sends the collected playing audio to the data processing device, and the data processing device determines the playing audio as the audio to be detected of the audio device to be detected.

[0051] In an implementation, to ensure the accuracy of the state detection result of the audio device to be detected, the test audio needs to be ensured to be lossless, so as to avoid misjudgment caused by the damaged test audio. In an implementation, to reduce the loss of energy transmission (the smaller the loss, the more conducive to the pickup device to collect the playing audio) and avoid the influence of external sound on the playing audio of the audio device to be detected collected by the pickup device (such as introducing noise), as shown in Figure 2 , the audio device to be detected and the pickup device can be placed in the same closed cavity, so as to effectively block the entry of external sound.

[0052] In other possible embodiments, the distance between the pickup device and the audio device to be detected can be limited to a set distance range, within which the playback audio of the audio device to be detected collected by the pickup device is better, and the set distance range can be obtained through experiments according to the device types of the pickup device and / or the audio device to be detected.

[0053] S302, processing the audio to be detected to determine a set of audio features of the audio to be detected.

[0054] In the embodiments of the present application, the set of audio features of the audio to be detected includes one or a combination of the first modal feature set and the second modal feature set. The first modal feature set and the second modal feature set are determined in different ways. The first modal feature set can be obtained by processing the frequency spectrum of the audio to be detected, and the second modal feature set can be obtained by processing at least two audio segments obtained by segmenting the audio to be detected. The source or form of each type of information can be referred to as a modality. The first modal feature set and the second modal feature set are determined in different ways, so they can be considered as two different modal feature sets.

[0055] In an embodiment, the first modal feature set of the audio to be detected can be determined by processing the audio to be detected in one or more of the following ways: pre-emphasis (the purpose of pre-emphasis is to enhance the high frequency part of the audio, so that the frequency spectrum becomes flat), sampling, framing, filtering, windowing (windowing can reduce the strength of the sidelobes after Fourier transform, and obtain a higher quality spectrum), and then performing Fourier transform on the processed audio to be detected to generate a frequency spectrum of the audio to be detected. The frequency spectrum of the audio to be detected is divided into blocks to obtain at least two image blocks. In possible embodiments, the frequency spectrum of the audio to be detected can be divided into blocks according to a set size or resolution. For example, if the resolution of the frequency spectrum of the audio to be detected is 600pixel*600pixel, and the frequency spectrum of the audio to be detected is divided into blocks according to a resolution of 60pixel*60pixel, 100 image blocks with the same resolution can be obtained. For example, as shown in FIG. 4, 41 is a frequency spectrum of the audio to be detected, and 42 and 43 are two image blocks obtained by dividing the frequency spectrum shown in 41. Figure 4

[0056] ​After the spectrum diagram is divided into at least two image blocks, a first modal feature of each image block is determined, and then a first modal feature set is determined according to the first modal features of the image blocks. The first modal feature of the image block can be determined according to a gradient diagram of the image block, and each first modal feature in the first modal feature set can be used to represent the gradient diagram feature of an image block of the spectrum diagram. In an embodiment, the first modal feature is a feature vector, and the first modal features of the image blocks can be fused (for example, spliced) to obtain a fused feature vector as the first modal feature set. The first modal features can be fused based on the arrangement order of the image blocks corresponding to the first modal features, and the image blocks can be arranged in the order from top to bottom and from left to right.

[0057] In an available embodiment, the first modal feature of the image block can be determined according to a histogram of oriented gradient (HOG) of the image block. In an embodiment, the horizontal gradient and the vertical gradient of the image block are calculated first, which can be achieved by filtering the image block by using a Sobel operator. The Sobel operator emphasizes the influence of the pixel points adjacent to the edge on the edge, and can better suppress noise.

[0058] The Sobel operator can be:

[0059] The Sobel operator is used to calculate the horizontal gradient and detect the vertical edge.

[0060] The Sobel operator is used to calculate the vertical gradient and detect the horizontal edge.

[0061] Then, the combined gradient of the horizontal and vertical direction gradients is calculated, including the amplitude and direction, and the calculation method is as shown in the following formula:

[0062]

[0063]

[0064] Wherein, Ix and Iy represent the gradient values in the horizontal and vertical directions, M(x, y) represents the amplitude value of the gradient, and θ(x, y) represents the direction of the gradient.

[0065] According to the calculated combined gradient, the histogram of oriented gradient (HOG) of the image block is determined, and the HOG feature is extracted for the histogram of oriented gradient, and the extracted HOG feature can be used as the first modal feature of the image block.

[0066] In an embodiment, the manner of processing the to-be-detected audio to determine the second modality feature set of the to-be-detected audio can be: segmenting the to-be-detected audio to obtain at least two audio segments. In a feasible implementation, the to-be-detected audio can be segmented according to a set frame length, for example, if the total frame length of the to-be-detected audio is 10 s, and the to-be-detected audio is segmented according to a frame length of 20 ms, 200 audio segments with the same frame length can be obtained.

[0067] After the to-be-detected audio is processed to obtain at least two audio segments, the second modality feature of each audio segment is determined, and then the second modality feature set is determined according to the second modality feature of each audio segment. The second modality feature of the audio segment can be determined according to the Mel spectrum of the audio segment, and each second modality feature in the second modality feature set can be used to represent the Mel spectrum feature of an audio segment of the to-be-detected audio. In an embodiment, the second modality feature is a feature vector, and the second modality features of the audio segments can be fused (for example, spliced) to obtain a fused feature vector, which is taken as the second modality feature set. The second modality features can be fused based on the arrangement order of the audio segments corresponding to the second modality features, and the audio segments can be arranged in the order from early to late according to the playing time.

[0068] In an embodiment, the manner of determining the second modality feature of the audio segment can be: performing one or more of the following processing on the audio segment, such as pre-emphasis (the purpose of pre-emphasis is to enhance the high frequency part of the audio, and the spectrum can become flat), sampling, framing, filtering, windowing (windowing can reduce the intensity of the sidelobes after Fourier transform, and a higher quality spectrum can be obtained, for example, a Hamming window can be used), and then performing Fourier transform on the processed audio segment to obtain the spectrum of the audio segment. Then the spectrum of the audio segment is mapped to the Mel scale using a triangular overlapping window, and the Mel spectrum of the audio segment is obtained; further, Mel features are extracted from the Mel spectrum, and the extracted Mel features are taken as the second modality feature of the audio segment. In a feasible implementation, the Mel spectrum of the audio segment is sequentially processed, such as taking logarithm and discrete cosine transform, and the Mel frequency cepstral coefficient (MFCC) on the Mel spectrum can be obtained after processing, which can be taken as the Mel feature of the audio segment, that is, the second modality feature.

[0069] S303, determining a state detection result of the to-be-detected audio device according to the audio feature set.

[0070] In an embodiment, a first state detection result of the audio device to be detected is determined according to a target modality feature set included in the audio feature set, the target modality feature set including the first modality feature set or the second modality feature set.

[0071] In an embodiment, when the target modality feature set is the first modality feature set, for any first modality feature in the first modality feature set, the any first modality feature is processed to determine a predicted state detection result of the audio device to be detected corresponding to the any first modality feature; a first number of first modality features in the first modality feature set corresponding to the predicted state detection result of a state normal and a second number of first modality features in the first modality feature set corresponding to the predicted state detection result of a state abnormal are determined according to the predicted state detection result corresponding to each first modality feature in the first modality feature set; the first number and the second number are compared, and a state detection result of the audio device to be detected determined according to the first modality feature set is determined according to a comparison result. In an embodiment, if the comparison result indicates that the first number is greater than the second number, the state detection result of the audio device to be detected is determined as a state normal; otherwise, if the comparison result indicates that the first number is less than the second number, the state detection result of the audio device to be detected is determined as a state abnormal. It should be noted that if the comparison result indicates that the first number is equal to the second number, it is not possible to determine whether the state detection result of the audio device to be detected is a state abnormal or a state normal, and in this case, a final state detection result of the audio device to be detected can be determined in combination with a state detection result of the audio device to be detected determined based on the second modality feature set.

[0072] Similarly, when the target modal feature set is the second modal feature set described above, for any second modal feature in the second modal feature set, the any second modal feature is processed to determine a prediction state detection result of the audio device to be detected corresponding to the any second modal feature; according to the prediction state detection results corresponding to each second modal feature in the second modal feature set, a third quantity of second modal features in the second modal feature set corresponding to the prediction state detection results of the state normal, and a fourth quantity of second modal features in the second modal feature set corresponding to the prediction state detection results of the state abnormal are determined; the third quantity and the fourth quantity are compared, and according to the comparison result, a state detection result of the audio device to be detected determined according to the second modal feature set is determined. In a feasible implementation manner, if the comparison result indicates that the third quantity is greater than the fourth quantity, it is determined that the state detection result of the audio device to be detected is state normal; otherwise, if the comparison result indicates that the third quantity is less than the fourth quantity, it is determined that the state detection result of the audio device to be detected is state abnormal. It should be noted that if the comparison result indicates that the third quantity is equal to the fourth quantity, it is not possible to determine whether the state detection result of the audio device to be detected is state abnormal or state normal, at this time, the final state detection result of the audio device to be detected can be determined in combination with the state detection result of the audio device to be detected determined based on the first modal feature set.

[0073] In a feasible implementation manner, the processing of the first modal feature set described above to determine the prediction state detection result of the audio device to be detected corresponding to each first modal feature can be inputting the first modal feature set into the first prediction model for processing, and after the processing of the first prediction model, the prediction state detection result of the audio device to be detected corresponding to each first modal feature can be obtained. Wherein, the inputting of the first modal feature set into the first prediction model described above can be inputting each first modal feature into the first prediction model in sequence or through multiple channels in parallel, or can be inputting a fusion feature vector obtained by fusing each first modal feature corresponding to the first modal feature set into the first prediction model as a whole.

[0074] Similarly, the processing of the second modal feature set described above to determine the prediction state detection result of the audio device to be detected corresponding to each second modal feature can be inputting the second modal feature set into the second prediction model for processing, and after the processing of the second prediction model, the prediction state detection result of the audio device to be detected corresponding to each second modal feature can be obtained. Wherein, the inputting of the second modal feature set into the second prediction model described above can be inputting each second modal feature into the second prediction model in sequence or through multiple channels in parallel, or can be inputting a fusion feature vector obtained by fusing each second modal feature corresponding to the second modal feature set into the second prediction model as a whole.

[0075] The first prediction model can be based on a Residual Network (Resnet) network, and the second prediction model can be based on a Dense Convolutional Network (Desnet) network. The Resnet network can solve the problem of gradient disappearance when the network is deep. The Desnet (or DenseNet) network is a convolutional neural network with dense connections, in which there are direct connections between any two layers, that is, the input of each layer of the network is the union of the outputs of all previous layers, and the feature maps learned by the layer are directly transmitted to all subsequent layers as input.

[0076] Further, if the first state detection result indicates that the state of the audio device to be detected is abnormal, it means that the state of the audio device to be detected is extremely likely to be abnormal. In this case, the first state detection result can be directly determined as the final state detection result of the audio device to be detected. In this way, it is not necessary to determine the final state detection result of the audio device to be detected based on another modal feature set, thereby effectively saving computing resources.

[0077] If the first state detection result indicates that the state of the audio device to be detected is normal, it means that the state of the audio device to be detected is extremely likely to be normal. However, in order to further ensure the accuracy of the state detection result, the final state detection result of the audio device to be detected can be accurately determined in combination with the state detection result of the audio device to be detected determined based on another modal feature set. If the first state detection result is determined based on the first modal feature set, a second state detection result of the audio device to be detected is determined based on the second modal feature set. If the first state detection result is determined based on the second modal feature set, a second state detection result of the audio device to be detected is determined based on the first modal feature set. Finally, the state detection result of the audio device to be detected is determined based on the second state detection result. In an embodiment, if the second state detection result indicates that the state of the audio device to be detected is normal, the first state detection result or the second state detection result is determined as the final state detection result of the audio device to be detected. If the second state detection result indicates that the state of the audio device to be detected is abnormal, the second state detection result is determined as the final state detection result of the audio device to be detected.

[0078] In an implementable embodiment, if the state detection result of the audio device to be detected is determined based on the above manner, one set of modal features (the first set of modal features or the second set of modal features) of the audio to be detected can be determined first, and then another set of modal features of the audio to be detected is determined when the audio device to be detected is determined to be in a normal state based on the one set of modal features, and the final state detection result of the audio device to be detected is determined based on the other set of modal features. In this way, when the audio device to be detected is determined to be in an abnormal state based on the one set of modal features, the other set of modal features of the audio to be detected does not need to be determined, thereby effectively saving computing resources.

[0079] In another embodiment, if the first state detection result determined based on the target set of modal features (the first set of modal features or the second set of modal features) indicates that the audio device to be detected is in an abnormal state, the second state detection result of the audio device to be detected determined based on the other set of modal features can also be combined to accurately determine the final state detection result of the audio device to be detected. If the second state detection result also indicates that the audio device to be detected is in an abnormal state, the first state detection result or the second state detection result is determined as the state detection result of the audio device to be detected; if the second state detection result indicates that the audio device to be detected is in a normal state, the state detection result of the audio device to be detected is determined again based on the target set of modal features. If the state detection result determined again still indicates that the audio device to be detected is in an abnormal state, the final state detection result of the abnormal state is obtained; if the state detection result determined again indicates that the audio device to be detected is in a normal state, the final state detection result of the normal state is obtained. In this way, the possibility of misjudgment can be effectively avoided, and the accuracy of the state detection result is further improved.

[0080] In yet another embodiment, when the audio feature set of the audio to be detected determined in step S303 includes the first set of modal features and the second set of modal features, the first set of modal features and the second set of modal features can be processed in parallel by using multiple threads to determine the state detection result of the audio device to be detected based on the first set of modal features and the second set of modal features in parallel. If both state detection results indicate that the audio device to be detected is in a normal state, the final state detection result of the normal state is obtained; if either one or both of the state detection results indicate that the audio device to be detected is in an abnormal state, the final state detection result of the abnormal state is obtained.

[0081] By adopting the embodiment of the present application, the audio feature set of the to-be-detected audio of the to-be-detected audio device is first determined, the audio feature set includes a first modal feature set obtained by processing a frequency spectrum of the to-be-detected audio, and / or a second modal feature set obtained by processing at least two audio segments segmented from the to-be-detected audio, and then the state detection result of the to-be-detected audio device can be quickly and accurately determined according to the audio feature set, which is simple and efficient.

[0082] As described above, the first modal feature set can be input into the first prediction model for processing, and after the processing of the first prediction model, the predicted state detection result of the to-be-detected audio device corresponding to each first modal feature can be obtained; the second modal feature set can be input into the second prediction model for processing, and after the processing of the second prediction model, the predicted state detection result of the to-be-detected audio device corresponding to each second modal feature can be obtained. The training process of the first prediction model and the second prediction model will be introduced below. It should be noted that the computer device for training the first prediction model or the second prediction model can be the same as the data processing device for executing the data processing method described above, or can be different.

[0083] As shown in Figure 5 the training process of the first prediction model includes but is not limited to the following steps:

[0084] S501, obtaining a training audio set, the training audio set including a plurality of training audios and a sample label of each training audio.

[0085] In the embodiment of the present application, the plurality of training audios include audios played by audio devices in a normal state and audios played by audio devices in an abnormal state; when the training audio is an audio played by an audio device in a normal state, its sample label can be 1, indicating that it is a white sample (i.e. a normal sample); when the training audio is an audio played by an audio device in an abnormal state, its sample label can be 0, indicating that it is a black sample (i.e. an abnormal sample). The standard audio (i.e. the audio input to the audio device for playing) corresponding to the plurality of training audios can be one or more, but in order to ensure the diversity of the samples, each standard audio needs to be input to both the audio device in a normal state and the audio device in an abnormal state, so that the normal training audio and the abnormal training audio of each standard audio are included in the training audio set.

[0086] S502, for any training audio in the plurality of training audios, determining a first modal feature set of the any training audio, wherein the first modal feature set is obtained by processing a frequency spectrum of the any training audio.

[0087] The manner of determining the first modality feature set of any one of the plurality of training audios can refer to the manner of determining the first modality feature set of the to-be-detected audio described above. Specifically: the training audio is processed by one or more of pre-emphasis, sampling, framing, filtering, windowing, and then the processed training audio is processed by Fourier transformation to generate a spectrogram of the training audio. The spectrogram of the training audio is processed by block processing to obtain at least two image blocks. The first modality feature of each image block is determined, and then the first modality feature set of the training audio is determined according to the first modality features of the image blocks. The first modality feature of the image block can be determined according to the gradient map of the image block (which can be determined according to the direction gradient histogram of the image block). In an embodiment, the first modality feature is a feature vector, and the first modality feature of each image block can be fused (for example, spliced) to obtain a fused feature vector, which is taken as the first modality feature set of the training audio. It should be noted that the embodiments not described in detail herein can refer to the related descriptions above, which will not be described herein.

[0088] S503, generating a first training data set according to the first modality feature set of each of the plurality of training audios. The first training data set includes a plurality of groups of training data pairs, each group of training data pairs including a first modality feature set of a training audio and a sample label of the first modality feature set, and the sample label of the first modality feature set is consistent with the sample label of the corresponding training audio.

[0089] S504, training the initial first prediction model using the first training data set to obtain a trained first prediction model.

[0090] In the embodiments of the present application, the first prediction model can be based on a residual network Resnet network. As shown in FIG. 5, at least part of the network structure of the first prediction model is shown, which includes a series of convolution operations (Conv), pooling operations (Pool), fusion operations (Concat), residual network (Resnet) processing, etc. Figure 6

[0091] ​In combination with the set model training framework and model hyperparameters, the initial first prediction model is trained using the first training data set described above to continuously adjust the model structure and / or model parameters of the first prediction model until the trained first prediction model meets the convergence condition, i.e., the trained first prediction model has a high state detection accuracy, and finally obtains the trained first prediction model. In a feasible implementation, TensorFlow framework (TensorFlow is a dataflow programming-based symbolic mathematical system, which is widely used in the programming implementation of various machine learning algorithms) can be selected as the model training framework. The set model hyperparameters can include batch (batch defines how many numbers of training data are combined together, for example, 128), epoch (epoch defines the number of times of learning algorithm working in the entire training data set, for example, 1000), learning rate (for example, 0.0001), and the like.

[0092] As shown in FIG. 6, the training process of the second prediction model includes but is not limited to the following steps: Figure 7

[0093] S701, obtaining a training audio set, the training audio set including a plurality of training audios and sample labels of each training audio.

[0094] In the embodiment of the present application, the plurality of training audios include audios played by audio devices in a normal state and audios played by audio devices in an abnormal state; when the training audio is an audio played by an audio device in a normal state, the sample label thereof can be 1, indicating that it is a white sample (i.e., a normal sample); when the training audio is an audio played by an audio device in an abnormal state, the sample label thereof can be 0, indicating that it is a black sample (i.e., an abnormal sample). The standard audio (i.e., the audio input to the audio device for playing) corresponding to the plurality of training audios can be one or more, but to ensure the diversity of samples, each standard audio needs to be input to both an audio device in a normal state and an audio device in an abnormal state, so that the training audio set includes normal training audios and abnormal training audios of each standard audio.

[0095] S702, for any training audio in the plurality of training audios, determining a second modal feature set of the any training audio, wherein the second modal feature set is obtained by processing at least two audio segments segmented from the any training audio.

[0096] ​The manner of determining the second modality feature set of any one of the plurality of training audios can refer to the manner of determining the second modality feature set of the to-be-detected audio described above. Specifically: the training audio is segmented to obtain at least two audio segments; the second modality feature of each audio segment is determined, and then the second modality feature set of the training audio is determined according to the second modality features of the audio segments. The second modality feature of the audio segment can be determined according to the Mel spectrum of the audio segment, and the Mel feature (or MFCC) can be extracted from the Mel spectrum, and the extracted Mel feature is taken as the second modality feature of the audio segment. In an embodiment, the second modality feature is a feature vector, and the second modality features of the audio segments can be fused (for example, spliced) to obtain a fused feature vector, and the fused feature vector is taken as the second modality feature set of the training audio. It should be noted that the embodiments not described in detail herein can refer to the related descriptions described above, and will not be described herein.

[0097] S703, generating a second training data set according to the second modality feature set of each training audio in the plurality of training audios. The second training data set includes a plurality of training data pairs, each training data pair including the second modality feature set of a training audio and a sample label of the second modality feature set, and the sample label of the second modality feature set is consistent with the sample label of the corresponding training audio.

[0098] S704, training the initial second prediction model using the second training data set to obtain a trained second prediction model.

[0099] In the embodiments of the present application, the second prediction model can be based on a Desnet network. As shown in FIG. 7, a part of the network structure of the second prediction model is shown, which includes a series of convolution operations (for example, 3*3 in FIG. 7 represents a convolution operation using a 3*3 convolution kernel, and the front value 16 represents the number of channels or convolution kernels, Convert to RGBA (RGBA is a color space representing Red, Green, Blue and Alpha), Maximum, etc.). Figure 8 Figure 8

[0100] ​​In combination with the set model training framework and model hyperparameters, the initial second prediction model is trained using the above-mentioned second training data set to continuously adjust the model structure and / or model parameters of the second prediction model until the trained second prediction model meets the convergence condition, i.e., the trained second prediction model has a high state detection accuracy, and finally obtains the trained second prediction model. In a feasible implementation manner, TensorFlow framework can be selected as the model training framework, and the set model hyperparameters can include batch parameter batch (e.g., 128), epoch (e.g., 5000), learning rate learning rate (e.g., 0.0001), and the like.

[0101] As shown in Figure 9 and Figure 10 , a training process and an inference process of a prediction model (or a neural network model) are respectively shown. As shown in Figure 9 , the training process of the prediction model can include but is not limited to the following steps:

[0102] Step 1: Collect data, including: collecting a training data set of the first prediction model, which is an image data set, including the frequency spectrum of the played audio of the audio device in a normal state and the frequency spectrum of the played audio of the audio device in an abnormal state. Collecting a training data set of the second prediction model, which is an audio data set, including the played audio of the audio device in a normal state and the played audio of the audio device in an abnormal state. The audio data in the audio data set can be obtained after sampling processing (e.g., sampling rate is 16k, sampling depth is 16bit) of the played audio. In a feasible implementation manner, for the convenience of processing, when the played audio of the audio device is multi-channel, it can be converted into single-channel before processing.

[0103] Step 2: Feature engineering and data enhancement, step 2 mainly uses signal data processing technology to obtain, process and extract meaningful image features and attributes from the data collected in step 1, to be sent to the two prediction models for training. Among them, the image feature engineering of the first prediction model includes the following steps: block the image files in the image data set collected in step 1 at a certain small resolution (adjustable, such as 60pixel*60pixel), divide the original image files into a plurality of image blocks. Extract HOG (Histogram of Oriented Gradients) features for each image block divided from a certain image file. The "small" HOG features extracted from each image block are connected in order, combined into a large one-dimensional vector, which is the final image feature of the certain image file. The image feature can be used to train the first prediction model (i.e., the model 1 described in Figure 9 ).

[0104] The image feature engineering of the second prediction model includes the following steps: segmenting the audio files in the audio data set collected in step 1 into a certain frame length (which can be adjusted, such as 80 sampling points, 10 milliseconds, and overlap of 40 sampling points) to divide the original audio file into a plurality of audio segments. For each audio segment divided from an audio file, such as with a sampling rate of 8k, hop_length = 512, windows as Hamming window, the Mel spectrum of the audio segment is extracted, and the Mel feature is extracted based on the Mel spectrum. The Mel features extracted from each audio segment are combined into a large one-dimensional vector, which is the final audio feature of the audio file. The audio feature can be used to train the second prediction model (i.e. the model 2 described in the Figure 9

[0105] Step 3: model design, the present application can adopt a binary classification method for classification, and some network models are filtered according to two conditions of whether containing a residual network substructure and whether containing batch normalization BatchNormalization. The present application can finally design a first prediction model (model architecture is shown in, for example, FIG. 3) with Resnet network as a backbone network model, and design a second prediction model (model architecture is shown in, for example, FIG. 4) with Desnet network as a backbone network model. Figure 6 Figure 8

[0106] Step 4: training model, according to the prediction model designed in step 3, the parameters and data required by the network model are met, TensorFlow framework can be selected as the training framework, the following hyperparameters batch (such as 128), epoch (such as 1000), learning rate (such as 0.0001) are set for the first prediction model, the following hyperparameters batch (such as 128), epoch (such as 5000), learning rate (such as 0.0001) are set for the second prediction model; then the image features (including the corresponding category labels or sample labels) determined in step 2 are used to train the first prediction model to obtain the final first prediction model; the audio features (including the corresponding category labels or sample labels) determined in step 2 are used to train the second prediction model to obtain the final second prediction model. In an embodiment, the obtained prediction model can be stored in a pb format file.

[0107] ​​​Step 5: model application, including application of the first prediction model obtained in step 4, obtaining an image file to be detected (which can be a spectrum image of playing audio of an audio device to be detected), performing processing such as image blocking and HOG feature extraction on the image file, inputting the HOG features of each image block into the first prediction model for prediction processing, and obtaining a prediction state detection result corresponding to each image block. If the number of image blocks corresponding to the prediction state detection result as abnormal is greater than the number of image blocks corresponding to the prediction state detection result as normal, it is determined that the final state detection result is abnormal, and vice versa.

[0108] Application of the second prediction model obtained in step 4, as shown in Figure 11 , includes the following steps: obtaining an audio file to be detected, which can be playing audio of an audio device to be detected, and when the audio file is multi-channel, it can be converted into a single channel first; performing processing such as frame segmentation and Mel feature extraction on the audio file, inputting the Mel features of each audio segment into the first prediction model for prediction processing, and obtaining a prediction state detection result corresponding to each audio segment. If the number of audio segments corresponding to the prediction state detection result as abnormal is greater than the number of audio segments corresponding to the prediction state detection result as normal, it is determined that the final state detection result is abnormal, and vice versa.

[0109] It should be noted that the execution subject for performing each step in the above method embodiment can be composed of hardware, software, or a combination of software and hardware.

[0110] Please refer to Figure 12 , Figure 12 for a structural schematic diagram of a data processing apparatus provided in the embodiments of the present application. The data processing apparatus described in the embodiments of the present application corresponds to the data processing device described above, and the apparatus includes:

[0111] The acquisition unit 121 is configured to acquire audio to be detected of an audio device to be detected.

[0112] The processing unit 122 is configured to process the audio to be detected and determine an audio feature set of the audio to be detected, the audio feature set including one or a combination of the first modality feature set and the second modality feature set; wherein the first modality feature set is obtained by processing a spectrum image of the audio to be detected, and the second modality feature set is obtained by processing at least two audio segments obtained by segmenting the audio to be detected;

[0113] The processing unit 122 is further configured to determine a state detection result of the audio device to be detected according to the audio feature set.

[0114] In an embodiment, the processing unit 122 is specifically configured to:

[0115] determine a first state detection result of the audio device to be detected according to a target modal feature set included in the audio feature set, the target modal feature set including the first modal feature set or the second modal feature set; and determine the first state detection result as the state detection result of the audio device to be detected if the first state detection result indicates that the audio device to be detected is in an abnormal state.

[0116] In an embodiment, the audio feature set includes the first modal feature set and the second modal feature set, and the processing unit 122 is further configured to:

[0117] determine a second state detection result of the audio device to be detected according to the second modal feature set if the first state detection result indicates that the audio device to be detected is in a normal state and the first state detection result is determined according to the first modal feature set, and determine the state detection result of the audio device to be detected according to the second state detection result; or determine a second state detection result of the audio device to be detected according to the first modal feature set if the first state detection result indicates that the audio device to be detected is in a normal state and the first state detection result is determined according to the second modal feature set, and determine the state detection result of the audio device to be detected according to the second state detection result.

[0118] In an embodiment, the processing unit 122 is further configured to: generate a spectrum graph of the audio to be detected; perform block processing on the spectrum graph to obtain at least two image blocks; for any image block in the at least two image blocks, determine a gradient graph of the any image block, and determine a first modal feature of the any image block according to the gradient graph; and determine the first modal feature set according to the first modal feature of each image block in the at least two image blocks.

[0119] In an embodiment, the processing unit 122 is further configured to: perform segmentation processing on the audio to be detected to obtain at least two audio segments; for any audio segment in the at least two audio segments, determine a mel spectrum of the any audio segment, and determine a second modal feature of the any audio segment according to the mel spectrum; and determine the second modal feature set according to the second modal feature of each audio segment in the at least two audio segments.

[0120] In an embodiment, the processing unit 122 is further configured to: for any first modality feature in the first modality feature set, determine a predicted state detection result of the audio device to be detected according to the any first modality feature; determine a first number of first modality features whose corresponding predicted state detection result is normal state and a second number of first modality features whose corresponding predicted state detection result is abnormal state according to the predicted state detection result corresponding to each first modality feature in the first modality feature set; compare the first number and the second number, and determine a state detection result of the audio device to be detected according to the first modality feature set according to a comparison result.

[0121] In an embodiment, when the obtaining unit 121 obtains the audio to be detected of the audio device to be detected, the obtaining unit 121 is specifically configured to: obtain test audio, send the test audio to the audio device to be detected, so that the audio device to be detected plays the test audio; obtain playing audio of the audio device to be detected by a pickup device in a process in which the audio device to be detected plays the test audio; and determine the playing audio obtained by the pickup device as the audio to be detected of the audio device to be detected.

[0122] In an embodiment, the audio device to be detected and the pickup device are placed in the same closed cavity.

[0123] It can be understood that the functions of the functional units of the data processing apparatus provided in the present application can be specifically implemented according to the methods in the above method embodiments, and the specific implementation process can refer to the related description in the above method embodiments, which will not be described here.

[0124] In an embodiment, the data processing apparatus provided in the present application can be implemented in a software manner, and the data processing apparatus can be stored in a memory in the form of a program and a plug-in, and include a series of units, including an obtaining unit and a processing unit; wherein the obtaining unit and the processing unit are configured to implement the data processing method provided in the present application.

[0125] In other possible embodiments, the data processing apparatus provided by the embodiments of the present application can also be implemented in a combination of software and hardware, for example, the data processing apparatus provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to perform the data processing method provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can use one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic elements.

[0126] According to the embodiments of the present application, the audio feature set of the to-be-detected audio of the to-be-detected audio device is determined first, the audio feature set includes a first modal feature set obtained by processing the frequency spectrum graph of the to-be-detected audio, and / or a second modal feature set obtained by processing at least two audio segments obtained by segmenting the to-be-detected audio, and then the state detection result of the to-be-detected audio device can be quickly and accurately determined according to the audio feature set, which is simple and efficient.

[0127] Please refer to Figure 13 , Figure 13 A structural schematic diagram of a computer device provided by the embodiments of the present application is shown. The computer device described in the embodiments of the present application corresponds to the data processing device or data processing apparatus described above, and includes a processor 131, a communication interface 132 and a memory 133. The processor 131, the communication interface 132 and the memory 133 can be connected by a bus or other means, and the embodiments of the present application take the connection by the bus as an example.

[0128] The processor 131 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device, which can parse various instructions in the computer device and process various data of the computer device. For example, the CPU can be used to parse the on-off instructions sent by the user to the computer device and control the computer device to perform on-off operation; for another example, the CPU can transmit various interactive data between the internal structures of the computer device, and the like. The communication interface 132 can optionally include a standard wired interface, a wireless interface (such as Wi-Fi, a mobile communication interface, etc.), and is controlled by the processor 131 to receive and send data. The memory 133 is a memory device in the computer device, used to store programs and data. It can be understood that the memory 133 herein can include the built-in memory of the computer device, and of course can also include the expansion memory supported by the computer device. The memory 133 provides a storage space that stores the operating system of the computer device, which can include but is not limited to: an Android system, an iOS system, a Windows Phone system, and the like, and the present application does not make any limitation thereto.

[0129] In the embodiment of the present application, the processor 131 executes the following operations by running the executable program code in the memory 133:

[0130] Obtain the to-be-detected audio of the to-be-detected audio device; process the to-be-detected audio to determine an audio feature set of the to-be-detected audio, the audio feature set including one or a combination of both of a first modal feature set and a second modal feature set; wherein the first modal feature set is obtained by processing a frequency spectrum graph of the to-be-detected audio, and the second modal feature set is obtained by processing at least two audio segments obtained by segmenting the to-be-detected audio; and determine a state detection result of the to-be-detected audio device according to the audio feature set.

[0131] In an embodiment, when the processor 131 determines the state detection result of the to-be-detected audio device according to the audio feature set, it is specifically used for: determining a first state detection result of the to-be-detected audio device according to a target modal feature set included in the audio feature set, the target modal feature set including the first modal feature set or the second modal feature set; and if the first state detection result indicates that the state of the to-be-detected audio device is abnormal, determining the first state detection result as the state detection result of the to-be-detected audio device.

[0132] In an embodiment, the audio feature set comprises the first modality feature set and the second modality feature set, and the processor 131 is further configured to: if the first state detection result indicates that the state of the audio device to be detected is normal, and the first state detection result is determined according to the first modality feature set, then determine a second state detection result of the audio device to be detected according to the second modality feature set, and determine the state detection result of the audio device to be detected according to the second state detection result; if the first state detection result indicates that the state of the audio device to be detected is normal, and the first state detection result is determined according to the second modality feature set, then determine a second state detection result of the audio device to be detected according to the first modality feature set, and determine the state detection result of the audio device to be detected according to the second state detection result.

[0133] In an embodiment, the processor 131 is further configured to: generate a spectrum graph of the audio to be detected; perform block processing on the spectrum graph to obtain at least two image blocks; for any image block in the at least two image blocks, determine a gradient graph of the any image block, and determine a first modality feature of the any image block according to the gradient graph; and determine the first modality feature set according to the first modality feature of each image block in the at least two image blocks.

[0134] In an embodiment, the processor 131 is further configured to: perform segmentation processing on the audio to be detected to obtain at least two audio segments; for any audio segment in the at least two audio segments, determine a mel spectrum of the any audio segment, and determine a second modality feature of the any audio segment according to the mel spectrum; and determine the second modality feature set according to the second modality feature of each audio segment in the at least two audio segments.

[0135] In an embodiment, the processor 131 is further configured to: for any first modality feature in the first modality feature set, determine a predicted state detection result of the audio device to be detected according to the any first modality feature; determine a first number of first modality features whose corresponding predicted state detection result is normal, and determine a second number of first modality features whose corresponding predicted state detection result is abnormal, according to the predicted state detection result corresponding to each first modality feature in the first modality feature set; compare the first number and the second number, and determine the state detection result of the audio device to be detected according to the first modality feature set according to a comparison result.

[0136] In an embodiment, the processor 131 obtains the to-be-detected audio of the audio device to be detected, specifically for: obtaining test audio, sending the test audio to the audio device to be detected through the communication interface 132, so that the audio device to be detected plays the test audio; in the process that the audio device to be detected plays the test audio, obtaining the playing audio of the audio device to be detected through the sound pickup device; and determining the playing audio obtained by the sound pickup device as the to-be-detected audio of the audio device to be detected.

[0137] In an embodiment, the audio device to be detected and the sound pickup device are placed in the same closed cavity.

[0138] In a specific implementation, the processor 131, the communication interface 132, and the memory 133 described in the embodiments of the present application can perform the implementation mode of the data processing device described in the data processing method provided by the embodiments of the present application, and can also perform the implementation mode described in the data processing apparatus provided by the embodiments of the present application, which will not be described here.

[0139] By using the embodiments of the present application, the audio feature set of the to-be-detected audio of the audio device to be detected is first determined, the audio feature set includes a first modal feature set obtained by processing the frequency spectrum graph of the to-be-detected audio, and / or a second modal feature set obtained by processing at least two audio segments obtained by segmenting the to-be-detected audio, and then the state detection result of the audio device to be detected can be quickly and accurately determined according to the audio feature set, which is simple and efficient.

[0140] The embodiments of the present application also provide a computer readable storage medium, the computer readable storage medium stores a computer program, when the computer program runs on a computer, the computer program makes the computer execute the data processing method as described in the embodiments of the present application. The specific implementation mode can be referred to the foregoing description, which will not be described here.

[0141] The embodiments of the present application also provide a computer program product, the computer program product includes a computer program or computer instructions, the computer program or computer instructions are executed by a processor to realize the steps of the data processing method provided by the embodiments of the present application. The specific implementation mode can be referred to the foregoing description, which will not be described here.

[0142] The embodiments of the present application also provide a computer program, the computer program includes computer instructions, the computer instructions are stored in a computer readable storage medium. The processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the data processing method as described in the embodiments of the present application. The specific implementation mode can be referred to the foregoing description, which will not be described here.

[0143] It should be noted that, for the aforementioned various method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, some steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0144] Those of ordinary skill in the art can understand that all or part of the steps of the various methods in the above embodiments can be completed by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium, which can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0145] The above disclosure is only some embodiments of the present application, of course, cannot limit the scope of the right of the present application, therefore, the equivalent changes made according to the claims of the present application still fall within the scope of the present application.

Claims

1. A data processing method, characterized by, The method comprises: obtaining to-be-detected audio of a to-be-detected audio device; processing the to-be-detected audio to determine an audio feature set of the to-be-detected audio, the audio feature set comprising a combination of one or both of a first modal feature set and a second modal feature set; wherein the first modal feature set is obtained by processing a frequency spectrum diagram of the to-be-detected audio, and the second modal feature set is obtained by processing at least two audio segments obtained by segmenting the to-be-detected audio; determining a state detection result of the to-be-detected audio device according to the audio feature set; wherein the processing of the to-be-detected audio to determine the audio feature set of the to-be-detected audio comprises: generating a frequency spectrum diagram of the to-be-detected audio, performing block processing on the frequency spectrum diagram to obtain at least two image blocks, for any one of the at least two image blocks, determining a gradient diagram of the any one image block, and determining a first modal feature of the any one image block according to the gradient diagram, and determining the first modal feature set according to the first modal features of the respective image blocks in the at least two image blocks; and / or performing segmentation processing on the to-be-detected audio to obtain at least two audio segments, for any one of the at least two audio segments, determining a mel-frequency spectrum of the any one audio segment, and determining a second modal feature of the any one audio segment according to the mel-frequency spectrum, and determining the second modal feature set according to the second modal features of the respective audio segments in the at least two audio segments.

2. The method of claim 1, wherein, The determination of the state detection result of the to-be-detected audio device according to the audio feature set comprises: determining a first state detection result of the to-be-detected audio device according to a target modal feature set included in the audio feature set, the target modal feature set comprising the first modal feature set or the second modal feature set; if the first state detection result indicates that the to-be-detected audio device is abnormal, determining the first state detection result as the state detection result of the to-be-detected audio device.

3. The method of claim 2, wherein, The audio feature set comprises the first modal feature set and the second modal feature set, and the method further comprises: if the first state detection result indicates that the to-be-detected audio device is normal, and the first state detection result is determined according to the first modal feature set, determining a second state detection result of the to-be-detected audio device according to the second modal feature set, and determining the state detection result of the to-be-detected audio device according to the second state detection result; if the first state detection result indicates that the to-be-detected audio device is normal, and the first state detection result is determined according to the second modal feature set, determining a second state detection result of the to-be-detected audio device according to the first modal feature set, and determining the state detection result of the to-be-detected audio device according to the second state detection result.

4. The method of any one of claims 1-3, wherein, The method further comprises: for any one of the first modal features in the first modal feature set, determining a predicted state detection result of the to-be-detected audio device according to the any one first modal feature; determine a first quantity of the first modal features whose corresponding prediction state detection results are normal, and determine a second quantity of the first modal features whose corresponding prediction state detection results are abnormal; compare the first quantity and the second quantity, and determine the state detection result of the audio device to be detected according to the first modal feature set according to a comparison result.

5. The method of claim 1, wherein, The obtaining of the audio to be detected of the audio device to be detected comprises: obtaining test audio, and sending the test audio to the audio device to be detected, so that the audio device to be detected plays the test audio; during the playing of the test audio by the audio device to be detected, obtaining playing audio of the audio device to be detected by a pickup device; determining the playing audio obtained by the pickup device as the audio to be detected of the audio device to be detected.

6. The method of claim 5, wherein, The audio device to be detected and the pickup device are placed in the same sealed cavity.

7. A data processing apparatus, characterized by, The apparatus comprises: an obtaining unit configured to obtain audio to be detected of an audio device to be detected; a processing unit configured to process the audio to be detected, and determine an audio feature set of the audio to be detected, the audio feature set comprising one or a combination of both of a first modal feature set and a second modal feature set; wherein the first modal feature set is obtained by processing a frequency spectrum diagram of the audio to be detected, and the second modal feature set is obtained by processing at least two audio segments obtained by segmenting the audio to be detected; the processing unit is further configured to determine a state detection result of the audio device to be detected according to the audio feature set; wherein when processing the audio to be detected to determine the audio feature set of the audio to be detected, the processing unit is specifically configured to: generate a frequency spectrum diagram of the audio to be detected, perform block processing on the frequency spectrum diagram to obtain at least two image blocks, for any image block in the at least two image blocks, determine a gradient diagram of the any image block, and determine a first modal feature of the any image block according to the gradient diagram, and determine the first modal feature set according to the first modal features of each image block in the at least two image blocks; and / or perform segmentation processing on the audio to be detected to obtain at least two audio segments, for any audio segment in the at least two audio segments, determine a mel spectrum of the any audio segment, and determine a second modal feature of the any audio segment according to the mel spectrum, and determine the second modal feature set according to the second modal features of each audio segment in the at least two audio segments.

8. A computer device, comprising: comprise: a processor, a communication interface and a memory, which are connected to each other, wherein the memory stores executable program code, and the processor is configured to invoke the executable program code to execute the data processing method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, which, when executed on a computer, causes the computer to perform the data processing method according to any one of claims 1-6.

10. A computer program product, characterised in that, The computer program product comprises a computer program or computer instructions, which, when executed by a processor, implement the steps of the data processing method according to any one of claims 1-6.

Images