Crying sound detection method, device and storage medium based on storage and computing integrated architecture

Abstract

The present invention relates to a crying detection method, device and storage medium based on a storage-computing integrated architecture. The method comprises: collecting an audio signal, extracting feature information from the audio signal, parsing a pre-trained long short-term memory network, and mapping the parsed computing task to the storage-computing integrated unit based on the parameter information of the storage-computing integrated unit, inputting the feature information into the long short-term memory network, controlling the corresponding storage-computing integrated unit to perform calculations according to the operation instructions of each layer in the long short-term memory network, obtaining a classification result, and determining whether the audio signal is a preset crying signal according to the classification result. The present invention compiles and maps the long short-term memory network model to the storage-computing integrated architecture, so that computing operations can be performed directly in the memory when detecting crying. By reducing data movement and reducing computational complexity, the power consumption of data transmission is significantly reduced, and the accuracy of calculation is improved.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to a crying detection method, device and storage medium based on a storage and computing integrated architecture. Background Art

[0002] With the development of speech recognition technology, speech recognition is applied to more and more fields, such as identifying various types of crying of babies to determine the corresponding conditions of babies. For the recognition of baby crying, the methods generally used in the prior art are based on traditional sound processing algorithms, which can generally detect the crying of babies more accurately. These algorithms usually rely on analyzing the frequency, amplitude and time characteristics of the sound signal, and determine whether it is a baby crying through a preset threshold or pattern matching technology.

[0003] However, in practical applications, these traditional algorithms are difficult to accurately extract effective features in noisy environments or complex background sounds. This noise will interfere with the baby's crying signal, making it difficult for the algorithm to accurately extract effective features, resulting in an increase in false positives or missed positives. In addition, the spectrum and time domain characteristics of some background sounds are similar to those of baby crying, further increasing the difficulty of detection, thus significantly reducing the detection accuracy. Secondly, traditional methods often require high-power processing units. This is because sound signal processing usually involves a lot of calculations, such as fast Fourier transform (FFT), filtering, feature extraction, and pattern matching. These operations place high demands on the processor's computing power and power consumption. Summary of the invention

[0004] The embodiments of the present invention provide a crying detection method, device and storage medium based on a storage-computing integrated architecture. By compiling and mapping a long short-term memory network model to the storage-computing integrated architecture, calculation operations can be performed directly in the memory when detecting crying. By reducing data movement and reducing calculation complexity, the power consumption of data transmission is significantly reduced, and the accuracy of calculation is improved.

[0005] An embodiment of the present invention provides a crying sound detection method based on a storage-computing integrated architecture, including:

[0006] Collecting audio signals and extracting feature information from the audio signals;

[0007] Parse the pre-trained long short-term memory network, and map the parsed computing tasks to the storage-computation-in-one unit based on the parameter information of the storage-computation-in-one unit;

[0008] Inputting the feature information into the long short-term memory network, and controlling the corresponding storage-computation integrated unit to perform calculations according to the operation instructions of each layer in the long short-term memory network to obtain a classification result;

[0009] Determine whether the audio signal is a preset crying signal according to the classification result.

[0010] In one embodiment, parsing the pre-trained long short-term memory network includes:

[0011] Extracting the multi-layer network structure and weight parameters of the long short-term memory network;

[0012] The multi-layer network structure is pruned, and the weight parameters are quantized and subjected to low-rank decomposition processing.

[0013] In one embodiment, mapping the parsed computing task to the storage-computing-in-one unit based on the parameter information of the storage-computing-in-one unit includes:

[0014] Obtain parameter information of the storage-computing integrated unit, the parameter information including at least: storage capacity, number of computing units, bandwidth, and delay time;

[0015] According to the parameter information, the computing tasks in the input gate, forget gate and output gate after the long short-term memory network is parsed are respectively mapped to the storage and computing integrated unit.

[0016] In one embodiment, the computing task is mapped to the corresponding storage-computing integrated unit, including:

[0017] Mapping the computing amount of the computing task to the computing unit in the storage-computation-in-one unit;

[0018] The input data of the input gate, the forget gate and the output gate are stored in a storage unit tightly coupled to the computing unit.

[0019] In one embodiment, after mapping the computing task to the storage-computing integrated unit, the method further includes:

[0020] Determining a transmission path for the computing data according to a usage rate of the computing data in the computing task;

[0021] Determine the corresponding calculation order in the calculation tasks and the parallel calculation tasks in the calculation tasks.

[0022] In one embodiment, after mapping the computing task to the storage-computing integrated unit, the method further includes:

[0023] Running the LSTM network in a simulation environment using a test data set and collecting test results, the test results including output data and performance data;

[0024] The long short-term memory network is iteratively optimized according to the test results.

[0025] In one embodiment, extracting feature information from the audio signal includes:

[0026] Performing a convolution operation on the frequency spectrum of the audio signal through a convolutional layer of a convolutional neural network to extract a feature map;

[0027] The feature map is downsampled through a pooling layer of a convolutional neural network to obtain feature information in the audio signal.

[0028] In one embodiment, determining whether the audio signal is a preset crying signal according to the classification result includes:

[0029] Extracting time domain features and frequency domain features of the audio signal;

[0030] Verifying the classification result according to the time domain features and the frequency domain features to determine whether the audio signal is a preset crying signal;

[0031] If so, an alarm message is generated to prompt the user.

[0032] The embodiment of the present invention further provides a crying detection device based on a storage and computing integrated architecture, comprising:

[0033] A collection module, used to collect audio signals and extract feature information from the audio signals;

[0034] A mapping module is used to parse the pre-trained long short-term memory network and map the parsed computing task to the storage-computation-in-one unit based on the parameter information of the storage-computation-in-one unit;

[0035] A classification module, used for inputting the feature information into the long short-term memory network, controlling the corresponding storage-computation integrated unit to perform calculations according to the operation instructions of each layer in the long short-term memory network, and obtaining a classification result;

[0036] A determination module is used to determine whether the audio signal is a preset crying signal according to the classification result.

[0037] An embodiment of the present invention also provides a computer-readable storage medium storing a plurality of computer programs, wherein the computer programs can be loaded by a processor to execute the above-mentioned crying detection method based on the storage-computing integrated architecture.

[0038] As can be seen from the above, the crying detection method based on the storage-computing integrated architecture provided in the embodiment of the present application collects audio signals and extracts feature information from the audio signals, parses the pre-trained long short-term memory network, and maps the parsed computing tasks to the storage-computing integrated unit based on the parameter information of the storage-computing integrated unit, inputs the feature information into the long short-term memory network, controls the corresponding storage-computing integrated unit to perform calculations according to the operation instructions of each layer in the long short-term memory network, obtains classification results, and determines whether the audio signal is a preset crying signal according to the classification results. The present invention parses and maps the long short-term memory network model to the storage-computing integrated architecture, and can directly perform calculation operations in the storage-computing integrated unit when detecting crying. By reducing data movement and reducing computational complexity, the delay and power consumption of data transmission are significantly reduced, and the deep learning technology of the present application can automatically extract and learn complex audio features from crying, thereby achieving high-accuracy crying detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 1 is a schematic diagram of a first process of a crying sound detection method based on a storage-computing integrated architecture provided by an embodiment of the present invention;

[0041] Figure 2 is a schematic diagram of the structure of a signal processing unit provided by an embodiment of the present invention;

[0042] Figure 3 It is a flow chart of combining a storage-computation-in-one unit with a long short-term memory network provided by an embodiment of the present invention;

[0043] Figure 4 is a second flow chart of a crying sound detection method based on a storage-computing integrated architecture provided by an embodiment of the present invention;

[0044] Figure 5 It is a structural schematic diagram of a crying detection device based on a storage and computing integrated architecture provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0046] It should be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.

[0047] It should be understood that, although the various steps in the flowchart in the embodiment of the present application are displayed in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and it can be performed in other orders. Moreover, at least a portion of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and their execution order is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0048] It should be noted that, in this article, step codes such as 101, 102, etc. are used for the purpose of expressing the corresponding content more clearly and concisely, and do not constitute a substantial limitation on the sequence. When implementing the step, those skilled in the art may execute 102 first and then 101, etc., but these should all be within the scope of protection of this application.

[0049] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0050] An embodiment of the present invention provides a crying detection method based on a storage-computing integrated architecture. The executor of the crying detection method based on the storage-computing integrated architecture can be the crying detection device based on the storage-computing integrated architecture provided by the embodiment of the present invention, or a server integrated with the crying detection device based on the storage-computing integrated architecture, wherein the crying detection device based on the storage-computing integrated architecture can be implemented in hardware or software.

[0051] like Figure 1 As shown, Figure 1 1 is a flow chart of a crying sound detection method based on a storage-computing integrated architecture provided by an embodiment of the present invention. The specific flow of the crying sound detection method based on the storage-computing integrated architecture may be as follows:

[0052] 101. Collect audio signals and extract feature information from the audio signals.

[0053] In one embodiment, the crying detection method based on the integrated computing architecture provided by the present application is implemented based on the Compute-In-Memory (CIM) architecture, which can directly complete the calculation in the memory, eliminating the frequent access between calculation and storage. The integrated computing architecture can be understood as embedding computing power in the memory, and performing two-dimensional and three-dimensional matrix multiplication / addition operations with a new computing architecture, rather than optimizing on traditional logic operation units or processes. This can essentially eliminate unnecessary data movement delays and power consumption, improve the computing efficiency of artificial intelligence and reduce costs. Among them, the integrated computing architecture can be implemented using a chip. The present invention can reduce the delay and power consumption of data movement through the integrated computing processor, which can provide more massive computing power and better energy efficiency ratio. The power consumption required to identify the crying of a baby is low and the efficiency is high.

[0054] Specifically, the integrated storage and computing processor may include a signal processing unit and a neural network processing unit, wherein the signal processing unit may be used to extract feature information from the audio signal, and the neural network processing unit may be used to perform subsequent audio classification tasks to identify whether it is a preset cry, such as a baby cry. The audio signal may be acquired through a microphone array, and after the audio signal is acquired, it may be further fed into an analog filter to remove noise.

[0055] In one embodiment, the signal processing unit is shown in Figure 2, specifically including a low noise amplifier (LNA), an attenuator (ATT), a variable gain amplifier (VGA), an automatic gain controller (AGC) and a quantization circuit. Among them, LNA amplifies the input signal according to a fixed gain, the positive input end of LNA is connected to the bias voltage through a resistor, and the reverse input end of LNA is connected to the bias voltage through a resistor. Moreover, a capacitor is connected between the positive input end and the reverse output end of LNA, and a capacitor is also connected between the reverse input end and the positive output end of LNA. LNA can provide a low-noise adjustable front-end gain amplifier (such as 4-20dB), and the ATT circuit attenuates the amplified signal of LNA when a loud sound is input in the near field, so that the signal amplitude range can meet the input range requirements of the filter (BPF, Berkeley Packet Filter). In practical applications, the intensity of human voice changes dynamically, so VGA and AGC are used to compensate for the changes in the intensity of the sound signal caused by different human voice sizes to achieve a relatively constant output signal intensity. The AGC mechanism is to adjust the VGA gain by monitoring the peak value or RMS (Root Mean Square) amplitude of the VGA output signal to keep the VGA output RMS amplitude at a stable value. For example, the VGA control range can be 0~34dB, 0.5dB / step.

[0056] Furthermore, before inputting the quantization circuit, the signal output by the VGA can be half-wave rectified by a half-wave rectifier (HWR), and the quantization circuit is used to quantize the rectified signal into a digital signal pulse and output it, thereby obtaining characteristic information in the audio signal. Among them, the quantization circuit can be implemented by IAF (Integrate and Fire, also called integral emission). In this embodiment, the analog signal is directly processed in the storage unit by using the signal processing unit in the storage-computing integrated processor, such as filtering and feature extraction, to reduce energy loss in the data transmission and conversion process.

[0057] 102. Parse the pre-trained long short-term memory network, and map the parsed computing task to the storage-computation-in-one unit based on the parameter information of the storage-computation-in-one unit.

[0058] It should be noted that the CIM (computing-in-one) hardware architecture is an architecture that closely integrates storage and computing, aiming to reduce the latency and energy consumption caused by data movement. In this architecture, the storage unit and the computing unit are tightly integrated, so that data can be calculated in situ without frequent transmission between the memory and the processor. LSTM (long short-term memory) is a type of RNN (recurrent neural network) that can effectively capture the long-term dependencies of time series data. The calculation of the LSTM model mainly focuses on the gating mechanism (forget gate, input gate, output gate) and the update of the unit state, which involve a large number of matrix multiplications and vector operations.

[0059] In one embodiment, the pre-trained LSTM model is first parsed to extract its network topology, hierarchical information, weight parameters, etc. Further, the parsed model can be optimized to reduce computational complexity and storage requirements, wherein the optimization methods may include pruning, quantization, and low-rank decomposition. Then, the resources of the CIM hardware architecture are analyzed, including storage capacity, number of computing units, bandwidth, and latency, etc., and based on the resource analysis results, each computing task of the LSTM is mapped to the corresponding CIM unit.

[0060] Specifically, the above mapping process includes gating mechanism mapping and unit state update mapping, wherein the matrix multiplication and activation function operations in the gating mechanism (forget gate, input gate, output gate) of LSTM can be first mapped to the computing unit in the CIM architecture, and the storage and computing tight coupling characteristics of the CIM architecture are utilized to directly store the weights and input data required by the gating mechanism in the storage unit near the computing unit to reduce data movement. Secondly, the unit state update operation also involves a large number of matrix multiplications and vector operations, which can be mapped to the computing unit with strong computing power in the CIM architecture, and then ensure that the intermediate results required for the unit state update can be efficiently stored and accessed inside or near the computing unit. Furthermore, the data flow control mechanism (such as DMA, pipeline, etc.) provided by the CIM architecture can also be used to accelerate the data transmission and computing process, and ensure the efficient transmission of data between the computing unit and the storage unit. In addition, the parallel processing capability of the CIM architecture can be used to parallelize multiple computing tasks in the LSTM model, such as processing computing tasks of different time steps or different LSTM units on different computing units at the same time.

[0061] In one embodiment, see Figure 3The specific process of combining the storage and computing unit with the LSTM model is as follows: first, a scheduling algorithm is designed to prioritize the computationally intensive computing tasks, and then the load in each CIM unit (that is, the LSTM operator) needs to be balanced. Relying on the parallel computing mechanism in the CIM hardware resources, the scheduling algorithm design and the synchronization of balancing the CIM unit load can be achieved. This synchronization mechanism can avoid data competition and conflict. Then, by using the global clock signal, the hardware instructions corresponding to each CIM unit are generated in the simulation environment to verify the various functions and performance of the CIM architecture. If there are no problems with the functions and performance, the generated hardware instructions can be further run on the CIM hardware platform, and subsequent crying detection can be performed, such as baby crying detection.

[0062] In one embodiment, the step of pre-training the long short-term memory network may include: collecting a large amount of baby crying data, which may come from hospitals, nursery centers or online resources. At the same time, in order to enhance the generalization ability of the model, non-baby crying data should also be collected as negative samples. Then the collected audio data is pre-processed, including format unification (such as conversion to WAV format), denoising, sampling rate adjustment (such as unification to 16KHz), mono processing, etc. In addition, the audio needs to be segmented or marked for subsequent feature extraction and annotation. In deep learning models such as LSTM, useful features need to be extracted from the audio data as input. For baby crying detection, the features may include Mel frequency cepstral coefficients (MFCC), logarithmic Mel spectrograms, etc. Then a supervised learning method can be used for pre-training, such as using labeled baby crying data (i.e., it is known which audio segments contain baby crying) to train the LSTM model. During the training process, the loss function (such as cross entropy loss) is optimized to enable the model to accurately distinguish between baby crying and non-baby crying. Finally, an independent test set is used to evaluate the pre-trained LSTM model, and the model is adjusted and optimized according to the evaluation results, including adjusting the model structure, hyperparameters, etc. This embodiment does not further limit this.

[0063] 103. Input the feature information into the long short-term memory network, and control the corresponding storage and computing unit to perform calculations according to the operation instructions of each layer in the long short-term memory network to obtain the classification result.

[0064] In one embodiment, the feature information in the above audio signal may include MFCC (Mel frequency cepstral coefficients), spectral energy, short-time zero-crossing rate, etc., and then the extracted audio feature sequence is sorted according to the format required by the long short-term memory network model. For example, the feature sequence can be converted into a tensor form and normalized or standardized, and the sorted feature sequence is input into the long short-term memory network model on the storage and computing unit. In the storage and computing unit, the storage and computing units are tightly coupled, which can significantly reduce the delay and energy consumption caused by data movement. The storage and computing unit will determine the associated computing tasks according to the operation instructions in the long short-term memory network model, and then control the computing unit that maps the computing task to perform computing tasks such as matrix multiplication and activation functions.

[0065] In the LSTM network, the input features of each time step interact with the hidden state of the previous time step to capture the long-term dependencies in the audio signal. The LSTM unit controls the flow of information through mechanisms such as forget gates, input gates, and output gates, thereby achieving effective modeling of audio sequences. After being processed by the LSTM network, the output layer will receive the hidden state of the last time step as output. The output layer can use a softmax function or other classifiers to map the hidden state to the target category, such as baby crying or non-baby crying.

[0066] 104. Determine whether the audio signal is a preset crying signal according to the classification result.

[0067] In one embodiment, the classification result output by the output layer is used to determine whether the current audio is a preset crying sound, such as a baby crying, wherein a certain threshold can be set to improve the accuracy and robustness of the classification. For example, only when the confidence of the classification result exceeds a certain threshold, the current audio is considered to be a baby crying. Furthermore, if the judgment result is a baby crying, a corresponding response can be triggered, such as sounding an alarm, displaying a prompt message on a monitoring screen, recording a log, etc., which is not further limited in this application.

[0068] In one embodiment, the step of determining whether the audio signal is a preset crying signal according to the classification result may include: extracting the time domain features and frequency domain features of the audio signal, verifying the classification result according to the time domain features and frequency domain features to determine whether the audio signal is a preset crying signal, and if so, generating an alarm message to prompt the user. Specifically, taking the preset crying sound as a baby crying as an example, after obtaining the classification result, further verification is performed according to the time domain features and frequency domain features, wherein the unique time domain features of the baby crying sound may include a strong sense of rhythm of the crying sound, a long duration, a large amplitude change, etc., and the baby crying sound has a specific frequency distribution and energy concentration area in the frequency domain, and by comparing the frequency spectrum of the audio signal with the typical frequency spectrum features of the baby crying sound, it can be further confirmed whether it is a baby crying sound.

[0069] The embodiment of the present application realizes high-accuracy detection of infant crying through the hybrid signal storage and computing technology. The combination of hybrid signal processing and deep learning processing in the storage and computing unit effectively reduces power consumption, improves real-time performance and processing efficiency, reduces data transmission delay and optimizes system performance compared to traditional solutions. In practical applications, this method can accurately identify infant crying in complex backgrounds and ensure timely response.

[0070] This solution is used in edge devices. When powered, running the LSTM deep learning network has outstanding effects, but power consumption will also increase significantly, and there are extremely high requirements for battery life. The mixed signal storage and computing module can complete most of the multiplication and accumulation calculations, and power consumption will be reduced by 1 / 3. The reduction in power consumption ensures accuracy and reduces false alarms, greatly improving the user experience. The storage and computing unit includes a signal processing unit, a neural network processing unit, and a memory, and the memory is used to store sound signals and intermediate calculation results. The storage and computing integrated hardware architecture of the present invention combines storage and computing functions to efficiently handle the detection task of baby crying. The hardware architecture consists of the following main components: an audio acquisition module, a storage and computing integrated unit, and an interface module. The overall design is designed to reduce data transmission delays, reduce power consumption, and increase processing speed.

[0071] As can be seen from the above, the crying detection method based on the storage-computing integrated architecture provided in this embodiment collects audio signals and extracts feature information from the audio signals, analyzes the pre-trained long short-term memory network, and maps the parsed computing tasks to the storage-computing integrated unit based on the parameter information of the storage-computing integrated unit, inputs the feature information into the long short-term memory network, controls the corresponding storage-computing integrated unit to perform calculations according to the operation instructions of each layer in the long short-term memory network, obtains classification results, and determines whether the audio signal is a preset crying signal according to the classification results. The present invention compiles and maps the long short-term memory network model to the storage-computing integrated architecture, and can perform calculation operations directly in the memory when detecting crying. By reducing data movement and reducing computational complexity, the delay and power consumption of data transmission are significantly reduced, and the deep learning technology of the present application can automatically extract and learn complex audio features from crying, thereby achieving high-accuracy crying detection.

[0072] Continue reading Figure 4 , Figure 4 Another flowchart of a crying sound detection method based on a storage-computing integrated architecture provided by an embodiment of the present invention includes the following steps:

[0073] 201. Collect audio signals, perform convolution operation on the frequency spectrum of the audio signals through the convolution layer of the convolutional neural network, and extract feature maps.

[0074] 202. Downsample the feature map through the pooling layer of the convolutional neural network to obtain feature information in the audio signal.

[0075] First, this embodiment can use a high-sensitivity microphone and a low-noise amplifier to ensure that the baby's crying signal can be clearly captured, and the analog signal is directly processed in the storage unit based on the storage-computing integrated architecture, such as filtering and feature extraction, to reduce energy loss during data transmission and conversion. Among them, when extracting features, it can be processed by a convolutional neural network. Specifically, the convolutional neural network (CNN) includes multiple convolutional layers and pooling layers. The convolution layer uses a set of convolution kernels to perform a convolution operation on the input spectrum map to extract local feature maps. The output of each convolution kernel is processed by the ReLU activation function to introduce nonlinear characteristics. The pooling layer (such as the maximum pooling layer) downsamples the convolved feature map to reduce the data dimension and retain important features. The combination of convolution and pooling operations effectively extracts local frequency patterns in audio signals.

[0076] 203. Extract the multi-layer network structure and weight parameters of the long short-term memory network.

[0077] 204. Prune the multi-layer network structure and quantize and perform low-rank decomposition on the weight parameters.

[0078] Specifically, the long short-term memory network can be parsed to extract the network topology, hierarchical information and weight parameters. The parsed model will be optimized to reduce computational complexity and storage requirements. The complexity of the model can be reduced by pruning the multi-layer network structure. Quantization refers to converting the floating-point weights in the model to lower-precision values ​​(such as INT8 or FP16) to reduce storage and computing requirements, and low-rank decomposition refers to reducing the number of parameters by decomposing a large weight matrix into the product of smaller matrices.

[0079] When implementing the above model optimization steps, the performance and compression ratio of the model (i.e., the ratio of the original model size to the optimized model size) can also be monitored in real time. By iteratively applying pruning, quantization, and low-rank decomposition, and adjusting the corresponding hyperparameters, the complexity and storage requirements of the model can be significantly reduced while maintaining its performance.

[0080] 205. Obtain parameter information of the storage-computation-in-one unit, and map the computing tasks in the input gate, forget gate, and output gate of the long short-term memory network to the corresponding storage-computation-in-one unit according to the parameter information.

[0081] Furthermore, the resource parameter information of the CIM hardware architecture is analyzed, which may specifically include storage capacity, number of computing units, bandwidth, latency, etc. Based on the resource analysis results, the computing tasks in the input gate, forget gate, and output gate of the long short-term memory network are allocated to the corresponding CIM units.

[0082] In one embodiment, when mapping a computing task to a corresponding integrated storage and computing unit, it can also be mapped to a computing unit in the integrated storage and computing unit according to the computing amount of the computing task, and the input data of the input gate, forget gate, and output gate are stored in a storage unit that is tightly coupled with the computing unit, wherein the distance between the above-mentioned computing unit and the storage unit can be regarded as the degree of coupling between the two, and therefore a storage unit that is tightly coupled with the computing unit is also a storage unit that is closer to the computing unit.

[0083] When the LSTM model is compiled and mapped to the CIM hardware architecture, the present embodiment makes full use of the storage and computing tightly coupled characteristics of the CIM architecture, and arranges computing-intensive operations in the units tightly coupled with storage and computing as much as possible. Through reasonable mapping strategies, data flow optimization, parallel processing, resource allocation, compilation and optimization, etc., the operating efficiency and performance of the LSTM model on the CIM architecture can be significantly improved.

[0084] 206. Generate corresponding operation instructions for the multi-layer network structure of the long short-term memory network.

[0085] Specifically, corresponding hardware operation instructions are generated for each LSTM layer. These instructions need to consider data access mode, calculation order, and parallel computing. Specific instructions include data loading instructions, matrix multiplication instructions, activation function calculation instructions (such as tanh and sigmoid), and data storage instructions.

[0086] Furthermore, the transmission path of the computing data can be determined according to the usage rate of the computing data in the computing task, the corresponding computing order in the computing task, and the parallel computing tasks in the computing task. Determining the transmission path of the data in the CIM architecture can reduce the data transmission delay, and designing a time scheduling scheme can ensure that the calculations of each level of LSTM are executed in sequence, while maximizing the use of CIM hardware resources.

[0087] In one embodiment, a test data set can also be used to run a long short-term memory network in a simulation environment, and the test results can be collected. The test results include output data and performance data, and the long short-term memory network can be iteratively optimized according to the test results. Specifically, a simulator that can simulate the behavior of CIM hardware can be built first, and then the compiled LSTM model can be loaded into the CIM hardware simulator, and the LSTM model can be run in the simulation environment using the test data set, and the output results and performance data can be collected. During the test, the debugging tool is used to monitor the hardware status to ensure that no errors or exceptions occur. Then the output results of the simulation test are compared with the expected results to verify the functional correctness of the LSTM model on the CIM hardware, and the reasoning speed and resource consumption of the LSTM model on the CIM hardware can be evaluated based on the collected performance data. Finally, performance optimization is performed according to the test results, including adjusting hardware instructions, optimizing data layout, improving parallel processing strategies, etc., and iterative testing is performed after optimization to verify the optimization effect.

[0088] 207. Input the feature information into the long short-term memory network, and control the corresponding storage-computation integrated unit to perform calculations according to the operation instructions of each layer in the long short-term memory network to obtain the classification result.

[0089] 208. Determine whether the audio signal is a preset crying signal according to the classification result.

[0090] It should be noted that in the traditional method, data needs to be frequently transferred between the memory and the processor, which not only increases the delay but also consumes a lot of energy. The hybrid signal storage and computing technology provided in the embodiment of the present application significantly reduces the overhead of data movement by performing calculations directly in the storage unit, thereby achieving high efficiency in data processing. This high efficiency enables the system to respond to the baby's crying in real time and provide parents with instant monitoring feedback.

[0091] And after the introduction of deep learning technology, the system can automatically extract and learn complex audio features from the baby's crying, which can more accurately reflect the baby's emotional state than the simple statistics in traditional methods. The powerful learning ability of the deep learning model enables the system to distinguish between baby crying and other sounds (such as environmental noise, adult conversations, etc.), thereby achieving high-precision crying detection. In addition, by continuously optimizing the model structure and training data, the accuracy and robustness of detection can be further improved.

[0092] In addition, the combination of mixed signal storage and computing technology and deep learning models not only improves processing efficiency, but also achieves optimal resource allocation. On the one hand, by reducing data movement and reducing computational complexity, the system reduces the demand for hardware resources, allowing embedded devices to carry more complex algorithms and higher performance requirements. On the other hand, through technical means such as model compression and quantization, the model size can be further reduced and energy consumption can be reduced, thereby extending the use time of the device.

[0093] As can be seen from the above, the crying detection method based on the storage-computing integrated architecture provided in this embodiment collects audio signals through 201, performs a convolution operation on the frequency spectrum of the audio signal through the convolution layer of the convolutional neural network, extracts a feature map, and downsamples the feature map through the pooling layer of the convolutional neural network to obtain feature information in the audio signal, extracts the multi-layer network structure and weight parameters of the long short-term memory network, prunes the multi-layer network structure, and quantizes and performs low-rank decomposition processing on the weight parameters, obtains the parameter information of the storage-computing integrated unit, and maps the computing tasks in the input gate, forgetting gate, and output gate of the long short-term memory network to the corresponding storage-computing integrated unit according to the parameter information, generates corresponding operation instructions for the multi-layer network structure of the long short-term memory network, inputs the feature information into the long short-term memory network, and controls the corresponding storage-computing integrated unit to perform operations according to the operation instructions of each layer in the long short-term memory network to obtain a classification result, and determines whether the audio signal is a preset crying signal according to the classification result. The present invention compiles and maps the long short-term memory network model to a storage and computing integrated architecture, so that calculation operations can be performed directly in the memory when detecting crying. By reducing data movement and reducing calculation complexity, the delay and power consumption of data transmission are significantly reduced. In addition, through the deep learning technology of the present application, complex audio features can be automatically extracted and learned from infant crying, thereby achieving high-accuracy infant crying detection.

[0094] Please continue reading Figure 5 , Figure 5 1 is a schematic diagram of a structure of a crying sound detection device based on a storage-computing integrated architecture provided by an embodiment of the present invention. The crying sound detection device based on a storage-computing integrated architecture may include:

[0095] The acquisition module 301 is used to acquire audio signals and extract feature information from the audio signals;

[0096] A mapping module 302 is used to parse the pre-trained long short-term memory network and map the parsed computing task to the storage-computation-in-one unit based on the parameter information of the storage-computation-in-one unit;

[0097] The classification module 303 is used to input the feature information into the long short-term memory network, and control the corresponding storage and calculation integrated unit to perform calculations according to the operation instructions of each layer in the long short-term memory network to obtain a classification result;

[0098] The determination module 304 is used to determine whether the audio signal is a preset crying signal according to the classification result.

[0099] All of the above technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described in detail here.

[0100] As can be seen from the above, the crying detection device based on the storage-computing integrated architecture proposed in the embodiment of the present invention can collect audio signals and extract feature information from the audio signals, parse the pre-trained long short-term memory network, and map the parsed computing tasks to the storage-computing integrated unit based on the parameter information of the storage-computing integrated unit, input the feature information into the long short-term memory network, and control the corresponding storage-computing integrated unit to perform calculations according to the operation instructions of each layer in the long short-term memory network to obtain classification results, and determine whether the audio signal is a preset crying signal according to the classification results. The present invention compiles and maps the long short-term memory network model to the storage-computing integrated architecture, and can perform calculation operations directly in the memory when detecting crying. By reducing data movement and reducing computational complexity, the delay and power consumption of data transmission are significantly reduced, and the deep learning technology of the present application can automatically extract and learn complex audio features from infant crying, thereby achieving high-accuracy infant crying detection.

[0101] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0102] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of computer programs are stored, and the computer program can be loaded by a processor to execute the steps of any one of the crying detection methods based on the storage-computing integrated architecture provided in the embodiment of the present application. For example, the computer program can execute the following steps:

[0103] Collecting audio signals and extracting feature information from the audio signals;

[0104] Parse the pre-trained long short-term memory network, and map the parsed computing tasks to the storage-computation-in-one unit based on the parameter information of the storage-computation-in-one unit;

[0105] Inputting the feature information into the long short-term memory network, and controlling the corresponding storage-computation integrated unit to perform calculations according to the operation instructions of each layer in the long short-term memory network to obtain a classification result;

[0106] Determine whether the audio signal is a preset crying signal according to the classification result.

[0107] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0108] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0109] Since the computer program stored in the storage medium can execute the steps of any one of the crying detection methods based on the storage-computing integrated architecture provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the crying detection methods based on the storage-computing integrated architecture provided in the embodiments of the present application can be achieved. For details, please refer to the previous embodiments and will not be repeated here.

[0110] The above is a detailed introduction to the crying detection method, device and storage medium based on the storage and computing integrated architecture provided in the embodiment of the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A crying sound detection method based on a storage and computing integrated architecture, characterized in that: The method comprises: Collecting audio signals, and extracting characteristic information from the audio signals in a storage unit in the storage-computing integrated unit; Extracting the multi-layer network structure and weight parameters of the pre-trained long short-term memory network, pruning the multi-layer network structure, and quantizing and low-rank decomposing the weight parameters, parsing the long short-term memory network after pruning, quantization and low-rank decomposition, and mapping the parsed computing task to the storage-computation-in-one unit based on the parameter information of the storage-computation-in-one unit, wherein the computing unit in the storage-computation-in-one unit is mapped according to the computing amount of the computing task, and the input data of the input gate, forgetting gate and output gate after the parsing of the long short-term memory network are stored in a storage unit tightly coupled with the computing unit; Inputting the feature information into the long short-term memory network, and controlling the corresponding storage-computation integrated unit to perform calculations according to the operation instructions of each layer in the long short-term memory network to obtain a classification result; Determine whether the audio signal is a preset crying signal according to the classification result.

2. The crying sound detection method based on the storage and computing integrated architecture according to claim 1 is characterized in that: The method of mapping the parsed computing task to the storage-computing-in-one unit based on the parameter information of the storage-computing-in-one unit includes: Obtain parameter information of the storage-computing integrated unit, the parameter information including at least: storage capacity, number of computing units, bandwidth, and delay time; According to the parameter information, the computing tasks in the input gate, forget gate and output gate after the long short-term memory network is parsed are respectively mapped to the storage and computing integrated unit.

3. The crying sound detection method based on the storage and computing integrated architecture according to claim 1 is characterized in that: After mapping the computing task to the storage-computing integrated unit, the method further includes: Determining a transmission path for the computing data according to a usage rate of the computing data in the computing task; Determine the corresponding calculation order in the calculation tasks and the parallel calculation tasks in the calculation tasks.

4. The crying sound detection method based on the storage and computing integrated architecture according to claim 1 is characterized in that: After mapping the computing task to the storage-computing integrated unit, the method further includes: Running the LSTM network in a simulation environment using a test data set and collecting test results, the test results including output data and performance data; The long short-term memory network is iteratively optimized according to the test results.

5. The crying sound detection method based on the storage-computing integrated architecture according to any one of claims 1 to 4, characterized in that: The extracting characteristic information from the audio signal comprises: Performing a convolution operation on the frequency spectrum of the audio signal through a convolutional layer of a convolutional neural network to extract a feature map; The feature map is downsampled through a pooling layer of a convolutional neural network to obtain feature information in the audio signal.

6. The crying sound detection method based on the storage-computing integrated architecture according to any one of claims 1 to 4, characterized in that: Determining whether the audio signal is a preset crying signal according to the classification result includes: Extracting time domain features and frequency domain features of the audio signal; Verifying the classification result according to the time domain features and the frequency domain features to determine whether the audio signal is a preset crying signal; If so, an alarm message is generated to prompt the user.

7. A crying sound detection device based on a storage and computing integrated architecture, characterized in that: include: A collection module, used for collecting audio signals and extracting characteristic information in the audio signals from a storage unit in the storage and computing integrated unit; A mapping module, used to extract the multi-layer network structure and weight parameters of the pre-trained long short-term memory network, prune the multi-layer network structure, quantize and perform low-rank decomposition processing on the weight parameters, parse the long short-term memory network after pruning, quantization and low-rank decomposition, and map the parsed computing task to the storage-computation-in-one unit based on the parameter information of the storage-computation-in-one unit, wherein the computing unit in the storage-computation-in-one unit is mapped according to the computing amount of the computing task, and the input data of the input gate, forgetting gate and output gate after parsing of the long short-term memory network are stored in a storage unit tightly coupled with the computing unit; A classification module, used for inputting the feature information into the long short-term memory network, controlling the corresponding storage-computation integrated unit to perform calculations according to the operation instructions of each layer in the long short-term memory network, and obtaining a classification result; A determination module is used to determine whether the audio signal is a preset crying signal according to the classification result.

8. A computer-readable storage medium, characterized in that: Multiple computer programs are stored, and the computer programs can be loaded by a processor to execute the crying detection method based on the storage-computing integrated architecture as described in any one of claims 1-6.

Images