Streaming voice reasoning method and device, vehicle machine, storage medium and vehicle
By streaming the input voice and dimensional recovery, the problem of difficulty in scaling the convolution operator in the prior art is solved, the universality and scalability of speech inference are realized, and the computing efficiency and flexibility are improved.
Patent Information
- Application Number
- CN202310525037.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-10
- Publication Date
- 2024-11-12
AI Technical Summary
In the existing streaming speech inference technology, specific convolution operators are difficult to expand and apply to other models, and lack universality, resulting in speech inference being limited to specific operators in a certain model, and cannot achieve universality and ability across models. Expansion.
By obtaining input voice and streaming parameters, converting them to the first frame tensor, and streaming calculations of the preset tensor dimensions, including disassembly, shape adjustment and alignment, we obtain calculations for indicating the input voice context-dependent data. As a result, it is then restored to the frame tensor dimension to realize speech inference.
It improves the versatility and scalability of speech inference, so that speech inference is no longer limited to specific operators in a certain model, but can be applied to various streaming calculations, improving computing efficiency and flexibility.
Smart Images

Figure CN118942484A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of speech reasoning technology, and in particular to a streaming speech reasoning method, device, vehicle computer, storage medium and vehicle. Background Art
[0002] Speech reasoning is to input a speech into a model and obtain a specified output through model reasoning calculation. Speech reasoning includes speech recognition, speech wake-up, and speech synthesis. In the field of speech reasoning technology, receiving speech while calculating is called streaming reasoning. For example, when the car computer performs speech recognition, it can realize the display screen while the user is speaking based on streaming reasoning. Streaming reasoning will segment the speech, which may cause the loss of context data. Therefore, current streaming reasoning technology mostly focuses on the convolution operators that rely on context data in the model. Specific convolution operators are designed for specific model structures to process the speech input to the model to obtain output consistent with the reasoning effect of the whole sentence. However, the model structure is diverse and complex, and specific convolution operators are difficult to extend and apply to other models, lacking versatility. Summary of the invention
[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a streaming speech reasoning method, device and electronic device, so that speech reasoning is no longer limited to a specific operator in a certain model, thereby improving the versatility and scalability of speech reasoning.
[0004] In order to achieve the above objectives, the technical solutions provided by the embodiments of the present disclosure are as follows:
[0005] In a first aspect, the present disclosure provides a streaming speech reasoning method, comprising:
[0006] Get the input voice and streaming parameters, where the streaming parameters are used to indicate the preset tensor dimensions;
[0007] Convert the input speech into a first frame tensor, and determine the frame tensor dimension of the first frame tensor;
[0008] Performing streaming calculation on the second frame tensor corresponding to the preset tensor dimension to obtain a calculation result, wherein the streaming calculation includes at least one of disassembly, shape adjustment and alignment, and the calculation result is used to indicate context-dependent data of the input speech;
[0009] The calculation result is dimensionally restored according to the frame tensor dimension to obtain a dimensionally restored result, which is used for speech inference.
[0010] As an optional implementation of the embodiment of the present disclosure, the calculation result is a decomposition result obtained by decomposing the second frame tensor corresponding to the preset tensor dimension, including:
[0011] Determine context-dependent parameters, which include operator kernel, step size, and hole;
[0012] According to the length of the operator kernel, the second frame tensor is split to obtain multiple first frame fragments;
[0013] For each frame segment in the plurality of first frame segments, splicing adjacent frames in adjacent frame segments to obtain a spliced frame segment;
[0014] The spliced frame fragments and operator kernels are used for calculation to obtain the disassembly results.
[0015] As an optional implementation of the embodiment of the present disclosure, the calculation result is an adjustment result obtained by adjusting the shape of the second frame tensor corresponding to the preset tensor dimension, including:
[0016] determining a shape-dependent parameter, the shape-dependent parameter indicating an adjustment method and a target frame tensor to be adjusted;
[0017] According to the shape-dependent parameter, the target frame tensor in the second frame tensor is adjusted corresponding to the adjustment method to obtain an adjustment result.
[0018] As an optional implementation of the embodiment of the present disclosure, the adjustment method is slicing, the target frame tensor is the frame tensor corresponding to the shape-dependent interval, and the shape-dependent interval is used to indicate the number of bits corresponding to the start frame tensor and the number of bits corresponding to the end frame tensor;
[0019] According to the shape-dependent parameter, the target frame tensor in the second frame tensor is adjusted in a corresponding manner to obtain an adjustment result, including:
[0020] According to the number of bits corresponding to the termination frame tensor, split the second frame tensor to obtain a plurality of second frame segments, wherein the number of frame tensors of each second frame segment in the plurality of second frame segments is the same as the number of bits corresponding to the termination frame tensor;
[0021] According to the number of bits corresponding to the starting frame tensor, multiple second frame fragments are traversed until the last second frame fragment is traversed, and the target frame fragment is cached. The number of frame tensors of the target frame fragment is the same as the number of bits corresponding to the starting frame tensor;
[0022] According to the number of bits corresponding to the termination frame tensor, a target frame tensor corresponding to the shape-dependent interval is extracted from the target frame fragment as an adjustment result.
[0023] As an optional implementation of the embodiment of the present disclosure, the calculation result is an alignment result obtained by aligning the second frame tensor corresponding to the preset tensor dimension, and the second frame tensor is a plurality of frame tensors from different paths, including:
[0024] Frame alignment is performed on multiple frame tensors from different paths until the last frame tensor of the multiple frame tensors from different paths is traversed to obtain an alignment result.
[0025] As an optional implementation of the embodiment of the present disclosure, converting the input speech into a first frame tensor and determining the frame tensor dimension of the first frame tensor includes:
[0026] Convert the input speech into a first frame tensor, and determine a data format of the first frame tensor, where the data format is used to define a frame tensor dimension of the first frame tensor;
[0027] Determines the frame tensor dimensions as defined by the data format.
[0028] As an optional implementation of the embodiment of the present disclosure, converting the input speech into a first frame tensor includes:
[0029] Preprocessing the input speech, including framing and windowing;
[0030] The preprocessed input speech is formatted to obtain the first frame tensor.
[0031] In a second aspect, the present disclosure provides a streaming speech reasoning device, the device comprising:
[0032] An acquisition module, used to acquire input speech and streaming parameters, where the streaming parameters are used to indicate preset tensor dimensions;
[0033] A conversion module, used to convert the input speech into a first frame tensor and determine a frame tensor dimension of the first frame tensor;
[0034] a calculation module, configured to perform streaming calculation on the second frame tensor corresponding to the preset tensor dimension to obtain a calculation result, wherein the streaming calculation includes at least one of disassembly, shape adjustment and alignment, and the calculation result is used to indicate context-dependent data of the input speech;
[0035] The dimension recovery module is used to restore the dimension of the calculation result according to the frame tensor dimension to obtain the dimension recovery result, and the dimension recovery result is used for speech reasoning.
[0036] As an optional implementation of the embodiment of the present disclosure, the calculation result is a decomposition result obtained by decomposing the second frame tensor corresponding to the preset tensor dimension;
[0037] A calculation module, specifically used to determine context-dependent parameters, which include operator kernel, step size and hole;
[0038] According to the length of the operator kernel, the second frame tensor is split to obtain multiple first frame fragments;
[0039] For each frame segment in the plurality of first frame segments, splicing adjacent frames in adjacent frame segments to obtain a spliced frame segment;
[0040] The spliced frame fragments and operator kernels are used for calculation to obtain the disassembly results.
[0041] As an optional implementation of the embodiment of the present disclosure, the calculation result is an adjustment result obtained by adjusting the shape of the second frame tensor corresponding to the preset tensor dimension;
[0042] A calculation module, specifically used to determine a shape-dependent parameter, the shape-dependent parameter indicating an adjustment method and a target frame tensor to be adjusted;
[0043] According to the shape-dependent parameter, the target frame tensor in the second frame tensor is adjusted corresponding to the adjustment method to obtain an adjustment result.
[0044] As an optional implementation of the embodiment of the present disclosure, the adjustment method is slicing, the target frame tensor is the frame tensor corresponding to the shape-dependent interval, and the shape-dependent interval is used to indicate the number of bits corresponding to the start frame tensor and the number of bits corresponding to the end frame tensor;
[0045] The computing module adjusts the target frame tensor in the second frame tensor in a corresponding adjustment manner according to the shape-dependent parameter to obtain an adjustment result, specifically for:
[0046] According to the number of bits corresponding to the termination frame tensor, split the second frame tensor to obtain a plurality of second frame segments, wherein the number of frame tensors of each second frame segment in the plurality of second frame segments is the same as the number of bits corresponding to the termination frame tensor;
[0047] According to the number of bits corresponding to the starting frame tensor, multiple second frame fragments are traversed until the last second frame fragment is traversed, and the target frame fragment is cached. The number of frame tensors of the target frame fragment is the same as the number of bits corresponding to the starting frame tensor;
[0048] According to the number of bits corresponding to the termination frame tensor, a target frame tensor corresponding to the shape-dependent interval is extracted from the target frame fragment as an adjustment result.
[0049] As an optional implementation of the embodiment of the present disclosure, the calculation result is an alignment result obtained by aligning a second frame tensor corresponding to a preset tensor dimension, and the second frame tensor is a plurality of frame tensors from different paths;
[0050] The computing module is specifically used to perform frame alignment on multiple frame tensors from different paths until traversing to the last frame tensor of the multiple frame tensors from different paths to obtain an alignment result.
[0051] As an optional implementation of the embodiment of the present disclosure, the conversion module is specifically used to convert the input speech into a first frame tensor, determine the data format of the first frame tensor, and the data format is used to define the frame tensor dimension of the first frame tensor;
[0052] The frame tensor dimension is determined according to the data format of the first frame tensor.
[0053] As an optional implementation of the embodiment of the present disclosure, the conversion module is specifically used to convert the input speech into a first frame tensor, including:
[0054] Preprocessing the input speech, including framing and windowing;
[0055] The preprocessed input speech is formatted to obtain the first frame tensor.
[0056] In a third aspect, the present disclosure provides a vehicle computer, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the streaming speech inference method as described in the first aspect or any optional embodiment thereof is implemented.
[0057] In a fourth aspect, the present disclosure provides a computer-readable storage medium, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the streaming speech reasoning method as described in the first aspect or any one of its optional embodiments is implemented.
[0058] In a fifth aspect, the present disclosure provides a vehicle, comprising: a streaming speech inference device as described in the second aspect or any optional embodiment thereof, or a vehicle computer as described in the third aspect.
[0059] In a sixth aspect, the present disclosure provides a computer program product, including: the computer program product includes a computer program, and when the computer program runs on a computer, the computer implements the streaming speech reasoning method as described in the first aspect or any one of its optional embodiments.
[0060] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:
[0061] The present invention first obtains input speech and streaming parameters, where the streaming parameters are used to indicate a preset tensor dimension, then converts the input speech into a first frame tensor, and determines the frame tensor dimension of the first frame tensor, and then performs streaming calculations on the second frame tensor corresponding to the preset tensor dimension indicated by the streaming parameters to obtain a calculation result for indicating context-dependent data of the input speech, wherein the streaming calculation includes at least one of disassembly, shape adjustment, and alignment; further, the calculation result under the preset tensor dimension is restored to the frame tensor dimension of the first frame tensor to obtain a dimensionality recovery result for speech reasoning, which can represent the context-dependent data of the input speech and is obtained for various streaming calculations, so that speech reasoning is no longer limited to a specific operator in a certain model, thereby improving the versatility and scalability of speech reasoning. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0064] Figure 1 A flow chart of a streaming speech inference method according to an embodiment of the present disclosure;
[0065] Figure 2A A disassembled schematic diagram of an embodiment of the present disclosure;
[0066] Figure 2B A schematic diagram of the whole sentence reasoning;
[0067] Figure 3A A schematic diagram of shape adjustment provided for an embodiment of the present disclosure;
[0068] Figure 3B This is a schematic diagram of the segmentation of the whole sentence;
[0069] Figure 4 A schematic diagram of the structure of a streaming speech inference device provided in an embodiment of the present disclosure;
[0070] Figure 5 The present invention is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0071] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.
[0072] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.
[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the technical terms required to be used in the embodiments or the prior art description are briefly introduced below:
[0074] Streaming refers to real-time, continuous input, and speech itself is a streaming input. Compared with the whole sentence reasoning that starts computing and processing after receiving the complete speech, computing and processing while receiving speech is called streaming reasoning. Streaming reasoning has the characteristic of low latency.
[0075] Streaming reasoning technology first divides the entire input speech into segments. The speech itself has context-dependent characteristics. To ensure the effect of speech reasoning, the segmented speech is required to retain context feature information. Therefore, current streaming reasoning technologies mostly focus on the convolution operators that depend on context data in the model. Specific convolution operators are designed for specific model structures to process the speech input to the model to obtain output consistent with the reasoning effect of the entire sentence. However, the model structure is diverse and complex, and specific convolution operators are difficult to extend and apply to other models, lacking universality.
[0076] In order to solve the above problems, the embodiments of the present disclosure provide a streaming speech reasoning method, device, vehicle computer, storage medium and vehicle. The method first obtains input speech and streaming parameters, and the streaming parameters are used to indicate a preset tensor dimension. Then, the input speech is converted into a first frame tensor, and the frame tensor dimension corresponding to the first frame tensor is determined. Then, a streaming calculation is performed on the second frame tensor corresponding to the preset tensor dimension to obtain a calculation result, wherein the streaming calculation includes at least one of disassembly, shape adjustment and alignment, and the calculation result is used to indicate the context-dependent data of the input speech; further, the calculation result under the preset tensor dimension is restored to the frame tensor dimension to obtain a dimension recovery result for speech reasoning. The dimension recovery result can represent the context-dependent data of the input speech, and is obtained for various streaming calculations, and can be used for further speech reasoning, so that speech reasoning is no longer limited to a specific operator in a certain model, which improves the versatility and scalability of speech reasoning.
[0077] A streaming speech inference recognition method provided in the embodiments of the present disclosure can be implemented by electronic devices, including but not limited to servers, personal computers, laptops, tablet computers, smart phones, vehicle-mounted devices, etc. Electronic devices include user devices and network devices. Among them, user devices include but are not limited to computers, smart phones, tablet computers, etc.; network devices include but are not limited to a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of computers or network servers in cloud computing, wherein cloud computing is a kind of distributed computing, a super virtual computer composed of a group of loosely coupled computer sets. Among them, the electronic device can be operated alone to implement the present disclosure, or it can be connected to the network and implement the present disclosure through interactive operations with other electronic devices in the network. Among them, the network where the electronic device is located includes but is not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (Virtual Private Network, VPN) network, etc.
[0078] It should be noted that the protection scope of the streaming speech inference method described in the embodiment of the present disclosure is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present disclosure are included in the protection scope of the present disclosure.
[0079] like Figure 1 As shown, Figure 1 FIG. 1 is a flow chart of a streaming speech inference method according to an embodiment of the present disclosure. The method can be executed by a streaming speech inference device, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 1 As shown, the method mainly includes the following steps S101 to S104:
[0080] S101: Obtain input voice and streaming parameters.
[0081] The streaming parameter is used to indicate the preset tensor dimension, expressed as stream_dim. Regarding tensors, in deep learning frameworks, data is often stored in the form of tensors. Tensors are an extension of matrices and can be considered as high-order matrices. 0-dimensional tensors represent scalars, 1-dimensional tensors represent vectors, 2-dimensional tensors represent matrices, and 3-dimensional tensors can represent the RGB channels of color images, etc.
[0082] The preset tensor dimension is the dimension of the tensor specified in advance according to the requirements. The preset tensor dimension is also called the streaming dimension. In the present disclosure, the fixed streaming dimension is used as the last dimension of the tensor for subsequent streaming calculations. Exemplarily, the data format of the tensor shown in Table 1 is used to define the frame tensor dimension of the tensor. When the streaming parameter stream_dim=3, it means that the w dimension is specified as the streaming dimension.
[0083] Table 1
[0084] n c h w 0 1 2 3
[0085] It should be noted that in Table 1, n represents the batch number of tensors, c represents the channel number of tensors, h represents the height of tensors, and w represents the width of tensors. The values of streaming parameters are set according to requirements.
[0086] S102: Convert the input speech into a first frame tensor, and determine the frame tensor dimension of the first frame tensor.
[0087] In some embodiments, the embodiments of the present disclosure provide an implementation method for converting input speech into a first frame tensor: after acquiring the input speech, the input speech is preprocessed, and the preprocessing includes framing processing and windowing processing. Specifically, for the input speech, the smallest pronunciation unit (such as phoneme, word, etc.) is used as a unit for recognition, and a sliding window is used to extract short-time audio segments. The sliding window can be a rectangular window, a Henning window, a Hamming window, etc., and the present disclosure does not limit this. The preprocessed input speech is then formatted to obtain a first frame tensor. The input speech is preprocessed, especially windowing is performed after the framing process to reduce the error caused by the framing of the input speech.
[0088] In some embodiments, the input speech is converted into a first frame tensor, and then the data format (data_format) of the first frame tensor is determined, the data format of the first frame tensor is used to define the frame tensor dimension of the first frame tensor, and then the tensor dimension of the first frame tensor is determined according to the data format of the first frame tensor. The frame tensor dimension of the first frame tensor is obtained by reading the data format of the first frame tensor.
[0089] S103: Perform streaming calculation on the second frame tensor corresponding to the preset tensor dimension to obtain a calculation result.
[0090] The streaming calculation includes at least one of disassembly, shape adjustment and alignment, and the calculation result is used to indicate context-dependent data of the input speech.
[0091] Different calculation results are obtained for different methods of pipeline calculation. If the stream calculation is disassembly, the calculation result is the disassembly result; if the stream calculation is shape adjustment, the calculation result is the adjustment result; if the stream calculation is alignment, the calculation result is the alignment result. It can be understood that the calculation result includes at least one of the following: the disassembly result obtained by disassembling the second frame tensor corresponding to the preset frame dimension; the adjustment result obtained by shape adjustment of the second frame tensor corresponding to the preset frame tensor; the alignment result obtained by aligning the second frame tensor corresponding to the preset frame tensor.
[0092] The following will introduce streaming computing in detail from three aspects: disassembly, shape adjustment and alignment:
[0093] (1) Disassembly
[0094] In some embodiments, in the process of disassembling the second frame tensor corresponding to the preset tensor dimension, the context-dependent parameters are first determined, and the context-dependent parameters include an operator kernel, a step size, and a hole, wherein the operator kernel can be a convolution kernel, etc., and the present disclosure is not limited to this. For example, the context-dependent parameters are conv1d k=1*3,s=1,d=1; then, according to the length of the operator kernel, the second frame tensor is split to obtain multiple first frame fragments; then, for each first frame fragment in the multiple first frame fragments, it is spliced with adjacent frames in adjacent frame fragments to obtain a spliced frame fragment. It can be understood that the start frame and / or the end frame of the adjacent frame fragments of the first frame fragment are connected to the first frame fragment to obtain a spliced frame fragment; further, the spliced frame fragment and the operator kernel are used for calculation to obtain a disassembly result, and the disassembly result can represent the context-dependent data of the input speech.
[0095] Exemplarily, the context-dependent parameter is a convolution operator, the operator kernel is a 1*3 convolution kernel, the step size is 1, and the hole is 1 (conv1d k=1*3, s=1, d=1). Figure 2A As shown, Figure 2AA schematic diagram of disassembly provided in an embodiment of the present disclosure shows a second frame tensor 200 and an operator core 210 corresponding to a preset frame tensor. First, the second frame tensor 200 is split according to the length 3 of the operator core 210 to obtain three first frame fragments 201, namely, a first frame fragment 201a, a first frame fragment 201b, and a first frame fragment 201c, each of which contains 3 frames. Then, for the first frame segment 201a, which is adjacent to the first frame segment 201b, the start frame of the first frame segment 201b is spliced to the end frame of the first frame segment 201a to obtain a spliced frame segment 202a; for the first frame segment 201b, which is adjacent to the first frame segment 201a and the first frame segment 201c, the end frame of the first frame segment 201a is spliced to before the start frame of the first frame segment 201b, and the start frame of the first frame segment 201c is spliced to after the end frame of the first frame segment 201b to obtain a spliced frame segment 202b; for the first frame segment 201c, which is adjacent to the first frame segment 201b, the end frame of the first frame segment 201b is spliced to before the start frame of the first frame segment 201c to obtain a spliced frame segment 202c;. Further, the operator core 210 is used to perform calculations with the spliced frame fragment 202a, the spliced frame fragment 202b, and the spliced frame fragment 202c, respectively, to obtain the disassembly results 203, including the disassembly results 203a, the disassembly results 203b, and the disassembly results 203c.
[0096] It should be noted that the decomposition results obtained by disassembling and splicing are consistent with the reasoning effect of the whole sentence. Figure 2B As shown, Figure 2B 2 is a schematic diagram of whole sentence reasoning. In the figure, the second frame tensor 200 corresponding to the preset frame tensor is processed as a whole, and the whole sentence reasoning result 204 is obtained by calculating the second frame tensor 200 and the operator core 210. Figure 2A The disassembly result 203 is shown after splicing, and Figure 2B The whole sentence reasoning result 204 shown is consistent.
[0097] The above-mentioned embodiment calculates the accurate front-and-back dependencies at the frame level through context dependency parameters, splices the first frame fragments after splitting at the frame level, and then calculates with the operator core. Compared with the method of splicing according to the operator kernel size, the amount of repeated calculations is reduced, the complexity of the calculation is reduced, thereby improving the calculation efficiency and reducing the delay.
[0098] (2) Shape adjustment
[0099] In some embodiments, during the process of performing shape adjustment on the second frame tensor corresponding to the preset tensor dimension, a shape-dependent parameter is first determined, the shape-dependent parameter indicates an adjustment method and a target frame tensor to be adjusted, and the adjustment method includes but is not limited to padding and slicing. Then, the target frame tensor in the second frame tensor is adjusted in an adjustment manner according to the shape-dependent parameter to obtain an adjustment result.
[0100] In some embodiments, the adjustment method is slicing, and the target frame tensor is the frame tensor corresponding to the shape-dependent interval, which is used to indicate the number of bits corresponding to the starting frame tensor and the number of bits corresponding to the ending frame tensor. For example, the shape-dependent interval [-5, -2) indicates that the number of bits corresponding to the starting frame tensor is 5, and the number of bits corresponding to the ending frame tensor is 2. It should be noted that the negative sign in the shape-dependent interval indicates the data reading direction, and -5 indicates that the starting frame tensor is counted from the last frame tensor, which can be understood as the fifth-to-last frame tensor. Similarly, -2 indicates the second-to-last frame tensor. The shape-dependent index is slice[-5, -2).
[0101] In the process of adjusting the target frame tensor in the second frame tensor in an adjustment manner according to the shape-dependent parameter, first, according to the number of bits corresponding to the termination frame tensor, the second frame tensor corresponding to the preset tensor dimension is split to obtain multiple second frame fragments, and the number of frame tensors of each second frame fragment is the same as the number of bits corresponding to the termination frame tensor. It can be understood that the second tensor dimension is divided into multiple second frame fragments with the same number of frame tensors, and this same number of frame tensors is equal to the number of bits corresponding to the termination frame tensor. Then, according to the number of bits corresponding to the starting frame tensor, the multiple second frame fragments are traversed until the last second frame fragment is traversed, and the target frame fragments with the same number of frame tensors as the starting frame tensor are cached. It can be understood that the number of bits corresponding to the starting frame tensor is used as the traversal unit, starting from the first second frame fragment, and read in sequence, and the read content is cached to determine whether it is in the shape-dependent interval, until the last second frame fragment is read. At this time, the cached target frame fragment is the last second frame fragment and the part of the frame fragment before it, and the number of frame tensors of the frame fragments in the cache is equal to the number of bits corresponding to the starting frame tensor indicated by the shape-dependent interval. Further, according to the number of bits corresponding to the termination frame tensor, a target frame tensor corresponding to the shape-dependent interval is extracted from the target frame segment as an adjustment result.
[0102] For example, the shape-dependent parameter is slice[-5, -2), which means slicing the second frame tensor corresponding to the preset tensor dimension to slice the fifth to the second to last frame of the second frame tensor. Figure 3A As shown, Figure 3AA schematic diagram of shape adjustment is provided for an embodiment of the present disclosure, wherein a second frame tensor 200 corresponding to a preset frame tensor is shown in the figure. First, the second frame tensor 200 is split according to the bit number 2 corresponding to the termination frame tensor to obtain five second frame fragments 30 with a frame number of 2, namely, a second frame fragment 301, a second frame fragment 302, a second frame fragment 303, a second frame fragment 304, and a second frame fragment 305. The traversal starts from the first second frame fragment 301 with a frame tensor number of 5. The first traversal obtains the target frame fragment 310 (including the second frame fragment 301 and the second frame fragment 302). It should be noted that the second frame fragment is processed as a whole before a second frame fragment is read. Therefore, the target frame fragment includes the complete second frame fragment 301 and the second frame fragment 302, but not the second frame fragment 301, the second frame fragment 302 and a part of the second frame fragment 303. During the traversal process, taking the second traversal as an example, the number of frame tensors is 5, and the target frame fragment 320 read includes: the termination frame of the second frame fragment 301, the second frame fragment 302, and the second frame fragment 303. Until the last second frame fragment is read, the target frame fragment 340 is obtained, which includes the termination frame of the second frame fragment 303, the second frame fragment 304, and the second frame fragment 305. Further, according to the number of bits 2 corresponding to the termination frame tensor, the penultimate frame in the target frame fragment 350 is found, and the two frame tensors after the penultimate frame are discarded to obtain the target frame tensor 400, which is the frame tensor corresponding to the penultimate frame of the second frame tensor from the fifth to the penultimate frame. It can be understood that starting from the first frame tensor of the second frame tensor 200 is received, it is split according to the number of bits corresponding to the termination frame tensor, and traversed according to the number of bits corresponding to the starting frame tensor. When the last frame tensor of the second frame tensor 200 is read, the shape adjustment result after slicing is obtained.
[0103] like Figure 3B As shown, Figure 3B This is a schematic diagram of whole sentence segmentation. The second frame tensor 200 corresponding to the preset frame tensor is taken as a whole sentence. The segmentation is started only after the last frame tensor of the second frame tensor 200 is received. The count is read from the last frame tensor of the entire second frame tensor, which results in a large delay.
[0104] Compared with the related calculations in whole sentence reasoning, the above shape adjustment does not need to wait until the complete sentence is received before starting the calculation. The start time of the shape adjustment is advanced, so that the shape adjustment result is obtained after the data reading is completed, which improves the calculation efficiency of the shape adjustment.
[0105] (3) Alignment
[0106] In some embodiments, the second frame tensor corresponding to the preset tensor dimension is a frame tensor from a different path. During the speech inference process, attention needs to be paid to the differences between the frame tensors from different paths, such as feature fusion operations such as addition and concatenation. When aligning the second frame tensor corresponding to the preset tensor dimension, multiple frame tensors from different paths are aligned until the last frame tensor of the multiple frame tensors from different paths is traversed to obtain the alignment result. It can be understood that frame tensors from different paths will be obtained during calculation, and the frame tensors of each path may be uneven. In the cache area, the frame tensors from different paths are aligned to obtain the alignment result.
[0107] S104. Perform dimension restoration on the calculation result according to the frame tensor dimension to obtain a dimension restoration result, and the dimension restoration result is used for speech inference.
[0108] The calculation result is restored according to the frame tensor dimension to obtain the dimension restoration result, which is to restore the calculation result under the preset tensor dimension to the frame tensor dimension, which can be understood as dimensionality increase. Exemplarily, the calculation result under the w dimension is restored to nchw, so as to use the dimension restoration result for subsequent speech reasoning, for example, after obtaining the disassembly result after the dimension restoration, it is aligned with other frame tensors.
[0109] In summary, the embodiments of the present disclosure provide a streaming speech reasoning method, which first obtains input speech and streaming parameters, where the streaming parameters are used to indicate a preset tensor dimension, then converts the input speech into a first frame tensor, and determines the frame tensor dimension corresponding to the first frame tensor, and then performs streaming calculations on the second frame tensor corresponding to the preset tensor dimension to obtain a calculation result, wherein the streaming calculation includes at least one of disassembly, shape adjustment, and alignment, and the calculation result is used to indicate the context-dependent data of the input speech; further, the calculation result under the preset tensor dimension is restored to the frame tensor dimension to obtain a dimension recovery result for speech reasoning. The dimension recovery result can represent the context-dependent data of the input speech, and is obtained for various streaming calculations, and can be used for further speech reasoning, so that speech reasoning is no longer limited to a specific operator in a certain model, thereby improving the versatility and scalability of speech reasoning.
[0110] like Figure 4 As shown, Figure 4 A structural diagram of a streaming speech inference device provided in an embodiment of the present disclosure, the streaming speech inference device comprising:
[0111] An acquisition module 401 is used to acquire input speech and streaming parameters, where the streaming parameters are used to indicate a preset tensor dimension;
[0112] A conversion module 402, configured to convert the input speech into a first frame tensor and determine a frame tensor dimension of the first frame tensor;
[0113] A calculation module 403 is used to perform streaming calculation on the second frame tensor corresponding to the preset tensor dimension to obtain a calculation result, wherein the streaming calculation includes at least one of disassembly, shape adjustment and alignment, and the calculation result is used to indicate context dependency data of the input speech;
[0114] The dimension recovery module 404 is used to perform dimension recovery on the calculation result according to the frame tensor dimension to obtain a dimension recovery result, and the dimension recovery result is used for speech reasoning.
[0115] As an optional implementation of the embodiment of the present disclosure, the calculation result is a decomposition result obtained by decomposing the second frame tensor corresponding to the preset tensor dimension;
[0116] The calculation module 403 is specifically used to determine context-dependent parameters, which include operator kernel, step size and hole;
[0117] According to the length of the operator kernel, the second frame tensor is split to obtain multiple first frame fragments;
[0118] For each frame segment in the plurality of first frame segments, splicing adjacent frames in adjacent frame segments to obtain a spliced frame segment;
[0119] The spliced frame fragments and operator kernels are used for calculation to obtain the disassembly results.
[0120] As an optional implementation of the embodiment of the present disclosure, the calculation result is an adjustment result obtained by adjusting the shape of the second frame tensor corresponding to the preset tensor dimension;
[0121] A calculation module 403 is specifically used to determine a shape-dependent parameter, where the shape-dependent parameter indicates an adjustment method and a target frame tensor to be adjusted;
[0122] According to the shape-dependent parameter, the target frame tensor in the second frame tensor is adjusted corresponding to the adjustment method to obtain an adjustment result.
[0123] As an optional implementation of the embodiment of the present disclosure, the adjustment method is slicing, the target frame tensor is the frame tensor corresponding to the shape-dependent interval, and the shape-dependent interval is used to indicate the number of bits corresponding to the start frame tensor and the number of bits corresponding to the end frame tensor;
[0124] The calculation module 403 adjusts the target frame tensor in the second frame tensor according to the shape-dependent parameter in a corresponding adjustment manner to obtain an adjustment result, specifically for:
[0125] According to the number of bits corresponding to the termination frame tensor, split the second frame tensor to obtain a plurality of second frame segments, wherein the number of frame tensors of each second frame segment in the plurality of second frame segments is the same as the number of bits corresponding to the termination frame tensor;
[0126] According to the number of bits corresponding to the starting frame tensor, multiple second frame fragments are traversed until the last second frame fragment is traversed, and the target frame fragment is cached. The number of frame tensors of the target frame fragment is the same as the number of bits corresponding to the starting frame tensor;
[0127] According to the number of bits corresponding to the termination frame tensor, a target frame tensor corresponding to the shape-dependent interval is extracted from the target frame fragment as an adjustment result.
[0128] As an optional implementation of the embodiment of the present disclosure, the calculation result is an alignment result obtained by aligning a second frame tensor corresponding to a preset tensor dimension, and the second frame tensor is a plurality of frame tensors from different paths;
[0129] The calculation module 403 is specifically used to perform frame alignment on multiple frame tensors from different paths until traversing to the last frame tensor of the multiple frame tensors from different paths to obtain an alignment result.
[0130] As an optional implementation of the embodiment of the present disclosure, the conversion module 402 is specifically used to convert the input speech into a first frame tensor, determine the data format of the first frame tensor, and the data format is used to define the frame tensor dimension of the first frame tensor;
[0131] Determines the frame tensor dimensions as defined by the data format.
[0132] As an optional implementation of the embodiment of the present disclosure, the conversion module 402 is specifically used to convert the input speech into a first frame tensor, including:
[0133] Preprocessing the input speech, including framing and windowing;
[0134] The preprocessed input speech is formatted to obtain the first frame tensor.
[0135] In summary, the disclosed embodiment provides a streaming speech reasoning device, which first obtains input speech and streaming parameters by an acquisition module, and the streaming parameters are used to indicate a preset tensor dimension, and then the input module converts the input speech into a first frame tensor, and determines the frame tensor dimension corresponding to the first frame tensor, and then the calculation module performs streaming calculation on the second frame tensor corresponding to the preset tensor dimension to obtain a calculation result, wherein the streaming calculation includes at least one of disassembly, shape adjustment and alignment, and the calculation result is used to indicate the context-dependent data of the input speech; further, the dimension recovery module restores the calculation result under the preset tensor dimension to the frame tensor dimension to obtain a dimension recovery result for speech reasoning. The dimension recovery result can represent the context-dependent data of the input speech, and is obtained for various streaming calculations, and can be used for further speech reasoning, so that speech reasoning is no longer limited to a specific operator in a certain model, and improves the versatility and scalability of speech reasoning.
[0136] like Figure 5 As shown, the embodiment of the present disclosure provides a vehicle computer, the electronic device includes: a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor, and the computer program is executed by the processor 501 to implement each process of the streaming speech reasoning method in the above method embodiment. And can achieve the same technical effect, to avoid repetition, it will not be repeated here.
[0137] The embodiments of the present disclosure provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, each process of the streaming speech inference method in the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0138] The computer readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0139] The embodiments of the present disclosure provide a computer program product, which stores a computer program. When the computer program is executed by a processor, each process of the streaming speech inference method in the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0140] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.
[0141] In several embodiments provided by the present disclosure, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0142] In the present disclosure, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0143] In the present disclosure, memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
[0144] In the present disclosure, computer-readable media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.
[0145] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0146] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A streaming speech inference method, characterized in that: include: Obtaining input speech and streaming parameters, where the streaming parameters are used to indicate a preset tensor dimension; Convert the input speech into a first frame tensor, and determine a frame tensor dimension of the first frame tensor; Performing streaming calculation on the second frame tensor corresponding to the preset tensor dimension to obtain a calculation result, wherein the streaming calculation includes at least one of disassembly, shape adjustment and alignment, and the calculation result is used to indicate context-dependent data of the input speech; The calculation result is dimensionally restored according to the frame tensor dimension to obtain a dimensionally restored result, and the dimensionally restored result is used for speech inference.
2. The method according to claim 1, characterized in that: The calculation result is a decomposition result obtained by decomposing the second frame tensor corresponding to the preset tensor dimension, include: Determining context-dependent parameters, wherein the context-dependent parameters include an operator kernel, a step size, and a hole; Splitting the second frame tensor according to the length of the operator kernel to obtain a plurality of first frame segments; For each frame segment in the plurality of first frame segments, splicing adjacent frames in adjacent frame segments to obtain a spliced frame segment; The spliced frame segments and the operator core are used to perform calculations to obtain the disassembly result.
3. The method according to claim 1, characterized in that The calculation result is an adjustment result obtained by adjusting the shape of the second frame tensor corresponding to the preset tensor dimension, including: Determining shape-dependent parameters, wherein the shape-dependent parameters indicate an adjustment method and a target frame tensor to be adjusted; According to the shape-dependent parameter, the target frame tensor in the second frame tensor is adjusted corresponding to the adjustment method to obtain the adjustment result.
4. The method according to claim 3, characterized in that The adjustment method is slicing, the target frame tensor is a frame tensor corresponding to a shape-dependent interval, and the shape-dependent interval is used to indicate the number of bits corresponding to the start frame tensor and the number of bits corresponding to the end frame tensor; The adjusting the target frame tensor in the second frame tensor according to the shape-dependent parameter corresponding to the adjustment method to obtain the adjustment result includes: According to the number of bits corresponding to the termination frame tensor, split the second frame tensor to obtain a plurality of second frame segments, wherein the number of frame tensors of each second frame segment in the plurality of second frame segments is the same as the number of bits corresponding to the termination frame tensor; According to the number of bits corresponding to the starting frame tensor, traverse the plurality of second frame segments until the last second frame segment is traversed, and cache the target frame segment, wherein the number of frame tensors of the target frame segment is the same as the number of bits corresponding to the starting frame tensor; According to the number of bits corresponding to the termination frame tensor, a target frame tensor corresponding to the shape-dependent interval is extracted from the target frame segment as the adjustment result.
5. The method according to claim 1, characterized in that The calculation result is an alignment result obtained by aligning a second frame tensor corresponding to the preset tensor dimension, where the second frame tensor is a plurality of frame tensors from different paths, including: Frame alignment is performed on the multiple frame tensors from different paths until the last frame tensor of the multiple frame tensors from different paths is traversed to obtain the alignment result.
6. The method according to claim 1, characterized in that The step of converting the input speech into a first frame tensor and determining a frame tensor dimension of the first frame tensor includes: Convert the input speech into a first frame tensor, and determine a data format of the first frame tensor, where the data format is used to define a frame tensor dimension of the first frame tensor; The frame tensor dimensions defined by the data format are determined.
7. The method according to claim 1, characterized in that The step of converting the input speech into a first frame tensor comprises: Preprocessing the input speech, wherein the preprocessing includes framing processing and windowing processing; The preprocessed input speech is format converted to obtain the first frame tensor.
8. A streaming speech inference device, characterized in that: include: An acquisition module, used to acquire input speech and streaming parameters, where the streaming parameters are used to indicate a preset tensor dimension; A conversion module, configured to convert the input speech into a first frame tensor and determine a frame tensor dimension of the first frame tensor; a computing module, configured to perform streaming computing on the second frame tensor corresponding to the preset tensor dimension to obtain a computing result, wherein the streaming computing includes at least one of disassembly, shape adjustment, and alignment, and the computing result is used to indicate context-dependent data of the input speech; A dimension recovery module is used to perform dimension recovery on the calculation result according to the frame tensor dimension to obtain a dimension recovery result, and the dimension recovery result is used for speech reasoning.
9. A vehicle computer, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the streaming speech inference method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: include: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the streaming speech inference method according to any one of claims 1 to 7 is implemented.
11. A vehicle, characterized in that: include: A streaming speech inference device as claimed in claim 8, or a vehicle computer as claimed in claim 9.
