Code generation method, code display method, device, equipment and storage medium

Abstract

The present application discloses a code generation method, a code display method, an apparatus, a device and a storage medium, and belongs to the field of artificial intelligence technology. The method comprises: obtaining target voice data; dividing the target voice data into frames to obtain multiple audio frames; determining the instruction elements corresponding to each audio frame to obtain multiple instruction elements; wherein the instruction elements are used to characterize the instruction information carried in the audio frame; based on the multiple instruction elements, generating reference program code corresponding to the voice data. Through the above method, the object only needs to input voice data to obtain the corresponding program code, which simplifies the operation of obtaining the program code and helps to increase the speed at which the object obtains the program code. At the same time, the program code can be obtained through voice control, which reduces the requirements for the capabilities of the relevant objects and expands the scope of the objects used.

Description

Technical Field

[0001] The present application relates to the field of computer and Internet technology, and in particular to a code generation method, a code display method, an apparatus, a device and a storage medium. Background Art

[0002] In order to lower the threshold for obtaining program code, it is necessary to simplify the way objects obtain program code.

[0003] In the related art, the text information input by the object is obtained, and the standard text is obtained by comparing the input text information with the standard text database. Then, the reference program code corresponding to the standard text is determined and the reference program code is returned to the object.

[0004] However, in the related art, the operation of inputting text into an object is cumbersome, which makes the efficiency of obtaining reference program codes by the object low. Summary of the invention

[0005] The present application provides a code generation method, a code display method, an apparatus, a device and a storage medium. The technical solution is as follows:

[0006] According to one aspect of an embodiment of the present application, a code generation method is provided, the method comprising:

[0007] Obtain target voice data;

[0008] Divide the target voice data into frames to obtain multiple audio frames;

[0009] Determine the instruction elements corresponding to each of the audio frames, and obtain a plurality of the instruction elements; wherein the instruction elements are used to represent the instruction information carried in the audio frames;

[0010] Based on the multiple instruction elements, a reference program code corresponding to the target voice data is generated.

[0011] According to one aspect of an embodiment of the present application, a code display method is provided, the method comprising:

[0012] Obtain target voice data;

[0013] Displaying a reference program code corresponding to the target voice data in a code display interface, wherein the reference program code is generated based on a plurality of instruction elements converted from the target voice data, wherein the instruction elements are used to represent instruction information carried in an audio frame of the target voice data;

[0014] In response to an operation on the reference program code, a first interactive content is displayed, where the first interactive content is used to represent the running process of the reference program code.

[0015] According to one aspect of an embodiment of the present application, a code generation device is provided, the device comprising:

[0016] A voice acquisition module, used to acquire target voice data;

[0017] A voice framing module, used for framing the target voice data to obtain multiple audio frames;

[0018] A single instruction module, used to determine the instruction elements corresponding to each of the audio frames, and obtain a plurality of the instruction elements; wherein the instruction elements are used to represent the instruction information carried in the audio frames;

[0019] A code generation module is used to generate reference program codes corresponding to the target voice data based on the multiple instruction elements.

[0020] According to one aspect of an embodiment of the present application, a code display device is provided, the device comprising:

[0021] A voice acquisition module, used to acquire target voice data;

[0022] A code display module, used to display a reference program code corresponding to the target voice data in a code display interface, wherein the reference program code is generated based on a plurality of instruction elements converted from the target voice data, wherein the instruction elements are used to represent instruction information carried in an audio frame of the target voice data;

[0023] The content display module is used to display first interactive content in response to an operation on the reference program code, wherein the first interactive content is used to represent the operation process of the reference program code.

[0024] According to one aspect of an embodiment of the present application, a computer device is provided, wherein the terminal device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the above-mentioned code generation method, or the above-mentioned code display method.

[0025] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned code generation method, or the above-mentioned code display method.

[0026] According to one aspect of an embodiment of the present application, a computer program product or a computer program is provided, wherein the computer program product or the computer program includes computer instructions, wherein the computer instructions are stored in a computer-readable storage medium, and a processor reads and executes the computer instructions from the computer-readable storage medium to implement the above-mentioned code generation method, or the above-mentioned code display method.

[0027] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0028] By acquiring voice data and converting the voice data into reference program codes, the user does not need to input text, which reduces the difficulty of the object to obtain the reference program code, simplifies the steps of the object to obtain the reference program code, and improves the efficiency of the object to obtain the reference program code. At the same time, since the object only needs to input voice to obtain the reference program code, the requirements for the object's ability are reduced, which helps to expand the scope of use of the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of a solution implementation environment provided by an exemplary embodiment of the present application;

[0030] Figure 2 is a flowchart of a code generation method provided by an exemplary embodiment of the present application;

[0031] Figure 3 is an audio graph of target speech data provided by an exemplary embodiment of the present application;

[0032] Figure 4 is a schematic diagram of framing voice data provided by an exemplary embodiment of the present application;

[0033] Figure 5 is a schematic diagram of a vector extraction model provided by an exemplary embodiment of the present application;

[0034] Figure 6 It is a schematic diagram of the corresponding relationship between instruction element combination and instruction provided by an exemplary embodiment of the present application;

[0035] Figure 7 is a schematic diagram of determining a combination of instruction elements through a second order provided by an exemplary embodiment of the present application;

[0036] Figure 8 is a schematic diagram of a frame vector extraction process provided by an exemplary embodiment of the present application;

[0037] Fig. 9 is a schematic diagram of a process of converting speech into code provided by an exemplary embodiment of the present application;

[0038] Fig.10 is a flowchart of a code display method provided by an exemplary embodiment of the present application;

[0039] Fig.11 is a schematic diagram of a code display interface provided by an exemplary embodiment of the present application;

[0040] Fig.12 is a block diagram of a code generating device provided by an exemplary embodiment of the present application;

[0041] Fig.13 is a block diagram of a code display device provided by an exemplary embodiment of the present application;

[0042] Fig.14 It is a structural block diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0044] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0045] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes natural language processing technology, machine learning / deep learning and other major directions.

[0046] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0047] Machine Learning (ML) is a multi-disciplinary cross-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory and other disciplines. It specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching. In this application, the frame vector corresponding to the audio frame of the voice data can be extracted by using the relevant techniques of machine learning, so that in the subsequent process, the reference program code corresponding to the voice data can be determined based on the frame vector to complete the process of converting the voice into code.

[0048] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common voice conversion program codes, children's education assistance, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0049] Short Time Fourier Transform (STFT) is a method used to process non-stationary signals (signals whose frequency components change over time) and convert time domain signals into frequency domain signals. By windowing the time domain signal, a relatively stable local time domain signal is obtained, and the local time domain signal is analyzed to obtain the frequency information corresponding to the local time domain signal.

[0050] Wavelet Transform (WT) is a method that uses basis functions with continuously changing frequencies to convert time domain signals into frequency domain signals.

[0051] Hilbert Transform (HT) is an audio conversion method. This method filters the input audio data (such as through a filter, or a system for filtering), obtains the impulse response h(t) of the audio data, and performs Fourier transform on h(t). By analyzing the real signal through Hilbert transform, the one-dimensional signal can be converted into a two-dimensional complex plane signal. The modulus and phase angle of the complex number represent the amplitude and phase of the signal respectively.

[0052] Figure 1 1 is a schematic diagram of a solution implementation environment provided by an exemplary embodiment of the present application. The solution implementation environment can be implemented as a computer system, such as a voice conversion system. The solution implementation environment can include: a terminal device 10 and a server 20.

[0053] The terminal device 10 may be an electronic device such as a PC (Personal Computer), a tablet computer, a mobile phone, a multimedia player, a wearable device, a smart home appliance, a vehicle-mounted terminal, etc. A client of a target application may be running in the terminal device 10, and the target application may be a program code learning application, or other application that converts speech into program code. The terminal device 10 can respond to a speech acquisition operation initiated by the subject, acquire target speech data, and convert the target speech data into reference program code.

[0054] The server 20 can provide background services for the target application running on the terminal device 10. For example, the server 20 can be the background server of the target application. The server 20 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud computing, cloud functions, cloud storage, network services, cloud communications, domain name services, security services, and big data and artificial intelligence platforms. The server 20 has at least the functions of data transmission and reception, calculation, storage, etc., and is used to receive application data uploaded by the client on the terminal device 10 in real time, such as voice data. The server 20 performs calculations based on the received application data, obtains calculation results, and stores or sends the calculation results to the terminal device 10.

[0055] The code generation method provided in the present application can be executed by a terminal device, or by a server, or by the terminal and the server interacting and cooperating to execute the code generation method.

[0056] Figure 2 is a flowchart of a code generation method provided by an exemplary embodiment of the present application. For example, the execution subject of the method may be Figure 1 The terminal device 10 in the example, the execution subject may be a client of the target application running on the terminal device 10, or may be Figure 1 For the convenience of description, the following describes the code generation method using a computer device as the execution subject. Figure 2 As shown, the method may include the following steps (210-240):

[0057] Step 210, obtaining target voice data.

[0058] The target voice data refers to the voice data used to convert and generate the reference program code. In some embodiments, the target voice data is valid voice data, that is, by converting the target voice data, the corresponding reference program code can be obtained. In other embodiments, the target voice data is invalid voice data, that is, the target voice data cannot be converted (for example, the voice data does not contain any identifiable information), or after the target voice data is converted, the reference program code cannot be obtained (for example, the reference program code matching the target voice data cannot be found).

[0059] The computer device obtains voice data for determining the reference program code. In some embodiments, when the execution subject of the code generation method is a terminal device, the terminal device can directly obtain voice data by recording the voice input by the object. In other embodiments, the computer device receives voice data sent by other devices, or obtains voice data from a storage space.

[0060] Step 220, dividing the target voice data into frames to obtain multiple audio frames.

[0061] Framing refers to the process of slicing the target voice data to obtain several audio frames. Optionally, an audio frame is called a voice data segment. In some embodiments, a audio frames are obtained by framing, wherein the duration of each audio frame is equal. The duration of the audio frame is determined by the frame length. For example, the value of the frame length can be: 20ms, 25ms, 30ms, 32ms, 40ms, 50ms, etc. The specific value of the frame length is determined according to the specific situation of the voice data and the actual situation such as the computing power of the computer equipment, and the present application does not limit it here. There is a situation where the duration of a certain target voice data is less than or equal to the frame length, that is, after framing the target voice data, only one audio frame can be obtained, and the audio frame is the target voice data. Please refer to Figure 3 , which shows a schematic diagram of a waveform of target speech data. In some embodiments, the horizontal axis of the waveform is time, and the vertical axis is amplitude or frequency. The waveform reflects the energy change of the target speech data over time; the waveform includes multiple sound wave information (such as a pronunciation) 310.

[0062] Normally, framing starts from the starting position of the target audio data. For example, the duration of a certain target audio data is 200ms, and the frame length is 50ms. By framing from the starting position of the audio data according to the frame length, 4 audio frames can be obtained, and the time ranges corresponding to the 4 audio frames are: (0-50ms), (50-100ms), (100-150ms) and (150-200ms). In some embodiments, after a certain target audio data is framed, the duration of the obtained audio frames is not exactly the same (such as the duration of the last audio frame is not equal to the duration of the remaining audio frames). For example, the duration of a target audio data is 210ms, and the frame length is 40ms. By dividing the audio data into frames from the starting position according to the frame length, 6 audio frames can be obtained. The time ranges corresponding to the 6 audio frames are: (0-40ms), (40-80ms), (80-120ms), (120-160ms), (160-180ms) and (180-210ms).

[0063] In some embodiments, after obtaining the audio frame, the computer device performs windowing on the audio frame. Windowing refers to multiplying the audio frame (signal) with a window function (such as a Hamming window) so that the amplitudes on both sides of the audio frame gradually fade to 0 to reduce spectrum leakage. Windowing weakens the signal at the edge of the audio frame, causing data loss at the edge of the audio frame. In order to avoid missing audio information, in the process of framing the voice data, it is necessary to control the stride of the framing by frame shifting.

[0064] Frame shift refers to the difference between the start times of two audio frames. Optionally, the two audio frames are adjacent audio frames, i.e., there is no other audio frame between the two audio frames. As described above, in order to avoid missing feature information in the audio data, the value of the frame shift is less than the framing duration, that is, there is an overlapping portion between two adjacent audio frames. In some embodiments, if there is no need to perform a windowing operation on the audio frame, the value of the frame shift can be equal to the framing duration, which is equivalent to the end time of the previous audio frame being the start time of the next audio frame.

[0065] like Figure 4 , which shows a schematic diagram of framing the target speech data. The framing duration is 50ms (ie, the duration of each speech segment 410 is 50ms), the frame shift is 30ms (50 minus 20), and the overlapping time of two adjacent speech segments is 20ms.

[0066] Step 230, determining the instruction elements corresponding to the respective audio frames, and obtaining a plurality of instruction elements; wherein the instruction elements are used to represent the instruction information carried in the audio frames.

[0067] The computer device processes each audio frame separately, and obtains the instruction element corresponding to the audio frame by extraction or conversion. In some embodiments, since the target voice data can be converted into a reference program code, the target voice data contains some information that can map the reference program code. The computer device determines the instruction element corresponding to the audio frame belonging to the target voice data, which is equivalent to extracting or converting information related to the reference program code from the target voice data. In some embodiments, a single instruction element does not have practical significance. By combining multiple instruction elements, information with practical significance (i.e., instruction commands) can be obtained. Optionally, the information has a task indication function. In other embodiments, a certain instruction element corresponds to information with a task indication function. For the specific implementation method of this process, please refer to the following embodiments.

[0068] By extracting the instruction elements corresponding to the audio frame, the voice data can be converted into program code without going through the intermediate process of converting the voice into text, which helps to speed up the conversion of voice data into program code, improve the efficiency of converting voice data into program code, and improve the response speed of terminal devices.

[0069] Step 240: Generate reference program code corresponding to the target voice data based on the multiple instruction elements.

[0070] Reference program code refers to information written according to the rules of a certain computer language and capable of realizing functions through compilation. In some embodiments, after obtaining multiple instruction elements, the computer device determines the reference program code corresponding to the target voice data through the obtained instruction elements. In some embodiments, multiple instruction elements may correspond to different reference program codes. For example, multiple instruction elements correspond to different reference program codes under different computer language frameworks, and these reference program codes can realize the same or similar functions. For another example, under the framework of a certain computer voice, multiple instruction elements can correspond to at least two reference program codes. In this case, it is necessary to select a reference program code from multiple reference program codes through some selection methods, or give up determining the reference program code from multiple instruction elements (the process of converting voice into code fails this time). In some embodiments, the computer device selects b instruction elements from the above-mentioned multiple instruction elements to determine the functions corresponding to b instruction elements, where b is a positive integer (the function is preset, that is, b instruction elements correspond to an instruction command that realizes the function). The reference program code corresponding to the instruction command can be determined by the preset mapping information. For the process of generating the corresponding reference program code based on the instruction element, please refer to the following embodiment.

[0071] In summary, by acquiring voice data and converting the voice data into reference program codes, the user does not need to input text, which reduces the difficulty of the object to obtain the reference program code, simplifies the steps of the object to obtain the reference program code, and improves the efficiency of the object to obtain the reference program code. At the same time, since the object only needs to input voice to obtain the reference program code, the requirements for the object's ability are reduced, which helps to expand the scope of use of the object.

[0072] The process of determining the instruction element corresponding to the audio frame is introduced and explained through several embodiments below.

[0073] In some embodiments, the instruction elements corresponding to each audio frame are determined to obtain multiple instruction elements, including: for a target audio frame among the multiple audio frames, performing time-frequency conversion on the target audio frame to obtain a spectrum matrix corresponding to the target audio frame, wherein the spectrum matrix is ​​used to characterize the spectrum information of the target audio frame; based on the spectrum matrix, determining a frame vector corresponding to the target audio frame, the frame vector is a vector representation of the target audio frame; based on the frame vector, determining the instruction element corresponding to the target audio frame; based on the instruction elements corresponding to each audio frame, obtaining multiple instruction elements.

[0074] The audio frames obtained after framing cannot be used directly for calculations. The audio frames need to be processed first to extract the information carried therein. In some embodiments, the computer device processes the audio frames by short-time Fourier transform, wavelet transform, and other methods to obtain a spectrum matrix corresponding to the audio frames. For example, after performing a short-time Fourier transform on the audio frames, a frequency-amplitude diagram corresponding to the audio frames is obtained. The horizontal axis of the spectrum diagram is time, and the vertical axis is frequency. The information carried in the spectrum diagram can be represented by the spectrum matrix.

[0075] In some embodiments, the dimension of the spectrum matrix is ​​related to the frame length and the amount of data carried in the audio frame. For example, if the frame length is 50ms, that is, the length of the audio frame obtained by framing is 50ms, the dimension of the spectrum matrix corresponding to the audio frame is 20*50, that is, the number of columns of the spectrum matrix is ​​related to the frame length, and the number of rows of the multidimensional vector is related to the amount of data in the audio frame.

[0076] In some embodiments, a computer device determines a frame vector corresponding to a target audio frame based on a spectrum matrix, including: inputting the spectrum matrix into a vector extraction model, where the vector extraction model is a model for extracting vectors from input information; and processing the spectrum matrix through the vector extraction model to obtain a frame vector corresponding to the target audio frame.

[0077] In some embodiments, the vector extraction model is a neural network model, such as a convolutional neural network, a deep neural network, a recurrent neural network, a long short-term memory neural network, etc.

[0078] The following takes the vector extraction model as a convolutional neural network model as an example to introduce the process of a computer device obtaining a frame vector corresponding to an audio frame.

[0079] In some embodiments, the computer device inputs the spectrum matrix of a target audio frame into the vector extraction model, and performs at least one first convolution on the spectrum matrix through the first convolution layer to obtain a first intermediate vector. In some embodiments, after the first convolution, the data volume of the first intermediate vector is large, which makes the calculation pressure of the computer device large. The vector extraction model obtains the second intermediate vector by performing at least one pooling operation on the first intermediate vector. The data volume of the second intermediate vector obtained after the pooling operation is less than the data volume of the first intermediate vector. Then the vector extraction model performs at least one second convolution on the second intermediate vector to obtain a third intermediate vector. The feature vector of the audio frame is obtained by performing multiple full connection operations on the third intermediate vector.

[0080] In some embodiments, a large-size convolution kernel (e.g., 5*5) is used for convolution in the first convolution layer and the second convolution layer to obtain a larger receptive field and facilitate the extraction of high-level features. The convolution kernel may also be of other sizes, such as 3*3, 4*4, etc. The size of the convolution kernel is determined according to the actual situation and is not limited in this application.

[0081] Exemplarily, network layer structural parameters of a vector extraction model are shown in Table 1 (Structure table of vector extraction model).

[0082] Table 1 Structure of vector extraction model

[0083] Layer Type Details Dimensions of a vector Input Layer - (1，20，50) Convolutional Layer 5x5 window, 6 channels (6，16，46) Pooling Layer 2x2 window, stride 2 (6，8，32) Convolutional Layer 5x5 window, 16 channels (16，4，19) Fully connected layer - (1216) Fully connected layer - (840) Fully connected layer - (10)

[0084] Please refer to Figure 5 , which exemplarily shows a schematic diagram of the process of extracting feature vectors by the vector extraction model. The computer device inputs the spectrum matrix [1, 20, 50] ([1, 20, 50] is the size of the spectrum matrix) into the vector extraction model. The vector extraction model uses a first convolution kernel of 5*5, 6channel (channel) to perform a first layer of convolution on the spectrum matrix to obtain a first intermediate vector [6, 16, 46]. Subsequently, the vector extraction model pools the first intermediate vector, uses a 2*2 filter, and a stride of 2 to pool the first intermediate vector to obtain a second intermediate vector [6, 8, 32]. The vector extraction model uses a 5*5, 16channel convolution kernel to perform a second layer of convolution on the second intermediate vector to obtain a third intermediate vector [16, 4, 19]. Subsequently, the third intermediate vector passes through three fully connected layers in the vector extraction model in succession to obtain a feature vector.

[0085] The following is an introduction to the training method of the vector extraction model. In some embodiments, the vector extraction model is trained, including: obtaining at least one training data, the training data including sample speech data and instruction elements corresponding to the sample speech data; performing frame division and time-frequency conversion on the sample speech data to obtain at least one spectrum matrix; processing the spectrum matrix through the vector extraction model to obtain a prediction vector; calculating the training loss of the vector extraction model according to the element vector corresponding to the prediction vector and the instruction element; and adjusting the parameters of the vector extraction model based on the training loss.

[0086] In some embodiments, the training process of the vector extraction model can be performed on a computer device. In other embodiments, the training process of the vector extraction model can be performed on other computer devices, and after the training of the vector extraction model is completed, the other computer device sends the trained vector extraction model to the computer device so that the computer device applies the vector extraction model to the process of converting speech into program code.

[0087] Sample voice data refers to voice data used for training. Multiple sample voice data that implement the same function can be obtained by manual collection or other methods. In some embodiments, the duration, timbre, expression method, etc. corresponding to multiple sample voice data that implement the same function are not completely the same.

[0088] In some embodiments, the sample speech data corresponds to at least one instruction element, and the element vector corresponding to the at least one instruction element is used as a label in the vector extraction model training process.

[0089] For any function that can be implemented by program code, at least one instruction element corresponding to the function can be set by pre-setting (the number of instruction elements corresponding to the function is related to the average of the length of the voice data corresponding to the function and the frame duration, and these instruction elements can form an instruction element combination in a certain order), and the element vectors corresponding to each instruction element in the above at least one instruction element can be written in advance.

[0090] In some embodiments, the element vector corresponding to the instruction element is manually labeled. The following is an introduction to the labeling method of the instruction element. For example, taking the "print" function as an example, assuming that the function corresponds to 4 instruction elements, the dimension of the element vector corresponding to the instruction element is 6 (in fact, the dimension of the element vector is determined by the dimension of the time-frequency matrix that the audio frame can be converted into and the network structure of the vector extraction network). By presetting, the element vector corresponding to the first instruction element can be set to [0, 0, 0, 2, 0, 0], the element vector corresponding to the second instruction element can be set to [1, 0, 9, 0, 0, 0], and the element vectors corresponding to the remaining instruction elements are also set in the above form.

[0091] Before the speech sample data vector extraction model is used, the speech sample data needs to be framed and time-frequency converted. For the specific process, please refer to the above embodiment. The specific content of the element vector corresponding to the instruction element (the numerical value of each position in the vector) can be determined arbitrarily. Since the vector extraction model uses the element vector as a label and uses the speech training sample for training, the prediction vector corresponding to the speech training sample audio frame extracted by the vector extraction model is obtained. And the training loss between the prediction vector and the element vector is calculated according to the loss function (such as the cross entropy function), and the model parameters are adjusted according to the training loss.

[0092] In some embodiments, based on the frame vector, determining the instruction element corresponding to the target audio frame includes: obtaining element vectors corresponding to multiple candidate instruction elements respectively; calculating the similarity between the frame vector and the element vectors corresponding to each candidate instruction element respectively; and determining the candidate instruction elements whose similarity meets the first condition as the instruction elements corresponding to the target audio frame.

[0093] The similarity is used to characterize the degree of proximity between the frame vector and the element vector corresponding to the instruction element. In some embodiments, the degree of proximity between the frame vector and the element vector corresponding to the instruction element can be represented by the numerical value of the similarity. In some embodiments, the numerical value of the similarity is positively correlated with the degree of similarity, that is, if the numerical value of the similarity between the frame vector and the instruction element corresponding to a candidate instruction element is larger, it means that the frame vector is closer to the instruction element, and conversely, if the numerical value of the similarity between the frame vector and the element vector corresponding to a candidate instruction element is smaller, it means that the difference between the frame vector and the instruction element is larger, then the instruction element cannot be used to replace the frame vector. In other embodiments, the numerical value of the similarity is negatively correlated with the degree of similarity, that is, if the numerical value of the similarity between the frame vector and the element vector corresponding to a candidate instruction element is larger, it means that the similarity between the frame vector and the instruction element is lower, and conversely, if the numerical value of the similarity between the frame vector and the element vector corresponding to a candidate instruction element is smaller, it means that the similarity between the frame vector and the instruction element is higher.

[0094] The first condition refers to a standard for determining the instruction element corresponding to the frame vector from at least one candidate instruction element by similarity. In some embodiments, the first condition can determine the instruction element that uniquely corresponds to the frame vector from at least one instruction element. For example, the first condition is: determining the instruction element with the highest similarity as the instruction element corresponding to the frame vector.

[0095] In other embodiments, more than one instruction element can be screened out from at least one instruction element by the first condition. For example, the first condition is to use the candidate instruction element whose similarity satisfies the similarity threshold as the instruction element corresponding to the frame vector. Assuming that there is a positive correlation between the numerical value of the similarity and the similarity, there is a frame vector A, and at least one candidate instruction element is instruction element a, instruction element b, and instruction element c, respectively. The similarities between frame vector A and instruction element a, instruction element b, and instruction element c are 0.6553, 0.6020, and 0.2095, respectively, and the similarity threshold is 0.6000. Then the computer device preliminarily determines instruction element a and instruction element b as the instruction elements corresponding to frame vector A. Normally, a frame vector can only correspond to one instruction element. Therefore, the computer device can select any one of instruction element a and instruction element b as the instruction element corresponding to frame vector A. For example, select instruction element b as the instruction element corresponding to frame vector A. In addition, the computer device can also determine the instruction element corresponding to frame vector A based on the instruction elements corresponding to adjacent frame vectors. For example, frame vector A is adjacent to frame vector B, the instruction element corresponding to frame vector B is instruction element a, instruction element a and instruction element a can form an instruction command or part of an instruction command, instruction element a and instruction element b cannot form any instruction, then the instruction element that can form an instruction with the adjacent instruction element can be preferentially determined as the instruction element corresponding to frame vector A (instruction element a).

[0096] In some embodiments, the frame vector is an x-dimensional column vector, and the instruction elements corresponding to at least one candidate instruction element are also x-dimensional column vectors, where x is a positive integer. The numerical value of the similarity can be obtained by multiplying the transpose of the element vector corresponding to a candidate instruction element with the frame vector (multiplying a row vector and a column vector of equal dimensions). For example, if the value obtained by multiplying the transpose of the element vector corresponding to a certain instruction element with a certain frame vector is y, then y is the numerical value of the similarity between the instruction element and the frame vector, and y is a real number.

[0097] In this case, the first condition may be: determining the instruction element corresponding to the maximum numerical similarity as the instruction element corresponding to the frame vector. For example, the frame vector C = [0.2543, 0.8599, 0.1091] T , the instruction element corresponding to instruction element m = [0, 1, 0] T , the instruction element corresponding to instruction element n = [1, 0, 0] T The computer device calculates the similarity values ​​corresponding to the frame vector C and the instruction element m and the instruction element n respectively: m T C = 0.8599, n TC=0.1091. Since 0.8599 is greater than 0.1091, it means that the frame vector C is close to the instruction element m, and the instruction element corresponding to the frame vector C is the instruction element m.

[0098] It should be noted that the dimensions of the frame vector, the dimensions of the instruction elements and the values ​​appearing in the above embodiments are only examples and should not be construed as limiting the dimensions of the frame vector, the dimensions of the instruction elements and the specific values ​​in this application.

[0099] Since the audio frame may contain interference signals such as noise, there may be deviations in the frame vector extracted from the time-frequency matrix. Using the frame vector converted into instruction elements instead of the frame vector helps to eliminate the errors in the frame vector and improve the accuracy of converting the target speech data into the reference program code. These errors may come from the interference information included in the audio frame itself, or may be caused in the process of extracting the frame vector (such as deviations in the vector extraction model).

[0100] The following introduces a method for obtaining reference program codes corresponding to target voice data based on instruction elements.

[0101] In some embodiments, a computer device generates a reference program code corresponding to the target voice data based on multiple instruction elements, including: determining at least one instruction element combination based on multiple instruction elements; determining the reference program code corresponding to each instruction element combination according to a mapping relationship; and obtaining the reference program code corresponding to the target voice data based on the reference program code corresponding to each instruction element combination.

[0102] The instruction element combination includes multiple instruction elements. There is a corresponding relationship between the instruction element and the command with the instruction function. In some embodiments, the instruction element combination can be composed of 1-6 instruction elements, and the experimental verification is improved. The instruction element combination of such a scale can cover 95% of the functions, and controlling the maximum number of instruction elements included in the instruction element combination helps to reduce the amount of calculation for determining the instruction element combination from multiple instruction elements.

[0103] In some embodiments, after obtaining multiple instruction elements, the computer device selects several instruction elements from the multiple instruction elements, and forms an instruction element combination through these instruction elements, and uses the reference program code corresponding to the instruction element combination as the program code corresponding to the target voice data. The computer device can select several instruction elements from the multiple instruction elements at will, or select several instruction elements from the multiple instruction elements according to certain conditions, for example, the computer device selects several instruction elements that can form a preset instruction element combination from the multiple instruction elements.

[0104] In some embodiments, a computer device determines at least one instruction element combination based on multiple instruction elements, including: arranging multiple instruction elements in a first order to obtain an element sequence; wherein the first order is related to time; determining at least one interval in the element sequence, the interval being used to determine the starting position of the instruction element combination; dividing the element sequence into at least one element subsequence according to the at least one interval; for a target element subsequence in at least one element subsequence, determining the instruction element combination in the target element subsequence according to a second order; wherein the second order is related to the first order; obtaining at least one instruction element combination based on the instruction element combinations determined separately based on at least one element subsequence.

[0105] The first order is used to sort the instruction elements so as to obtain the rule of the element sequence. In some embodiments, the first order is related to time. Since the multiple instruction elements obtained by the computer device correspond to a frame vector respectively, and these frame vectors belong to the audio frames of the target voice data, there is a corresponding relationship between the instruction elements and the audio frames. Because the positions of different audio frames in the target voice data are different, that is, the distribution of audio frames in the target voice data is different. Therefore, different instruction elements have different corresponding positions in the target voice data. Arrange the instruction elements according to the first order so that the determined element sequence can correspond to the target voice data.

[0106] For example, a target speech data with a duration of 10ms is divided into frames to obtain 5 audio frames, each of which has a length of 200ms. Audio frame 1 (0-200ms) corresponds to instruction element 1, audio frame 2 (200-400ms) corresponds to instruction element 2, audio frame 3 (400-600ms) corresponds to instruction element 3, audio frame 4 (600-800ms) corresponds to instruction element 4, and audio frame 5 (800-1000ms) corresponds to instruction element 5. Then it can be determined that instruction element 1 is distributed in 0-200ms of the speech data, instruction element 2 is distributed in 200-400ms of the speech data, and so on. Instruction element 5 is distributed in 800-1000ms of the speech data. In some embodiments, the first order is the order of time development (along the time axis direction), or the reverse order of time development (against the time axis direction). In other embodiments, the first order is the order in which the computer device obtains the instruction elements.

[0107] The interval is used to indicate the start of determining the combination of instruction elements, or the interval is used to segment the sequence of instruction elements. In some embodiments, the interval is at least divided into: a sound wave interval and a program interval. The sound wave interval refers to the interval in which the target voice data itself exists. Optionally, the sound wave interval is determined by detecting whether the decibel of the audio frame is lower than the rated decibel, or detecting whether the frequency of the audio frame is lower than the rated frequency. The sound wave interval can be understood as a gap in the target voice data, that is, a position where there is a pause in the voice signal. It is not difficult to understand that the instruction element used to indicate the sound wave interval does not contain information related to the reference program code, or the instruction element is a blank vector that does not carry information (the values ​​of each position of the vector are 0).

[0108] In some embodiments, the computer device detects each instruction element in the element sequence to determine the sound wave interval of the element sequence. For example, there is an instruction element [0, 0, 0, 0, 0, 0], which does not include any information. It is assumed that the instruction element corresponds to an instruction command S (also called a sound wave interval instruction), that is, the instruction command S is used to identify the sound wave interval.

[0109] The program interval is artificially set to indicate the starting position of determining the instruction from the element sequence. In some cases, there is no sound wave interval in the element sequence of a certain target voice data, such as the duration of the voice data is short, and the pronunciation in the voice data is continuous. In this case, the computer device needs to determine the starting position of the command combination through the program interval, so that the process of determining the command combination through the interval can be carried out normally. In some embodiments, the computer device determines the position of q, 2q...nq in the element sequence as the program interval, and n, q are all positive integers. For example, the computer device will be in the element sequence, the 6th position, the 12th position...determined as the program interval (for example, the 6th position and the 12th position in the element sequence are placed to indicate the instruction element of the interval).

[0110] In some embodiments, the priority of the program interval is lower than the sound wave interval. The computer device first determines whether there is a sound wave interval in the instruction sequence. If there is no sound wave interval in the element sequence, the computer device determines the program interval in the element sequence.

[0111] In some embodiments, the duration of the target voice data is very short, which means that there is neither a sound wave interval nor a program interval in the element sequence, and the computer device uses the position of the first instruction element in the element sequence as the interval, or uses the position of the last instruction element in the element sequence as the interval. According to the interval determined above, the computer device determines the instruction corresponding to the element sequence.

[0112] After at least one interval is determined from the element sequence, the computer device divides the element sequence according to the interval to obtain at least one element subsequence. For example, the element sequence M is [a, b, b, d, a, f], where the instruction element d corresponds to the instruction command S alone, and the instruction command S is used to characterize the sound wave interval, which means that the position of the instruction element d in the element sequence M is the sound wave interval. Based on the instruction element d, the computer device divides the element sequence M into two element subsequences, namely M1[a, b, b] and M2[a, c]. For another example, the element sequence N is [a, b, e, d, a, c], where the instruction element c corresponds to the instruction command S alone, and the instruction command S is used to characterize the sound wave interval, then the position of the instruction element c in the element sequence M is the sound wave interval, and the element sequence N is divided into an element subsequence, namely N1[a, b, b, d, a]. In the above two examples, since the instruction element for indicating the sound wave interval does not include information related to the instruction, the element subsequence divided according to the sound wave interval does not include the instruction element d or the instruction element c.

[0113] An element subsequence is part of an element sequence, and the number of instruction elements contained in the element subsequence is less than or equal to the number of instruction elements in the element sequence. In some embodiments, an element subsequence contains all instruction elements related to instructions in the element sequence. An element subsequence may contain one instruction; an element subsequence may also contain no instructions, that is, the computer device cannot identify any preset instruction from the element subsequence. Segmenting the element sequence helps to reduce the amount of calculation in the process of determining the candidate element combination, improve the efficiency of converting the target voice data into reference program code, and improve the response speed of the device.

[0114] In some embodiments, the computer device is capable of obtaining all instruction elements of the voice data, in which case the computer device sorts the multiple instruction elements according to the first order (distribution of the instruction elements in the voice data) to obtain an element sequence. The computer device determines at least one interval (sound wave interval or program interval) from the element sequence, and divides the element sequence into at least one element subsequence based on the interval, and determines the instructions corresponding to each element subsequence.

[0115] In some embodiments, the computer device cannot obtain all the instruction elements corresponding to the voice data at the same time. For example, after the computer device obtains the voice data, it will frame the voice data, obtain at least one audio frame belonging to the voice data, and then perform feature extraction on the at least one audio frame to obtain the frame vectors corresponding to each audio frame, and the instruction element with the highest similarity to the frame vector. Therefore, the computer device can only determine the instruction elements intermittently or one by one.

[0116] In this case, after determining a certain instruction element, the computer device determines whether the instruction element corresponds to an instruction indicating a sound wave interval (such as the instruction element is [0, 0, 0, 0, 0]), or determines whether the instruction element is a program interval; if not, the computer device places the instruction element in the element sequence in the first order, and if so, the computer device determines the element sequence as an element subsequence and clears the element sequence. Subsequently, the computer device continues to obtain the next instruction element. For example, the instruction element s is [0, 0, 0, 0] (corresponding to an instruction indicating a sound wave interval), and the computer device obtains the instruction elements a, b, s, o, l, u, t, e in the first order. The computer device successively obtains instruction elements a and b. Since instruction elements a and b are not sound wave intervals, the current element sequence is [a, b]. After the computer device obtains the indicator element s, it is determined that this is a sound wave interval. The computer device uses [a, b] as an element subsequence. The computer device obtains instruction elements o, l, u, t, e one by one. At this time, the current element sequence is [o, l, u, t, e]. The computer device uses instruction element e as a program interval to determine another element subsequence [o, l, u, t, e]. Optionally, the first order is the order in which the computer device determines the instruction elements. For example, the time period corresponding to audio frame 1 is (0-50ms) and the time period corresponding to audio frame 1 is (20-70ms). The computer device determines the instruction element corresponding to audio frame 1 before determining the instruction element corresponding to audio frame 2.

[0117] In some embodiments, in a target element subsequence, an instruction element combination is determined according to a second order, including: determining at least one candidate element combination corresponding to the nth instruction element in the target element subsequence according to the second order, where n is a positive integer; and determining the candidate element combination that satisfies a second condition as the instruction element combination, and the second condition is used to improve the performance of the speech conversion process.

[0118] In some embodiments, the element subsequence is determined according to the interval period, so the position distribution of the starting instruction element in the element sequence is related to the interval period. In some embodiments, the starting instruction element in the element subsequence is determined by the interval period, and starting from the starting instruction element, the instruction element combination is searched in the element subsequence. In some embodiments, the starting instruction element is the first instruction element in the element subsequence, or the last instruction element, and each element subsequence has a starting instruction element.

[0119] For example, the instruction element in the element sequence that is adjacent to the instruction element used to indicate the sound wave interval can be used as the starting instruction element. Assuming that a certain element sequence is [a, c, k, o, n, g, s], s is the instruction element used to indicate the sound wave interval, then based on the instruction element s, the above element sequence can be segmented to obtain an element subsequence, specifically [a, c, k, o, n, l]. Since in this element sequence, the instruction element g is adjacent to the instruction element s, the instruction element g can be determined as the starting instruction element of the above element subsequence.

[0120] In some embodiments, the instruction element combination is preset. In a computer speech framework, there is a mapping relationship between the instruction element combination and the instruction command, and different instruction element combinations correspond to different instruction commands, that is, one instruction element combination is used to represent a specific task.

[0121] like Figure 6 As shown, it shows a schematic diagram of the correspondence between a command element combination and an indication command. The command element combination corresponding to indication command A is [a, b, c], the command element combination corresponding to indication command B is [a, b, c, d], the command element combination corresponding to indication command C is [a, b], and the command element combination corresponding to indication command D is [c, d].

[0122] The second order is used to specify the order of determining the instruction element combination from the element subsequence. In some embodiments, the second order is opposite to the first order. For example, the first order is the order of time development (along the time axis direction), and the second order is the reverse order of time development (against the time axis direction).

[0123] In some embodiments, the second condition includes: the length of the candidate element combination is greater than or equal to the first threshold; or the influence of the candidate element combination on the target element subsequence is less than or equal to the second threshold.

[0124] The second condition is used to determine the instruction element combination corresponding to the element subsequence from at least one instruction element combination. For example, the second condition is that the number of instruction elements contained in the instruction element combination is the largest. For another example, the second condition is that the number of remaining instruction elements in the element subsequence is the smallest.

[0125] In some embodiments, the second condition is related to the length of the candidate element combination, for example, the second condition is to select the instruction element combination with the largest length from at least one candidate element combination. The longer the instruction element combination is, the more instruction elements it contains, and the more accurate the information expressed is.

[0126] In some embodiments, the second condition is related to the influence of the candidate element combination on the target element subsequence. The influence is related to the influence of the candidate element combination on the number of instruction elements that may remain in the target element subsequence. For example, the target instruction element subsequence is [c, k, s, f, j, u], a candidate element combination 1 is [c, k, s, f], and another candidate element combination 2 includes [c, k, s] and [f, j, u]. If candidate element combination 1 is determined as the instruction element combination, the remaining instruction elements that cannot be matched in the target instruction element subsequence are instruction elements j instruction elements u. If candidate element combination 2 is determined as the instruction element combination, the target instruction element subsequence does not have remaining instruction elements that cannot be matched, that is to say, candidate element combination 2 has the least influence on the target subsequence, so candidate element combination 2 can be determined as the instruction element combination corresponding to the target subsequence.

[0127] Please refer to Figure 7 , which shows a schematic diagram of determining the combination of instruction elements by the second order. The second order is from right to left (such as Figure 7 ). The element subsequence [a, b, c, d, c], pre-set using Figure 6The correspondence between the instruction element combination and the instruction command shown in the figure assumes that instruction element a is the starting instruction element of the element subsequence. The computer device starts with instruction element a and determines the instruction element combination in the element subsequence. Since the first digit in instruction A, instruction B and instruction C is instruction element a, the instruction corresponding to the element subsequence may be instruction A, instruction B, or instruction C. The computer device continues to read from the element subsequence in the second order to obtain instruction element b. The computer device traces back (starting from instruction element b, reverse the second order, and finds the instruction element combination with the largest number of instruction elements including instruction element b). Since there is only instruction element a before instruction element b, the computer device obtains the instruction element combination [a, b] (corresponding to instruction C). The computer device continues to read from the element subsequence in the second order to obtain instruction element c. The computer device starts from instruction element c and traces back to obtain instruction elements b and c (instruction elements b and c cannot correspond to the existing instruction element combination, but instruction elements b and c are components of the instruction element combination [a, b, c], so the backtracking process can continue), and continues to trace back to obtain the instruction element combination [a, b, c] (corresponding to instruction B). Continuing the above process, the instruction element combination [a, b, c, d] (corresponding to instruction A) can be obtained. Before reading instruction element c from the element subsequence, the instruction element combination [a, b, c, d] is the instruction element combination containing the largest number of instruction elements. The computer device reads the last instruction element c in the element subsequence, and traces back from instruction element c. Instruction element c and the adjacent instruction element d can form an instruction element combination [c, d] (corresponding to instruction command D). Continue to trace back to obtain instruction element c. Since [c, d, c] does not belong to the predefined instruction element combination, the current instruction element combination is still [c, d]. The computer device starts to backtrack from instruction element c. Since [c, b] is not a predefined element combination, the current instruction element combination is still [c, d]. The computer device continues to backtrack along instruction element b and obtains [a, b, c], which corresponds to instruction command A. The instruction command corresponding to this element subsequence is a combination of instruction commands A and D.

[0128] like Figure 8 As shown, it shows a schematic diagram of the frame vector extraction process. The computer device converts the audio frame into a spectrum matrix, and inputs the spectrum matrix into a vector extraction model, and the vector extraction model processes the spectrum matrix to obtain a frame vector corresponding to the audio frame.

[0129] like Fig. 9As shown, it shows a schematic diagram of converting speech into code. First, a preprocessing process is performed, such as training a vector extraction model, using artificially set instruction elements as labels, obtaining training speech data corresponding to the instruction elements, and training the vector extraction model through the training speech data. In addition, it is also necessary to set a mapping relationship between the reference program code and the instruction command. In the process of converting the target speech data into the reference program code, it is necessary to obtain the target speech data, perform vectorization processing to obtain a frame vector; elementize the frame vector instruction to determine its corresponding instruction element; and perform instructionization operations to determine the instruction command corresponding to the instruction element combination, and determine the reference program code corresponding to the instruction command.

[0130] By performing a backtracking operation in the element subsequence and determining the candidate element combination that meets the second condition as the instruction element combination, it is possible to ensure that each instruction tuple in the element subsequence participates in the process of determining the reference program code, providing more relevant information for the reference program code determination process, which is conducive to improving the accuracy of the reference program instructions obtained by converting the target voice data.

[0131] Fig.10 A flowchart of a code display method provided by an exemplary embodiment of the present application. For example, the execution subject of the method may be Figure 1 The terminal device 10 in the implementation environment of the solution. For example, the execution subject may be a client of the target application running on the terminal device 10. Fig.10 As shown, the method may include the following steps (1010-1030):

[0132] Step 1010, obtaining target voice data.

[0133] In some embodiments, the terminal device receives voice data of an object in response to a voice acquisition operation. Voice data refers to the voice input by the object that is acquired by the terminal device through recording or other means. The implementation methods of the voice acquisition operation include, but are not limited to, touch operation, gesture operation, voice operation, etc. For example, a voice entry control is displayed on the user interface, and the terminal device acquires the target voice data in response to a voice acquisition operation on the voice entry control (such as clicking or long pressing the voice entry control). In some embodiments, the terminal device determines that the voice input is complete through a second operation on the voice entry control (such as canceling contact with the voice entry control).

[0134] Step 1020, displaying the reference program code corresponding to the target voice data in the code display interface, the reference program code is generated based on multiple instruction elements converted from the target voice data, and the instruction elements are used to represent the instruction information carried in the audio frame of the target voice data.

[0135] After obtaining the target voice data, the terminal device converts the target voice data into reference program code and displays the reference program code in the code display interface. For details on converting the target voice data into reference program code, please refer to the embodiment of the code generation method side.

[0136] Step 1030 , in response to the running operation on the reference program code, displaying first interactive content, where the first interactive content is used to represent the running process of the reference program code.

[0137] The interactive content is used to display the running process or running results of the reference program code. The types of interactive content include but are not limited to: prompt information, interactive effects, demonstration animations, etc. In some embodiments, the display effects of the reference program code that can be successfully run are different from those of the reference program code that cannot be successfully run. Among them, the prompt information is used to prompt the running results of the reference program code. For example, the interactive content does not prompt information. When the reference program code runs successfully, the interactive content can be displayed as prompt information related to the successful operation, such as "correct answer", "accept", etc. In the case where the reference program code fails to run, the interactive content can be displayed as prompt information related to the failure of the operation, such as "wrong answer", "please try again", etc. The demonstration animation is used to intuitively display the running process of the reference program code, so as to facilitate the object to analyze the running logic of the reference program code. For example, the interactive content includes virtual creatures (and other virtual objects). When the reference program code is used to indicate "moving forward", the interactive content can be displayed as the virtual creature moving forward according to the instructions of the reference program code.

[0138] In some embodiments, the display form of the interactive content is affected by the reference program code. Since the reference program code is used to solve a specific problem, and there is more than one reference program code that can successfully solve the specific problem, the interactive content displayed during the operation of the reference program code with different structures for solving the same problem may be the same or different.

[0139] In some embodiments, the code display interface displays multiple reference program codes, which can be determined based on the voice data input by the object once, or can be determined based on the voice data input by the object multiple times. In some embodiments, the terminal device responds to the operation of running the reference program code and runs a displayed reference program code separately. For example, the terminal device responds to the selection operation of the target reference program code (such as clicking on the reference program code) and displays the reference program code in a differentiated manner. The terminal device responds to the operation of running the target reference program code and displays the interactive content corresponding to the target reference program code. In other embodiments, the terminal device responds to a running operation and runs multiple reference program codes at a time according to the display order of the reference program code (that is, the order of multiple reference program codes determined according to voice data). The multiple instructions can be all the reference program codes displayed by the terminal device, or multiple reference program codes selected by the object.

[0140] By running part of the reference program code separately, the object can refer to the running effect of the part of the reference program code and debug the part of the reference program code to assist the object in analyzing problems and improve the flexibility of running the reference program code.

[0141] In some embodiments, the interactive content is determined by the terminal device. The terminal device determines the interactive content based on the running status of the reference program code. The determination of the interactive content by the terminal device helps to reduce the pressure on the server. In other embodiments, the interactive content is determined by the server. After determining the reference program code corresponding to the voice data, the server can also pre-run the reference program code to determine the interactive content corresponding to the reference program code. Subsequently, the server sends the reference program code and the interactive content corresponding to the reference program code to the terminal device to further reduce the computing power of the terminal device, improve the response speed of the terminal device, and reduce the waiting time of the object.

[0142] Optionally, the server may simultaneously send the reference program code and the interactive content corresponding to the reference program code to the terminal device. In this case, the terminal device displays the reference program code on the object interface, and displays the corresponding interactive content in response to the operation of running the reference program code. Optionally, the server sends the reference program code and the interactive effect corresponding to the reference program code to the terminal device respectively. For example, after the server determines the reference program code, the server sends the reference program code to the terminal device. The terminal device receives and displays the reference program code. In response to the operation of running the reference program code, the terminal device sends interactive content request information to the server, and the server sends interactive content to the terminal device based on the interactive content request information, and the terminal device receives and displays the interactive content.

[0143] like Fig.11 As shown, it shows a schematic diagram of a code display interface.

[0144] The terminal device responds to the voice acquisition operation (such as the object clicks the voice input control 1110) to acquire voice data, and the terminal device acquires the reference program code based on the voice data and displays it in the program code display box 1120. The terminal device responds to the running operation for the reference program code (such as the object clicks the running control 1130) and displays the interactive content in the interactive content display box 1140.

[0145] By displaying corresponding interactive content (eg, demonstration animation) in response to the running operation on the reference program code, it helps to improve the sense of interactivity of running the reference program code and stimulate the learning interest of the subject.

[0146] In some embodiments, taking the target application as an application for a children's programming game as an example, when a child (or other user (such as a supervisor who helps children learn programming)) is unable to enter the correct program code by themselves, the child or other user can enter target voice data that can indicate the effect of a certain program code through the client.

[0147] For example, if a child or other user wants to obtain a program code for judgment, the input target voice data may be "Move right when "condition a" is met". After the target voice data is processed by the background, the reference program code corresponding to the voice data may be:

[0148] "while(condition a){int x++;}"

[0149] Wherein, x is used to represent the horizontal coordinate value of the position of the position pointer (variable in the program code, or understood as the current position of the program code running process) before executing the reference program code. It should be noted that the representation form of the reference program code is only used as an example, and the corresponding computer language (or pseudocode) form does not limit the computer language to which the reference program code that can be obtained in this method belongs.

[0150] After the reference program code is displayed in the code display interface, the child (or other object using the client) can trigger an operation to run the reference program code. In response to the operation to run the reference program code, the client displays the interactive content corresponding to the reference program code. For example, in this embodiment, the interactive content can be a virtual object in the screen moving one unit to the right. Optionally, the interactive content can be Fig.10 The interactive content is demonstrated in the display box shown.

[0151] In summary, through the above method, the subject only needs to input the target voice data to obtain the reference program code, which reduces the difficulty for the subject to obtain the reference program code. In addition, during the operation of the reference code, the interactive content can reflect the operation logic of the reference program code, which reduces the learning difficulty of the subject. For parents who are tutoring their children in program code, they do not need to have relevant knowledge, and can obtain the reference program code by inputting voice, so as to help their children learn program code.

[0152] By converting the target voice data into instruction elements and determining the reference program code corresponding to the target voice data, the process of converting the target voice data into text is saved, which helps to reduce the time required to convert voice data into text and improve the efficiency of obtaining the reference program code.

[0153] The code display method is introduced below through several embodiments.

[0154] In some embodiments, after displaying the reference program code corresponding to the target voice data in the code display interface, it also includes: displaying an editing pointer and an editing control in response to an editing operation on the reference program code; obtaining editing content in response to an input operation on the editing control; and displaying the editing content at the display position of the editing pointer.

[0155] In some embodiments, the editing operation can be triggered by sliding, clicking, pressing, gesture, voice, etc. For example, the terminal device displays an editing pointer and an editing control in response to a long press operation on the display position of the reference program code. The editing pointer is used to indicate the currently selected target reference program code, and the editing control is used to receive editing information input by the object, such as a keyboard, a control for deleting the reference program code, etc.

[0156] In some embodiments, the terminal device obtains the editing information of the object input in response to the input operation of the editing control. The editing information can be text information, such as the modification and replacement information of the reference program code part or all areas input by the object, or functional information, such as the deletion information indicated by the delete key.

[0157] The terminal device can display the edited content input by the object in the code display interface in real time, so that the object can determine whether to continue to modify according to the displayed content in the code display interface.

[0158] In some embodiments, the editing of the reference program code by the object includes but is not limited to: changing the display position of the reference program instruction; adding new program code to the reference program code displayed in the code display interface, deleting the reference program code, etc. For example, the object can delete the reference program code in the selected state (such as a part of the reference program code displayed in the code display interface). For another example, the object can directly modify the text content of the reference program code displayed on the code display interface.

[0159] There may be a gap between the reference program code determined by voice data and the target reference program code that the object wants to obtain. By displaying the modified reference program code in response to the object's modification operation on the reference program code, the flexibility of the reference program code display method is improved and the success rate of the reference program code operation is increased.

[0160] In some embodiments, after displaying the reference program code corresponding to the target voice data in the code display interface, it also includes: determining auxiliary information corresponding to the reference program code, the auxiliary information is used to annotate the reference program code; determining the display position of the auxiliary information based on the display position of the reference program code; and displaying the auxiliary information according to the display position of the auxiliary information.

[0161] Auxiliary information is used to explain the functions of the reference program code. In some embodiments, the auxiliary information may be the functions of the reference program code, syntax annotations, etc. For example, the auxiliary information corresponding to the reference program code print("32") may be "print value 32", or "print(values, sep, end, file)", etc. By displaying auxiliary information, the subject can become familiar with the usage of the reference program code. For fields such as children's programming games, it helps to improve children's understanding of the reference program code, helps to subtly increase the subject's understanding of the program code, and improves the learning efficiency of children in the process of learning the program code.

[0162] In some embodiments, after displaying the reference program code corresponding to the target voice data in the code display interface, it also includes: displaying an interface switching control in the code display interface; displaying a code editing interface in response to an operation on the interface switching control; displaying the experimental program code in the code editing interface in response to the code editing operation; displaying a second interactive content in response to an operation on the experimental program code, the second interactive content being used to characterize the running process of the experimental program code.

[0163] After the object triggers the interface switching control, the terminal displays the code editing interface. The code editing interface is used to manually write the experimental program code. In some embodiments. By providing an interface switching control, the conversion between the code display interface and the code editing interface is realized, providing two ways for the object to learn program code. The interface switching control can convert the code display interface into the code editing interface, and can also convert the code editing interface into the code display interface.

[0164] In some embodiments, no voice entry control is provided in the code editing interface. In some embodiments, the display position of the code editing interface in the user interface may be the same as the display position of the code display interface, that is, the code display interface is replaced by the code editing interface (such as by displaying / canceling the display of the voice entry control to achieve switching between the code display interface and the code editing interface). By replacing the code display interface with the code editing interface, after triggering the interface switching control, the adjustment of other controls or interfaces in the user interface is reduced during the process of the terminal device switching the code editing interface and the code display interface, which helps to reduce the amount of calculation of the terminal device. In other embodiments, the display position of the code editing interface in the user interface is different from the display position of the code display interface in the user interface, and the code editing interface and the code display interface can be displayed on the user interface at the same time.

[0165] Manually writing experimental codes in the code editing interface helps to trigger the subjects to think and improve the learning effect of program codes. Displaying reference program codes in the code display interface provides subjects with at least one reference program code that can solve the problem. On the one hand, it helps the subjects find the wrong parts in the input experimental program code and improves the subjects' ability to correct errors. On the other hand, the reference program code and interactive content help the subjects (especially children) improve their understanding of the operating logic of the program code.

[0166] In some embodiments, after displaying the experimental program code in the code editing interface, the method further includes: displaying the reference program code and the experimental program code in comparison in the code editing interface.

[0167] In some embodiments, the terminal device displays the reference program code and the experimental program code line by line to obtain a comparison display according to function. For example, the reference program code and the experimental program code that implement the same function are compared and displayed to facilitate the subject to analyze the difference between the two, which helps to assist the subject to find errors in time.

[0168] In some embodiments, the method of converting the above-mentioned speech into code is applied to children's programming games as an example. The child manually enters the experimental program code in the code editing interface and tries to solve the problems raised by the programming game level by editing the experimental program code. After completing the input of the experimental program code, the child triggers the operation for the experimental program code. In response to the operation, the terminal device runs the experimental program code and generates interactive content corresponding to the experimental program code. In the case where there is an error in the experimental code, the child can edit part of the content in the experimental program code in a targeted manner according to the display of the interactive content, so as to obtain a new experimental program code. Through multiple modification attempts, the experimental code written by the child may run successfully and can solve the problem of the programming game level. However, the experimental program code may not run successfully, or the child may actively expect to obtain the reference program code of the current programming game level (for example, the child hopes to compare the reference program code with the experimental program code to learn to optimize the program code structure). In these cases, the child or the parent first needs to trigger the interface switching control. The terminal device switches the code editing interface to the code display interface in response to the operation of the interface switching control. The code display interface is provided with a voice input control, and the parent or child inputs the target voice data through the voice input control. After the terminal device obtains the target voice data, the target voice data is converted into a reference program code by the terminal device or the server, and then the terminal device displays the reference program code in the code display interface. In some embodiments, the terminal device can also display the reference program code for comparison, so as to increase the speed at which the child corrects errors in the experimental program code.

[0169] Through the above method, the client in the terminal device can provide the object with at least two methods of obtaining program codes. In the application scenario where parents tutor their children to learn, parents can obtain reference program codes through voice data to help their children learn. This method requires a relatively simple level or ability of parents and does not waste too much time of parents. For children, they can manually write experimental program codes to trigger thinking so that they can master the relevant content as soon as possible. By comparing the reference program code and the experimental program code, it is also possible to find the location of the error more quickly, which helps to reduce the learning cost of children. At the same time, children can also directly obtain the reference program code through voice input. The method of obtaining the reference program code is simple and easy to operate.

[0170] In other application scenarios, obtaining reference program codes through voice data can help improve the work efficiency of staff and simplify the operation process of staff.

[0171] The following are device embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0172] Fig.12 A block diagram of a code generation device provided by an exemplary embodiment of the present application is shown. The device can be implemented as all or part of a terminal device through software, hardware or a combination of both. The device 1200 may include: a speech acquisition module 1210, a speech framing module 1220, a single instruction module 1230 and a code generation module.

[0173] The voice acquisition module 1210 is used to acquire target voice data.

[0174] The speech framing module 1220 is used to frame the target speech data to obtain multiple audio frames.

[0175] The single instruction module 1230 is used to determine the instruction elements corresponding to each of the audio frames, and obtain a plurality of the instruction elements; wherein the instruction elements are used to represent the instruction information carried in the audio frames.

[0176] The code generation module 1240 is used to generate reference program codes corresponding to the target voice data based on the multiple instruction elements.

[0177] In some embodiments, the single instruction module 1230 includes: a matrix determination submodule, used to perform time-frequency conversion on a target audio frame among the multiple audio frames to obtain a spectrum matrix corresponding to the target audio frame, wherein the spectrum matrix is ​​used to characterize the spectrum information of the target audio frame; a vector determination submodule, used to determine a frame vector corresponding to the target audio frame based on the spectrum matrix, wherein the frame vector is a vector representation of the target audio frame; based on the frame vector, determine the instruction element corresponding to the target audio frame; an element determination submodule, used to obtain the multiple instruction elements based on the instruction elements corresponding to each of the audio frames.

[0178] In some embodiments, the vector determination submodule is used to: input the spectrum matrix into a vector extraction model, where the vector extraction model is a model for extracting vectors from input information; and process the spectrum matrix through the vector extraction model to obtain a frame vector corresponding to the target audio frame.

[0179] In some embodiments, the training process of the vector extraction model is as follows: obtaining at least one training data, the training data including the sample speech data and the instruction elements corresponding to the sample speech data; performing frame division and time-frequency conversion on the sample speech data to obtain at least one spectrum matrix; processing the spectrum matrix through the vector extraction model to obtain a prediction vector; calculating the training loss of the vector extraction model based on the prediction vector and the element vector corresponding to the instruction element; and adjusting the parameters of the vector extraction model based on the training loss.

[0180] In some embodiments, the element determination submodule is used to: obtain element vectors corresponding to multiple candidate instruction elements respectively; calculate the similarity between the frame vector and the element vectors corresponding to each of the candidate instruction elements respectively; and determine the candidate instruction elements whose similarity meets the first condition as the instruction elements corresponding to the target audio frame.

[0181] In some embodiments, the code generation module 1240 includes: a combination determination submodule, used to determine at least one instruction element combination based on the multiple instruction elements; a code determination submodule, used to determine the reference program code corresponding to each of the instruction element combinations according to a mapping relationship; the code determination submodule is also used to obtain the reference program code corresponding to the target speech data based on the reference program code corresponding to each of the instruction element combinations.

[0182] In some embodiments, the combination determination submodule includes: a sequence determination unit, used to arrange the multiple instruction elements in a first order to obtain an element sequence; wherein the first order is related to time; an interval determination unit, used to determine at least one interval in the element sequence, and the interval is used to determine the starting position of the instruction element combination; a subsequence determination unit, used to divide the element sequence into at least one element subsequence according to at least one of the intervals; a combination determination unit, used to determine the instruction element combination in a second order in the target element subsequence in the at least one element subsequence; wherein the second order is related to the first order; a combination selection unit, used to obtain the at least one instruction element combination based on the instruction element combinations respectively determined by the at least one element subsequence.

[0183] In some embodiments, the combination determination unit is used to: determine at least one candidate element combination corresponding to the nth instruction element in the target element subsequence according to the second order, where n is a positive integer; and determine the candidate element combination that satisfies a second condition as the instruction element combination, where the second condition is used to improve the performance of the speech conversion process.

[0184] In some embodiments, the second condition includes: the length of the candidate element combination is greater than or equal to a first threshold; or the influence of the candidate element combination on the target element subsequence is less than or equal to a second threshold.

[0185] Fig.13 A block diagram of a code display device provided by an exemplary embodiment of the present application is shown. The device can be implemented as all or part of a terminal device through software, hardware, or a combination of both. The device 1300 may include: a voice acquisition module 1310, a code display module 1320, and a content display module 1330.

[0186] The voice acquisition module 1310 is used to acquire target voice data.

[0187] The code display module 1320 is used to display the reference program code corresponding to the target voice data in the code display interface. The reference program code is generated based on multiple instruction elements converted from the target voice data, and the instruction elements are used to represent the instruction information carried in the audio frame of the target voice data.

[0188] The content display module 1330 is used to display first interactive content in response to an operation on the reference program code, where the first interactive content is used to represent the operation process of the reference program code.

[0189] In some embodiments, the device 1300 further includes: a code editing module, which is used to display an editing pointer and an editing control in response to an editing operation on the reference program code; obtain editing content in response to an input operation on the editing control; and display the editing content at the display position of the editing pointer.

[0190] In some embodiments, the device 1300 also includes: a code experiment module, used to display an interface switching control in the code display interface; in response to an operation on the interface switching control, display a code editing interface; in response to a code editing operation, display the experimental program code in the code editing interface; in response to a running operation on the experimental program code, display a second interactive content, the second interactive content being used to characterize the running process of the experimental program code.

[0191] In some embodiments, the apparatus 1300 further includes: a comparison display module, configured to display the reference program code and the experimental program code in comparison in the code editing interface.

[0192] It should be noted that the device provided in the above embodiment, when realizing its function, only uses the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device provided in the above embodiment belongs to the same concept as the method embodiment. The specific implementation process is detailed in the method embodiment, which will not be repeated here. Please refer to the description of the method side embodiment for the beneficial effects of the device provided in the above embodiment, which will not be repeated here.

[0193] Fig.14 The structure block diagram of a computer device provided by an exemplary embodiment of the present application is shown. The computer device 1400 may be the terminal device described above, or may be the server described above.

[0194] Typically, the computer device 1400 includes a processor 1401 and a memory 1402 .

[0195] The processor 1401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1401 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1401 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0196] The memory 1402 may include one or more computer-readable storage media, which may be tangible and non-transitory. The memory 1402 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1402 stores a computer program, which is loaded and executed by the processor 1401 to implement the code generation method or code display method provided in the above-mentioned method embodiments.

[0197] Those skilled in the art will understand that Fig.14 The structure shown in the figure does not constitute a limitation on the computer device 1400, and the computer device 1400 may include more or less components than shown in the figure, or combine some components, or adopt a different component arrangement.

[0198] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the code generation method or code display method provided in the above-mentioned method embodiments.

[0199] The computer readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Computer storage media include RAM (Random Access Memory), ROM (Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technology, DVD (Digital Video Disc) or other optical storage, cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer storage medium is not limited to the above-mentioned ones.

[0200] An embodiment of the present application also provides a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor reads and executes the computer instructions from the computer-readable storage medium to implement the code generation method or the code display method provided in the above-mentioned method embodiments.

[0201] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0202] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the target voice data involved in this application are all obtained with full authorization.

[0203] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent switching, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A code generation method, characterized in that: The method comprises: Obtain target voice data; Divide the target voice data into frames to obtain multiple audio frames; Determine the instruction elements corresponding to each of the audio frames to obtain a plurality of instruction elements; wherein the instruction elements are used to represent the instruction information carried in the audio frames; Arranging the plurality of instruction elements in a first order to obtain an element sequence; wherein the first order is an order of time development or a reverse order of time development; Determine at least one interval in the element sequence, the interval being used to determine a starting position of an instruction element combination; According to at least one of the intervals, dividing the sequence of elements into at least one subsequence of elements; For a target element subsequence in the at least one element subsequence, determining the instruction element combination in the target element subsequence according to a second order; wherein the second order is opposite to the first order; Obtain at least one instruction element combination based on the instruction element combinations respectively determined by the at least one element subsequence; Determine, according to the mapping relationship, the reference program code corresponding to each of the instruction element combinations; Based on the reference program codes corresponding to the respective instruction element combinations, the reference program code corresponding to the target voice data is obtained.

2. The method according to claim 1, characterized in that The determining of the instruction elements corresponding to the respective audio frames to obtain a plurality of instruction elements comprises: For a target audio frame among the multiple audio frames, perform time-frequency conversion on the target audio frame to obtain a frequency spectrum matrix corresponding to the target audio frame, wherein the frequency spectrum matrix is ​​used to represent frequency spectrum information of the target audio frame; Based on the frequency spectrum matrix, determining a frame vector corresponding to the target audio frame, where the frame vector is a vector representation of the target audio frame; Based on the frame vector, determining an instruction element corresponding to the target audio frame; The multiple instruction elements are obtained based on the instruction elements respectively corresponding to the audio frames.

3. The method according to claim 2, characterized in that The determining, based on the frequency spectrum matrix, a frame vector corresponding to the target audio frame includes: Inputting the spectrum matrix into a vector extraction model, wherein the vector extraction model is a model for extracting vectors from input information; The frequency spectrum matrix is ​​processed by the vector extraction model to obtain a frame vector corresponding to the target audio frame.

4. The method according to claim 3, characterized in that The training process of the vector extraction model is as follows: Acquire at least one training data, wherein the training data includes sample speech data and instruction elements corresponding to the sample speech data; Performing frame division and time-frequency conversion on the sample speech data to obtain at least one spectrum matrix; Processing the frequency spectrum matrix by using the vector extraction model to obtain a prediction vector; Calculating the training loss of the vector extraction model according to the prediction vector and the element vector corresponding to the instruction element; Based on the training loss, parameters of the vector extraction model are adjusted.

5. The method according to claim 2, characterized in that: The determining, based on the frame vector, the instruction element corresponding to the target audio frame comprises: Obtain element vectors corresponding to multiple candidate instruction elements respectively; Calculating the similarity between the frame vector and the element vectors corresponding to each of the candidate instruction elements; The candidate instruction element whose similarity satisfies the first condition is determined as the instruction element corresponding to the target audio frame.

6. The method according to claim 1, characterized in that The step of determining the instruction element combination in the target element subsequence according to the second order includes: Determine, in the target element subsequence according to the second order, at least one candidate element combination corresponding to the nth instruction element, where n is a positive integer; A candidate element combination satisfying the second condition is determined as the instruction element combination.

7. The method according to claim 6, characterized in that The second condition includes: The length of the candidate element combination is greater than or equal to a first threshold; or, The influence of the candidate element combination on the target element subsequence is less than or equal to a second threshold.

8. A code display method, characterized in that: The method comprises: Obtain target voice data; The reference program code corresponding to the target voice data is displayed in a code display interface, the reference program code corresponding to the target voice data is generated based on a plurality of instruction elements converted from the target voice data, the instruction elements are used to characterize instruction information carried in an audio frame of the target voice data, the plurality of instruction elements are used to determine at least one instruction element combination, and then, according to a mapping relationship, the reference program code corresponding to each instruction element combination is determined, and the reference program code corresponding to each instruction element combination is used to obtain the reference program code corresponding to the target voice data, the at least one instruction element combination is determined for at least one element subsequence in a second order, the at least one element subsequence is obtained by dividing the element sequence according to at least one interval in the element sequence, and the element sequence is obtained by arranging the plurality of instruction elements in a first order; wherein the first order is an order of time development or a reverse order of time development, the second order is opposite to the first order, and the interval is used to determine the starting position of the instruction element combination; In response to an operation on the reference program code, a first interactive content is displayed, where the first interactive content is used to represent the running process of the reference program code.

9. The method according to claim 8, characterized in that After displaying the reference program code corresponding to the target voice data in the code display interface, the method further includes: In response to an editing operation on the reference program code, displaying an editing pointer and an editing control; In response to an input operation on the editing control, acquiring editing content; The edit content is displayed at the display position of the edit pointer.

10. The method according to claim 8, characterized in that After displaying the reference program code corresponding to the target voice data in the code display interface, the method further includes: Determining auxiliary information corresponding to the reference program code, wherein the auxiliary information is used to annotate the reference program code; determining a display position of the auxiliary information based on a display position of the reference program code; The auxiliary information is displayed according to the display position of the auxiliary information.

11. The method according to claim 8, characterized in that After displaying the reference program code corresponding to the target voice data in the code display interface, the method further includes: Displaying an interface switching control in the code display interface; In response to an operation on the interface switching control, displaying a code editing interface; In response to the code editing operation, displaying the experimental program code in the code editing interface; In response to the running operation on the experimental program code, a second interactive content is displayed, where the second interactive content is used to represent the running process of the experimental program code.

12. The method according to claim 11, characterized in that After the experimental program code is displayed in the code editing interface, the method further includes: In the code editing interface, the reference program code and the experimental program code are displayed in comparison.

13. A code generating device, characterized in that: The device comprises: A voice acquisition module, used to acquire target voice data; A voice framing module, used for framing the target voice data to obtain multiple audio frames; A single instruction module, used to determine the instruction elements corresponding to each of the audio frames, and obtain multiple instruction elements; wherein the instruction elements are used to represent the instruction information carried in the audio frames; A code generation module is used to arrange the multiple instruction elements in a first order to obtain an element sequence; wherein the first order is the order of time development or the reverse order of time development; determine at least one interval in the element sequence, and the interval is used to determine the starting position of the instruction element combination; according to at least one of the intervals, divide the element sequence into at least one element subsequence; for a target element subsequence in the at least one element subsequence, determine the instruction element combination in the target element subsequence according to a second order; wherein the second order is opposite to the first order; obtain at least one instruction element combination based on the instruction element combinations respectively determined by the at least one element subsequence; determine the reference program code corresponding to each of the instruction element combinations according to the mapping relationship; obtain the reference program code corresponding to the target voice data based on the reference program code corresponding to each of the instruction element combinations.

14. A code display device, characterized in that: The device comprises: A voice acquisition module, used to acquire target voice data; A code display module, used for displaying the reference program code corresponding to the target voice data in a code display interface, wherein the reference program code corresponding to the target voice data is generated based on a plurality of instruction elements converted from the target voice data, wherein the instruction elements are used to characterize the instruction information carried in the audio frame of the target voice data, wherein the plurality of instruction elements are used to determine at least one instruction element combination, and then, according to a mapping relationship, determine the reference program code corresponding to each of the instruction element combinations, wherein the reference program code corresponding to each of the instruction element combinations is used to obtain the reference program code corresponding to the target voice data, wherein the at least one instruction element combination is determined for at least one element subsequence in a second order, wherein the at least one element subsequence is obtained by dividing the element sequence according to at least one interval in the element sequence, and wherein the element sequence is obtained by arranging the plurality of instruction elements in a first order; wherein the first order is an order of time development or a reverse order of time development, the second order is opposite to the first order, and the interval is used to determine the starting position of the instruction element combination; The content display module is used to display first interactive content in response to an operation on the reference program code, wherein the first interactive content is used to represent the operation process of the reference program code.

15. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the code generation method according to any one of claims 1 to 7, or the code display method according to any one of claims 8 to 12.

16. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which is loaded and executed by a processor to implement the code generation method according to any one of claims 1 to 7, or the code display method according to any one of claims 8 to 12.

17. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor reads and executes the computer instructions from the computer-readable storage medium to implement the code generation method as described in any one of claims 1 to 7, or the code display method as described in any one of claims 8 to 12.

Images