Test log generation method and device, equipment, medium and program product

Abstract

The embodiment of the invention discloses a test log generation method and device, equipment, a medium and a program product. A specific embodiment of the method comprises the following steps: acquiring at least one section of question and answer voice recorded and at least one section of question and answer text output in a process of performing a question and answer function test on a to-be-tested application; for each section of question and answer voice in the at least one section of question and answer voice, executing a generation step: determining at least one voice recognition text corresponding to the question and answer voice by utilizing at least one audio-to-text engine; generating voice error information according to the at least one voice recognition text and a question and answer text corresponding to the question and answer voice; and generating a test log corresponding to the obtained at least one piece of voice error information. The implementation mode is related to voice recognition, information with voice semantic errors can be quickly and accurately positioned through at least one audio-to-text engine, and effective testing of the question and answer function corresponding to the to-be-tested application is achieved.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more specifically to test log generation methods, apparatus, electronic devices, computer-readable media, and program products. Background Technology

[0002] Currently, in scenarios where question-and-answer applications output large amounts of text (e.g., over 1000 characters) and require simultaneous playback of corresponding PCM (Pulse Code Modulation) audio files, the server, for efficiency and file size considerations, splits the complete text into multiple smaller text segments and converts each segment into a PCM audio file in parallel. Existing testing of the question-and-answer functionality of these applications often involves manually verifying whether the audio content of the multiple audio segments matches the text content of the multiple text segments. Manual verification suffers from low testing efficiency and difficulty in locating audio errors. Summary of the Invention

[0003] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0004] Some embodiments of this disclosure provide test log generation methods, apparatuses, devices, media, and program products to address the technical problems mentioned in the background section above.

[0005] In a first aspect, some embodiments of this disclosure provide a test log generation method, including: acquiring at least one segment of question-and-answer speech and at least one segment of question-and-answer text recorded during the question-and-answer function test of the application to be tested; for each segment of the at least one segment of question-and-answer speech, performing a generation step: using at least one audio-to-text engine to determine at least one speech recognition text corresponding to the question-and-answer speech; generating speech error information based on the at least one speech recognition text and the question-and-answer text corresponding to the question-and-answer speech; and generating a test log corresponding to the at least one speech error information obtained.

[0006] Optionally, the above-mentioned use of at least one audio-to-text engine to determine at least one speech recognition text corresponding to the above-mentioned question-and-answer speech includes: for each audio-to-text engine, using the audio-to-text engine, sequentially performing text recognition on the speech segments in the above-mentioned question-and-answer speech at target durations to obtain a recognition information sequence as the speech recognition text.

[0007] Optionally, generating speech error information based on the at least one speech recognition text and the question-and-answer text corresponding to the question-and-answer speech includes: determining the text similarity between each of the at least one speech recognition text and the question-and-answer text corresponding to the question-and-answer speech, to obtain at least one text similarity; in response to the at least one text similarity being lower than a first similarity, performing a text comparison between the at least one speech recognition text and the question-and-answer text corresponding to the question-and-answer speech, to obtain at least one erroneous text; determining at least one speech segment representing the presence of erroneous information in the question-and-answer speech based on the at least one erroneous text; and generating speech error information based on the at least one speech segment.

[0008] Optionally, the above method further includes: in response to the existence of a text similarity higher than the second similarity in at least one of the above text similarities, generating speech error information represented as empty information.

[0009] Optionally, determining the text similarity between each of the at least one speech recognition texts and the question-and-answer text corresponding to the question-and-answer speech includes: performing text preprocessing on the speech recognition texts to obtain preprocessed speech texts; using the preprocessed speech texts as speech recognition texts, and employing a multi-dimensional weight allocation method to determine the text similarity between the speech recognition texts and the question-and-answer text corresponding to the question-and-answer speech.

[0010] Optionally, before obtaining at least one segment of question-and-answer voice recording and at least one segment of question-and-answer text output during the question-and-answer function test of the application under test, the method further includes: in response to accessing the device corresponding to the application under test, instructing the device to initialize the built-in recording application to obtain an initialized application; sending a question-and-answer function test instruction to the device corresponding to the application under test to enable the question-and-answer function of the application under test; performing a question-and-answer interaction with the application under test based on the question-and-answer function interaction simulation instruction, and recording voice using the initialized application during the question-and-answer interaction to obtain at least one segment of question-and-answer voice recording; and instructing the application under test to store at least one segment of question-and-answer text during the question-and-answer interaction.

[0011] Secondly, some embodiments of this disclosure provide a test log generation apparatus, comprising: an acquisition unit configured to acquire at least one segment of question-and-answer speech and at least one segment of question-and-answer text recorded during a question-and-answer function test of an application under test; an execution unit configured to perform a generation step for each segment of the at least one segment of question-and-answer speech: using at least one audio-to-text engine to determine at least one speech recognition text corresponding to the question-and-answer speech; generating speech error information based on the at least one speech recognition text and the question-and-answer text corresponding to the question-and-answer speech; and a generation unit configured to generate a test log corresponding to the at least one segment of speech error information.

[0012] Optionally, the execution unit can be configured to: for each audio-to-text engine, use the aforementioned audio-to-text engine to sequentially perform text recognition on the speech segments in the aforementioned question-and-answer speech at target durations, and obtain a recognition information sequence as speech recognition text.

[0013] Optionally, the execution unit may be configured to: determine the text similarity between each of the at least one speech recognition texts and the question-and-answer text corresponding to the question-and-answer speech, to obtain at least one text similarity; in response to the at least one text similarity being lower than a first similarity, perform a text comparison between the at least one speech recognition text and the question-and-answer text corresponding to the question-and-answer speech, to obtain at least one erroneous text; determine at least one speech segment representing the presence of erroneous information in the question-and-answer speech based on the at least one erroneous text; and generate speech error information based on the at least one speech segment.

[0014] Optionally, the execution unit can be configured to generate speech error information represented as empty information in response to the existence of a text similarity higher than the second similarity in at least one of the above text similarities.

[0015] Optionally, the execution unit can be configured to: perform text preprocessing on the above-mentioned speech recognition text to obtain preprocessed speech text; use the above-mentioned preprocessed speech text as speech recognition text, and use a multi-dimensional weight allocation method to determine the text similarity between the above-mentioned speech recognition text and the question-and-answer text corresponding to the above-mentioned question-and-answer speech.

[0016] Optionally, the apparatus further includes: in response to accessing the device corresponding to the application under test, instructing the device to initialize the built-in recording application to obtain an initialized application; sending a question-and-answer function test instruction to the device corresponding to the application under test to enable the question-and-answer function of the application under test; performing a question-and-answer interaction with the application under test based on the question-and-answer function interaction simulation instruction, and recording voice using the initialized application during the question-and-answer interaction to obtain at least one segment of question-and-answer voice; and instructing the application under test to store at least one segment of question-and-answer text during the question-and-answer interaction.

[0017] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, such that when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0018] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0019] Fifthly, some embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0020] The above embodiments of this disclosure have the following beneficial effects: The test log generation method of some embodiments of this disclosure, using at least one audio-to-text engine, can quickly and accurately locate information containing speech semantic errors, thus achieving effective testing of the question-and-answer function of the application under test. Specifically, manual verification suffers from low testing efficiency and difficulty in locating audio errors. Based on this, the test log generation method of some embodiments of this disclosure first acquires at least one segment of question-and-answer speech and at least one segment of output question-and-answer text recorded during the question-and-answer function testing of the application under test. Here, acquiring at least one segment of question-and-answer speech and at least one segment of question-and-answer text can serve as test data, facilitating the determination of the functional performance of the question-and-answer function of the application under test. Then, for each segment of the at least one segment of question-and-answer speech, the generation step is performed: First, using at least one audio-to-text engine, at least one speech recognition text corresponding to the question-and-answer speech can be accurately and efficiently determined. Here, by using at least one audio-to-text engine, different audio-to-text tools can be utilized to determine the speech content corresponding to the question-and-answer speech, ensuring the effectiveness and accuracy of the question-and-answer function test. The second step involves accurately generating speech error information based on at least one speech-recognized text and the corresponding question-and-answer text, enabling effective and accurate location of speech errors. Finally, a test log corresponding to the at least one speech error message is generated to facilitate adjustments for subsequent question-and-answer function testing of the application under test. In summary, utilizing at least one audio-to-text engine allows for comprehensive, efficient, and accurate testing of the question-and-answer functionality of the application under test. Attached Figure Description

[0021] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0022] Figure 1 This is a schematic diagram of an application scenario of a test log generation method according to some embodiments of the present disclosure; Figure 2 This is a flowchart of some embodiments of the test log generation method according to this disclosure; Figure 3 This is a flowchart of some other embodiments of the test log generation method according to this disclosure; Figure 4 This is a schematic diagram of the structure of some embodiments of the test log generation apparatus according to the present disclosure; Figure 5 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0023] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0024] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0025] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0026] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0027] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0028] Before performing any of the operations involving the collection, storage, and use of user personal information (such as question-and-answer text and voice) disclosed in this disclosure, the relevant organizations or individuals shall fulfill their obligations, including conducting personal information security impact assessments, informing personal information subjects, and obtaining prior authorization and consent from personal information subjects.

[0029] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0030] Figure 1 This is a schematic diagram of an application scenario of a test log generation method according to some embodiments of the present disclosure.

[0031] exist Figure 1In this application scenario, firstly, the electronic device 101 can acquire at least one segment of question-and-answer speech 103 and at least one segment of question-and-answer text 104 recorded during the question-and-answer function test of the application 102 under test. In this application scenario, the application 102 under test can be a question-and-answer application A. Then, for each segment of question-and-answer speech 104, the electronic device 101 can perform the following generation steps: First, using at least one audio-to-text engine 105, determine at least one speech recognition text corresponding to the question-and-answer speech. In this application scenario, for question-and-answer speech A, at least one audio-to-text engine 105 can generate at least one speech recognition text 106. At least one audio-to-text engine 105 can include: audio-to-text engine A and audio-to-text engine B. At least one speech recognition text 106 can include: speech recognition text A output by audio-to-text engine A and speech recognition text B output by audio-to-text engine B. Second, based on the at least one speech recognition text and the question-and-answer text corresponding to the question-and-answer speech, generate speech error information. In this application scenario, a voice error message A is generated based on at least one speech recognition text 106 and the question-and-answer text A corresponding to the question-and-answer speech A. Finally, the electronic device 101 can generate a test log 108 corresponding to the obtained at least one voice error message 107.

[0032] It should be noted that the aforementioned electronic device 101 can be either hardware or software. When the electronic device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the electronic device is software, it can be installed in the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0033] It should be understood that Figure 1 The number of electronic devices shown is merely illustrative. Any number of electronic devices can be used depending on the implementation requirements.

[0034] Continue to refer to Figure 2 The diagram illustrates a flow 200 of some embodiments of a test log generation method according to the present disclosure. The test log generation method includes the following steps: Step 201: Obtain at least one segment of question-and-answer audio and at least one segment of question-and-answer text recorded during the question-and-answer function test of the application under test.

[0035] In some embodiments, the execution body of the above-described test log generation method (e.g.) Figure 1The electronic device 101 shown can acquire at least one segment of recorded question-and-answer audio and at least one segment of output question-and-answer text during the question-and-answer function test of the application under test via wired or wireless connection. The application under test can be an application (APP) undergoing question-and-answer function testing. Question-and-answer function testing can test the accuracy of the application's voice output in question-and-answer interaction, and also test the logical coherence between various question-and-answer texts or audio clips. That is, it tests the consistency between the output text content and the audio playback content during question-and-answer interaction, and the content coherence between various audio or text segments. Question-and-answer interaction can be a question-and-answer interaction between the user of the application under test and the AI ​​model built into the application under test. At least one segment of recorded audio can be the audio recording of the response played by the application under test during the question-and-answer interaction. Here, at least one segment of recorded audio can be one audio clip or multiple audio clips. The number of audio clips varies depending on the specific scenario. In practice, the audio playback duration is longer for applications under test that output more text content. For efficiency and file size considerations, the server will split the complete text into multiple smaller text segments and convert each segment into a PCM audio file in parallel. Therefore, the application under test, while outputting a large amount of text, will split the output text into at least one question-and-answer text segment. Thus, when the length of the text output by the application under test exceeds the target number, the number of corresponding question-and-answer text segments and the number of corresponding question-and-answer audio segments will both exceed one segment. When the length of the text output by the application under test is less than the target number, the number of corresponding question-and-answer text segments and the number of corresponding question-and-answer audio segments will both be one segment. At least one question-and-answer text segment can be the split question-and-answer text segments output by the AI ​​model corresponding to the application under test. There is a one-to-one correspondence between the question-and-answer audio segments and the question-and-answer text segments. The total text corresponding to the combination of at least one question-and-answer text segment can be the question-and-answer output result of the AI ​​model. The total audio corresponding to the combination of at least one question-and-answer audio segment is the question-and-answer playback result of the AI ​​model.

[0036] It should be noted that at least one piece of question-and-answer text can be stored on the server by the application under test during the question-and-answer interaction process.

[0037] In some optional implementations of certain embodiments, before step 201, the steps further include: The first step, in response to the access to the device corresponding to the application under test, instructs the device to initialize the built-in recording application, thus obtaining an initialized application. The device corresponding to the application under test can be an electronic device on which the application under test is installed. In practice, the device corresponding to the application under test can be a mobile device. That is, the built-in recording application can be an application built into the execution entity for voice recording. The initialization here may include, but is not limited to, at least one of the following: setting the sampling rate, setting the number of channels, setting the audio format, and setting the storage path.

[0038] The second step is to send the Q&A function test command to the device corresponding to the application under test to enable the Q&A function of the application. The Q&A function test command can be an automated command instructing the device to activate the Q&A function for testing. Here, the application whose TTS (Text To Speech) function needs to be tested is opened via the Q&A function test command, and the user is navigated to the interface that triggers the voice playback.

[0039] The third step involves conducting a question-and-answer interaction with the application under test based on simulated question-and-answer commands. During this interaction, the application is initialized to record audio, resulting in at least one audio recording of the question-and-answer session. These simulated commands can be instructions that instruct the application under test to perform question-and-answer processing based on simulated user interactions. These simulated user interactions can include clicking buttons and inputting commands.

[0040] The fourth step is to instruct the application under test to store at least one piece of question and answer text during the question and answer interaction process.

[0041] It should be noted that after the Q&A interaction ends, the executing entity can monitor for 10 seconds (configurable duration) if the content of the application page under test does not change. If so, it will call the built-in recording application to stop recording and proceed with the testing phase.

[0042] Step 202: For each segment of the question-and-answer audio recording mentioned above, perform the generation step: Step 2021: Using at least one audio-to-text engine, determine at least one speech-recognized text corresponding to the above question-and-answer speech.

[0043] In some embodiments, the aforementioned executing entity may utilize at least one audio-to-text engine to determine at least one speech-recognized text corresponding to the aforementioned question-and-answer speech. The audio-to-text engine may be an engine that supports converting audio content into text content. The conversion methods of the various audio-to-text engines within the at least one audio-to-text engine differ. In practice, the neural network models used by the various audio-to-text engines have different model architectures. Therefore, the audio-to-text performance of the various audio-to-text engines within the at least one audio-to-text engine differs. Each audio-to-text engine may be a commercially available engine or an internally developed engine. There is a one-to-one correspondence between the audio-to-text engines within the at least one audio-to-text engine and the speech-recognized text within the at least one speech-recognized text. The speech-recognized text may be the result of the speech-to-text engine converting the question-and-answer speech into text form.

[0044] As an example, the aforementioned entity can directly input the question-and-answer speech into at least one audio-to-text engine to obtain at least one speech-recognized text.

[0045] Step 2022: Generate speech error information based on at least one speech recognition text and the question-and-answer text corresponding to the above question-and-answer speech.

[0046] In some embodiments, the executing entity can generate speech error information based on the at least one speech-recognized text and the corresponding question-and-answer text. The speech error information can be information about the differences between the speech content in the question-and-answer speech and the text content in the question-and-answer text. That is, information about playback errors present in the question-and-answer speech when the question-and-answer text is correct.

[0047] As an example, firstly, the aforementioned execution entity can filter out text locations from at least one speech recognition text that have detected more than two speech errors, thus obtaining at least one text location. Then, using a multimodal large language model, it determines at least one speech segment corresponding to the at least one text location. Finally, it summarizes the at least one speech segment and the at least one error information corresponding to the at least one text location to obtain speech error information.

[0048] As another example, firstly, prompt information is generated for at least one speech recognition text and the corresponding question-and-answer text for the aforementioned question-and-answer speech. This prompt information can be generated by a multimodal large language model using prompt words to indicate speech errors based on the at least one speech recognition text and the corresponding question-and-answer text. Then, the generated prompt information is input into the multimodal large language model to obtain the speech error information.

[0049] Step 203: Generate a test log corresponding to at least one obtained voice error message.

[0050] In some embodiments, the aforementioned executing entity may generate test logs corresponding to at least one obtained voice error message. These test logs may be test records of question-and-answer functionality testing of the application under test.

[0051] As an example, the aforementioned execution entity can add at least one voice error message and the test process corresponding to the application under test to the log template to obtain the test log.

[0052] The above embodiments of this disclosure have the following beneficial effects: The test log generation method of some embodiments of this disclosure, using at least one audio-to-text engine, can quickly and accurately locate information containing speech semantic errors, thus achieving effective testing of the question-and-answer function of the application under test. Specifically, manual verification suffers from low testing efficiency and difficulty in locating audio errors. Based on this, the test log generation method of some embodiments of this disclosure first acquires at least one segment of question-and-answer speech and at least one segment of output question-and-answer text recorded during the question-and-answer function testing of the application under test. Here, acquiring at least one segment of question-and-answer speech and at least one segment of question-and-answer text can serve as test data, facilitating the determination of the functional performance of the question-and-answer function of the application under test. Then, for each segment of the at least one segment of question-and-answer speech, the generation step is performed: First, using at least one audio-to-text engine, at least one speech recognition text corresponding to the question-and-answer speech can be accurately and efficiently determined. Here, by using at least one audio-to-text engine, different audio-to-text tools can be utilized to determine the speech content corresponding to the question-and-answer speech, ensuring the effectiveness and accuracy of the question-and-answer function test. The second step involves accurately generating speech error information based on at least one speech-recognized text and the corresponding question-and-answer text, enabling effective and accurate location of speech errors. Finally, a test log corresponding to the at least one speech error message is generated to facilitate adjustments for subsequent question-and-answer function testing of the application under test. In summary, utilizing at least one audio-to-text engine allows for comprehensive, efficient, and accurate testing of the question-and-answer functionality of the application under test.

[0053] Further reference Figure 3 The diagram illustrates a flow 300 of another embodiment of the test log generation method according to this disclosure. This test log generation method includes the following steps: Step 301: Obtain at least one segment of question-and-answer audio and at least one segment of question-and-answer text recorded during the question-and-answer function test of the application under test.

[0054] Step 302: For each segment of the question-and-answer audio recording mentioned above, perform the generation step: Step 3021: For each audio-to-text engine, the audio-to-text engine is used to sequentially perform text recognition on the speech segments in the question-and-answer speech at target durations to obtain a recognition information sequence, which is used as the speech recognition text.

[0055] In some embodiments, the executing entity (e.g. Figure 1 The electronic device 101 shown can, for each audio-to-text engine, sequentially perform text recognition on speech segments in the question-and-answer speech at target durations to obtain a recognition information sequence, which serves as the speech recognition text. The target duration can be a predetermined duration. The purpose of setting the target duration is to facilitate subsequent question tracing. For example, the target duration can be 5 seconds. The duration of the audio segment corresponding to each recognition information in the recognition information sequence is 5 seconds. The recognition information can be the result of converting the speech segment into text content. The speech segment can be a local speech segment in the question-and-answer speech. That is, according to a fixed target duration, the text recognition information (i.e., recognition information) in the question-and-answer speech is recorded sequentially to obtain the recognition information sequence.

[0056] In addition, a sequence number can be recorded for each piece of recognition information to ensure temporal alignment of the text recognition processes for each audio-to-text engine. This sequence number can be used for subsequent tracing of problematic audio. That is, after locating the text discrepancies, the corresponding sequence number can be determined. Based on the corresponding sequence number, the corresponding audio segment can be found, enabling rapid tracing of problematic audio.

[0057] Step 3022: Determine the text similarity between each of the at least one speech recognition texts and the question-and-answer texts corresponding to the question-and-answer speech, and obtain at least one text similarity.

[0058] In some embodiments, the executing entity may determine the text similarity between each of the at least one speech-recognized texts and the question-and-answer text corresponding to the question-and-answer speech, thereby obtaining at least one text similarity. The text similarity characterizes the degree of similarity in the semantic content of the speech-recognized text and the question-and-answer text. The text similarity can be a value between 0 and 1. The higher the value, the more similar the semantic content of the text.

[0059] As an example, the aforementioned execution entity can determine the cosine similarity between the text vector corresponding to each speech recognition text and the text vector corresponding to the question-and-answer text, as the text similarity.

[0060] In some optional implementations of certain embodiments, the execution entity may determine the text similarity between each of the at least one speech-recognized texts and the question-and-answer text corresponding to the question-and-answer speech, including the following steps: The first step is to perform text preprocessing on the above speech recognition text to obtain preprocessed speech text. In practice, text preprocessing may include, but is not limited to, at least one of the following: punctuation removal, space removal. The preprocessed speech text can be plain text information.

[0061] The second step involves using the preprocessed speech text as the speech recognition text. A multi-dimensional weighting method is then employed to determine the text similarity between the preprocessed speech recognition text and the corresponding question-and-answer text. This multi-dimensional weighting method can involve assigning different weights to the similarity of the two texts across multiple dimensions and then summing the results. These dimensions may include: structural integrity, semantic similarity, word order accuracy, and key data consistency. In practice, the weight for structural integrity can be 0.1, the weight for semantic similarity can be 0.4, the weight for word order accuracy can be 0.4, and the weight for key data consistency can be 0.1.

[0062] As an example, firstly, the first similarity between the aforementioned speech recognition text and the corresponding question-and-answer text is determined in the structural integrity dimension. Secondly, the second similarity is determined in the semantic similarity dimension. Thirdly, the third similarity is determined in the word order accuracy dimension. Fourthly, the fourth similarity is determined in the key data consistency dimension. Then, the first similarity is multiplied by the weight corresponding to the structural integrity dimension to obtain the first multiplication result. The second similarity is multiplied by the weight corresponding to the semantic similarity dimension to obtain the second multiplication result. The third similarity is multiplied by the weight corresponding to the word order accuracy dimension to obtain the third multiplication result. The fourth similarity is multiplied by the weight corresponding to the key data consistency dimension to obtain the fourth multiplication result. Finally, the first, second, third, and fourth multiplication results are added together to obtain the text similarity score.

[0063] Step 3023: In response to the fact that at least one of the above text similarities are all lower than the first similarity, the at least one speech recognition text is compared with the question and answer text corresponding to the above question and answer speech to obtain at least one erroneous text.

[0064] In some embodiments, in response to at least one text similarity score being lower than a first similarity score, the executing entity may perform a text comparison between the at least one speech-recognized text and the question-and-answer text corresponding to the question-and-answer speech, obtaining at least one erroneous text. The first similarity score may be a pre-set similarity value. The first similarity score can characterize the degree of difference between the speech-recognized text and the question-and-answer text. That is, a score lower than the first similarity score can indicate a significant difference between the speech-recognized text and the question-and-answer text. For example, the first similarity score could be 0.7. The erroneous text may be text information with semantic differences.

[0065] As an example, the aforementioned executing entity can perform a character-by-character comparison between the at least one speech-recognized text and the corresponding question-and-answer text to determine a set of character difference location information. This character difference location information can be the location information where characters differ. It can include multiple character positions corresponding to multiple consecutive characters. Then, character difference location information with a frequency less than the target frequency is removed from the set of character difference location information to obtain the remaining set of character difference location information, which serves as at least one erroneous text.

[0066] In some optional implementations of certain embodiments, the steps further include: In response to the existence of a text similarity score higher than the second similarity score in at least one of the aforementioned text similarities, the executing entity can generate speech error information represented as empty information. The second similarity score can be a measure of the similarity between the speech recognition text and the question-and-answer text. The second similarity score is higher than the first similarity score. For example, the second similarity score could be 95%. The speech error information represented as empty information can indicate the absence of speech errors.

[0067] Step 3024: Based on the above-mentioned at least one erroneous text, determine at least one speech segment in the above-mentioned question-and-answer speech that represents the presence of erroneous information.

[0068] In some embodiments, the executing entity may determine at least one speech segment representing the presence of error information in the question-and-answer speech based on the at least one erroneous text. Each erroneous text corresponds to a specific speech segment.

[0069] As an example, firstly, the aforementioned execution entity can determine the corresponding recognition information for each erroneous text. Then, based on the recognition information, it determines the corresponding speech segment for each erroneous text.

[0070] Step 3025: Generate speech error information based on at least one of the above speech segments.

[0071] In some embodiments, the aforementioned execution entity may generate voice error information based on at least one of the aforementioned voice segments.

[0072] As an example, the aforementioned execution entity can tag at least one speech segment and the error description information corresponding to at least one error text to obtain speech error information.

[0073] Step 303: Generate a test log corresponding to at least one of the obtained voice error messages.

[0074] In some embodiments, the specific implementation of steps 301 and 303 and their resulting technical effects can be found in [reference needed]. Figure 2 Steps 201 and 203 in the corresponding embodiments will not be repeated here.

[0075] from Figure 3 It can be seen from this that, with Figure 2 Compared to the description of some corresponding embodiments, Figure 3 In some corresponding embodiments, the test log generation method process 300 performs text recognition on the speech segments in the above question-and-answer speech at target durations, thereby recording the text in segments at fixed time intervals (e.g., 5 seconds). By comparing the ASR-converted text with the original text, the time period corresponding to the abnormal information in the entire recording process can be located. The time of the abnormal information can be calculated by adding the start time to the time period, thus enabling rapid location of the specific time point when the problem occurred.

[0076] Further reference Figure 4 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a test log generation apparatus, which are similar to... Figure 2 Corresponding to the method embodiments shown, this test log generation device can be specifically applied to various electronic devices.

[0077] like Figure 4 As shown, a test log generation device 400 includes: an acquisition unit 401, an execution unit 402, and a generation unit 403. The acquisition unit 401 is configured to acquire at least one segment of question-and-answer audio and at least one segment of question-and-answer text recorded during the question-and-answer function testing of the application under test. The execution unit 402 is configured to perform the following generation steps for each segment of the at least one segment of question-and-answer audio: using at least one audio-to-text engine to determine at least one speech recognition text corresponding to the question-and-answer audio; generating speech error information based on the at least one speech recognition text and the question-and-answer text corresponding to the question-and-answer audio; and the generation unit 403 is configured to generate a test log corresponding to the at least one segment of speech error information.

[0078] In some optional implementations of some embodiments, the execution unit 402 may be further configured to: for each audio-to-text engine, use the audio-to-text engine to sequentially perform text recognition on the speech segments in the question-and-answer speech at target durations to obtain a recognition information sequence as speech recognition text.

[0079] In some optional implementations of some embodiments, the execution unit 402 may be further configured to: determine the text similarity between each of the at least one speech recognition texts and the question-and-answer text corresponding to the question-and-answer speech, and obtain at least one text similarity; in response to the at least one text similarity being lower than a first similarity, perform a text comparison between the at least one speech recognition text and the question-and-answer text corresponding to the question-and-answer speech, and obtain at least one erroneous text; determine at least one speech segment in the question-and-answer speech that represents erroneous information based on the at least one erroneous text; and generate speech error information based on the at least one speech segment.

[0080] In some optional implementations of some embodiments, the execution unit 402 may be further configured to generate speech error information representing empty information in response to the existence of a text similarity higher than the second similarity in at least one of the text similarities.

[0081] In some optional implementations of some embodiments, the execution unit 402 may be further configured to: perform text preprocessing on the speech recognition text to obtain preprocessed speech text; use the preprocessed speech text as speech recognition text, and use a multi-dimensional weight allocation method to determine the text similarity between the speech recognition text and the question-and-answer text corresponding to the question-and-answer speech.

[0082] In some optional implementations of certain embodiments, the apparatus 500 further includes: an initialization unit, a sending unit, an interaction unit, and an instruction unit (not shown in the figure). The initialization unit can be configured to: in response to accessing the device corresponding to the application under test, instruct the device to initialize the launched built-in recording application to obtain an initialized application. The sending unit can be configured to: send a question-and-answer function test command to the device corresponding to the application under test to enable the question-and-answer function of the application under test. The interaction unit can be configured to: perform question-and-answer interaction with the application under test based on the question-and-answer function interaction simulation command, and record voice during the question-and-answer interaction using the initialized application to obtain at least one segment of question-and-answer voice. The instruction unit can be configured to: instruct the application under test to store at least one segment of question-and-answer text during the question-and-answer interaction.

[0083] It is understandable that the units recorded in the test log generation device 400 are related to the reference. Figure 2 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the test log generation device 400 and the units contained therein, and will not be repeated here.

[0084] The following is for reference. Figure 5 It illustrates electronic devices suitable for implementing some embodiments of this disclosure (e.g., Figure 1 A schematic diagram of the structure of electronic device 101)500. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0085] like Figure 5 As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory 502 or a program loaded from a storage device 508 into a random access memory 503. The random access memory 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, the read-only memory 502, and the random access memory 503 are interconnected via a bus 504. An input / output interface 505 is also connected to the bus 504.

[0086] Typically, the following devices can be connected to the input / output interface 505: input devices 506 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 507 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 508 including, for example, magnetic tape, hard disk, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 5 Each box shown can represent a device or multiple devices as needed.

[0087] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a read-only memory 502. When the computer program is executed by the processing device 501, it performs the functions defined above in the methods of some embodiments of this disclosure.

[0088] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0089] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0090] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire at least one segment of question-and-answer speech and at least one segment of question-and-answer text recorded during question-and-answer function testing of the application under test; for each segment of the at least one segment of question-and-answer speech, perform the following generation steps: using at least one audio-to-text engine to determine at least one speech-recognition text corresponding to the aforementioned question-and-answer speech; generating speech error information based on the at least one speech-recognition text and the question-and-answer text corresponding to the aforementioned question-and-answer speech; and generating a test log corresponding to the at least one speech error information obtained.

[0091] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0092] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0093] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an acquisition unit, an execution unit, and a generation unit. The names of these units do not necessarily limit the unit itself; for example, an acquisition unit may be described as "a unit that acquires at least one segment of question-and-answer audio and at least one segment of question-and-answer text recorded during question-and-answer function testing of the application under test."

[0094] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0095] Some embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements any of the test log generation methods described above.

[0096] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for generating test logs, comprising: Obtain at least one segment of question-and-answer audio and at least one segment of question-and-answer text recorded during the question-and-answer function test of the application under test; For each segment of the at least one question-and-answer speech, perform the generation step: Using at least one audio-to-text engine, determine at least one speech-recognized text corresponding to the question-and-answer speech; Based on the at least one speech recognition text and the question-and-answer text corresponding to the question-and-answer speech, a speech error message is generated; Generate a test log corresponding to at least one of the obtained voice error messages.

2. The method according to claim 1, wherein, The step of using at least one audio-to-text engine to determine at least one speech-recognized text corresponding to the question-and-answer speech includes: For each audio-to-text engine, the audio-to-text engine is used to sequentially perform text recognition on the speech segments in the question-and-answer speech at target durations to obtain a recognition information sequence, which is used as the speech recognition text.

3. The method according to claim 2, wherein, The step of generating speech error information based on the at least one speech recognition text and the question-and-answer text corresponding to the question-and-answer speech includes: Determine the text similarity between each speech recognition text in the at least one speech recognition text and the question-and-answer text corresponding to the question-and-answer speech, and obtain at least one text similarity; In response to the fact that the similarity of at least one text is lower than the first similarity, the at least one speech recognition text is compared with the question and answer text corresponding to the question and answer speech to obtain at least one erroneous text; Based on the at least one erroneous text, determine at least one speech segment in the question-and-answer speech that represents the presence of erroneous information; Based on the at least one speech segment, generate speech error information.

4. The method according to claim 3, wherein, The method further includes: In response to the existence of a text similarity higher than the second similarity among the at least one text similarity, speech error information represented as empty information is generated.

5. The method according to claim 3, wherein, Determining the text similarity between each of the at least one speech recognition texts and the question-and-answer text corresponding to the question-and-answer speech includes: The speech-recognized text is preprocessed to obtain preprocessed speech-text. The preprocessed speech text is used as the speech recognition text. A multi-dimensional weight allocation method is used to determine the text similarity between the speech recognition text and the question-and-answer text corresponding to the question-and-answer speech.

6. The method according to claim 1, wherein, Before acquiring at least one segment of question-and-answer audio recorded during the question-and-answer function test of the application under test, and at least one segment of question-and-answer text output, the method further includes: In response to the access of the device corresponding to the application under test, the device is instructed to initialize the built-in recording application to obtain the initialized application; The question-and-answer function test command is sent to the device corresponding to the application under test in order to enable the question-and-answer function of the application under test. Based on the question-and-answer function interaction simulation command, the system performs question-and-answer interaction with the application under test, and records voice during the question-and-answer interaction using the initialization application to obtain at least one question-and-answer voice recording. The application under test is instructed to store at least one piece of question-and-answer text during the question-and-answer interaction.

7. A test log generation device, comprising: The acquisition unit is configured to acquire at least one segment of question-and-answer audio and at least one segment of question-and-answer text recorded during the question-and-answer function test of the application under test; The execution unit is configured to perform a generation step for each segment of the question-and-answer speech in the at least one segment of question-and-answer speech: using at least one audio-to-text engine to determine at least one speech recognition text corresponding to the question-and-answer speech; Based on the at least one speech recognition text and the question-and-answer text corresponding to the question-and-answer speech, a speech error message is generated; The generation unit is configured to generate test logs corresponding to at least one voice error message.

8. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images