NLU test method and device, electronic equipment and storage medium

Abstract

The invention provides an NLU test method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring voice information and synchronization information corresponding to the voice information; the synchronization information comprises at least one of mouth shape information, body movement information and emotion information; and according to the voice information and the synchronization information, determining whether an instruction corresponding to the voice information is valid. According to the method, the NLU test is carried out by using the voice data, the mouth shape information corresponding to the voice data, the body movement information and the emotion information, so that the comprehensiveness of the test data set is effectively improved, and the test precision is improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, specifically to an NLU testing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of automotive intelligence, in-vehicle voice interaction systems play a crucial role in enhancing the driving experience and ensuring road safety. During driving, drivers need to constantly pay attention to road conditions, and manually operating various in-vehicle devices (such as navigation, multimedia, and air conditioning) can distract them and increase the risk of traffic accidents. In-vehicle voice interaction systems allow drivers to control various vehicle functions via voice commands, eliminating the need for manual operation. This allows drivers to focus more on driving, significantly improving road safety.

[0003] In in-vehicle voice systems, Natural Language Understanding (NLU) technology is the cornerstone for accurate recognition and understanding of voice commands. When a driver issues a voice command, the in-vehicle voice system first converts the voice signal into text information using speech recognition technology. However, this text information is often just a literal expression, and the computer cannot directly understand its true meaning. This is where NLU technology comes into play. It performs in-depth analysis and processing of the text information, including lexical analysis, syntactic analysis, and semantic analysis, to accurately understand the driver's intent. Therefore, only when NLU technology can accurately understand the driver's intent can the in-vehicle voice system respond correctly and provide the appropriate service. If NLU technology cannot accurately understand the voice command, it will lead to system errors or failure to respond, causing driver confusion and dissatisfaction, and reducing the practicality and reliability of the in-vehicle voice system. Therefore, before applying NLU technology to new products or scenarios, thorough testing is necessary to verify its feasibility.

[0004] However, existing NLU testing methods have low testing accuracy, which leads to the system's inability to accurately identify and respond to certain instructions, and even makes them unsuitable for new products. Summary of the Invention

[0005] This invention provides an NLU testing method, apparatus, electronic device, and storage medium to address the shortcomings of low NLU testing accuracy in the prior art.

[0006] This invention provides an NLU testing method, comprising:

[0007] Acquire voice information and corresponding synchronization information; the synchronization information includes at least one of: lip movement information, body movement information, and emotional information.

[0008] Based on the voice information and synchronization information, determine whether the instruction corresponding to the voice information is valid.

[0009] According to the NLU testing method provided by the present invention, determining whether the instruction corresponding to the voice information is valid based on the voice information and synchronization information includes:

[0010] Based on the voice information, determine whether the instruction corresponding to the voice information is valid; if invalid, further determine whether the instruction corresponding to the voice information is valid based on the synchronization information.

[0011] According to the NLU testing method provided by the present invention, determining whether the instruction corresponding to the voice information is valid based on the voice information includes:

[0012] The preprocessed speech information is input into the NLU model to obtain semantic analysis results based on ASR text;

[0013] Based on the semantic analysis results, key semantic information is extracted, including operation object information and operation action information.

[0014] The extracted operation object information is matched with a preset set of effective operation objects to determine whether there are any matching operation objects.

[0015] The extracted operation action information is matched with a preset set of effective operation actions to determine whether there is an operation action matching item;

[0016] Check whether the semantic analysis results contain content that violates the preset taboo rules;

[0017] If there is a matching item for the operation object and a matching item for the operation action, and no content that violates the taboo rule is detected, then the instruction corresponding to the voice information is determined to be valid; otherwise, it is invalid.

[0018] According to the NLU testing method provided by the present invention, further determining whether the instruction corresponding to the voice information is valid based on the synchronization information includes:

[0019] The synchronization information is input into its corresponding analysis model to obtain at least one of the following: lip movement analysis result, body movement analysis result, and emotion analysis result.

[0020] Based on the semantic analysis results, and at least one of the lip-reading analysis results, body movement analysis results, and emotion analysis results, determine whether the instruction corresponding to the voice information is valid.

[0021] According to the NLU testing method provided by the present invention, the lip-reading analysis result is the lip-reading matching degree with speech; the body movement analysis result is the body movement matching degree with semantic tendency; and the emotion analysis result is the emotion matching degree with semantics.

[0022] According to the NLU testing method provided by the present invention, determining whether the instruction corresponding to the speech information is valid based on the semantic analysis result, and at least one of the lip-reading analysis result, body movement analysis result, and emotion analysis result includes:

[0023] Determine the weights for semantic analysis, lip-reading analysis, body language analysis, and emotion analysis.

[0024] Based on each weight and the analysis results corresponding to each weight, the comprehensive score corresponding to the voice information is determined;

[0025] If the overall score is greater than a certain preset threshold, the instruction corresponding to the voice information is valid.

[0026] According to the NLU testing method provided by the present invention, the speech information includes: at least two contextual speech information;

[0027] At least two contextual speech information are preprocessed to obtain at least two preprocessed contextual speech information;

[0028] The preprocessed at least two contextual speech information are respectively input into the NLU model to obtain at least two contextual speech analysis results.

[0029] Calculate the cosine similarity between the vectors corresponding to the at least two context speech analysis results, and determine the final semantic analysis result corresponding to the at least two context speech information based on the cosine similarity.

[0030] The present invention also provides a testing apparatus for NLU, comprising:

[0031] An acquisition unit is used to acquire voice information and corresponding synchronization information; the synchronization information includes at least one of: lip-sync information, body movement information, and emotion information.

[0032] The determining unit is used to determine whether the instruction corresponding to the voice information is valid based on the voice information and the synchronization information.

[0033] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the program, implements the NLU testing method as described in any of the preceding claims.

[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements any of the NLU testing methods described above.

[0035] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the NLU testing method as described in any of the preceding claims.

[0036] The NLU testing method, apparatus, electronic device, and storage medium provided by this invention determine whether the instruction corresponding to the voice information is valid by utilizing voice information and synchronization information, thereby reducing the probability of missed and false judgments and avoiding the misjudgment of valid instructions as invalid or invalid instructions as valid. The introduction of synchronization information can significantly reduce such errors and improve the accuracy of the test. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating the NLU testing method provided by the present invention;

[0038] Figure 2 A schematic diagram of the NLU testing device provided by this invention;

[0039] Figure 3 A schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0041] Existing NLU testing methods rely solely on the voice information itself (such as text content) to determine the validity of commands, neglecting the fact that when a driver expresses their intention, their voice is often simultaneously conveyed with lip movements, body language, and emotions. This synchronous information is a crucial supplement to the intention. For example, when a driver says "turn on the air conditioning," if the lip movements match the pronunciation of "air conditioning," the gesture points to the air conditioning control panel, and the tone is calm (without interfering emotions), the authenticity of the command can be confirmed. However, if the voice says "turn off navigation," but the lip movements show the actual pronunciation as "turn on navigation," and the driver is emotionally agitated (possibly due to accidental device activation), the voice command can be determined to be invalid (possibly due to mispronunciation or interference). To obtain NLU test results more accurately, this invention utilizes voice information and its corresponding synchronous information to jointly determine whether the command corresponding to the voice information is valid.

[0042] Figure 1 The flowchart of the NLU testing method provided by this invention is as follows: Figure 1 The method includes the following schemes:

[0043] Step 101: Obtain the voice information and the corresponding synchronization information; the synchronization information includes at least one of lip-sync information, body movement information, and emotional information.

[0044] Specifically, a large amount of speech data covering different scenarios and different speakers is collected. At the same time, lip-sync information (obtained by capturing facial images through a high-definition camera), body movement information (collected using motion capture devices or depth cameras), and emotion information (extracted from ASR audio by features such as intonation and speech rate, and determined by combining facial expression recognition results) are collected.

[0045] The NL testing method provided by this invention, compared with traditional methods that rely solely on speech information, effectively eliminates speech recognition errors (such as text errors caused by accents or noise), false triggers (such as idle chatter or external interference), or ambiguous commands (such as semantic ambiguity) by introducing synchronous information such as lip movements, body language, and emotions. This allows for a more accurate distinction between valid and invalid commands. The collected data undergoes preprocessing, including noise reduction of speech data, alignment and cropping of lip-sync images, standardization of body language data, and quantification of emotional information, ensuring data quality and consistency.

[0046] Step 102: Determine whether the instruction corresponding to the voice information is valid based on the voice information and synchronization information.

[0047] Specifically, the voice information is analyzed in conjunction with the aforementioned synchronization information to determine whether the instruction corresponding to the voice information is valid.

[0048] The NLU testing method provided by this invention determines whether the instruction corresponding to the voice information is valid by utilizing voice information and synchronization information, thereby reducing the probability of missed and false judgments and avoiding the misjudgment of valid instructions as invalid (such as voice text being blurred due to noise) or invalid instructions as valid (such as children's jokes in a car). The introduction of synchronization information can significantly reduce such errors and improve the accuracy of the test.

[0049] Preferably, the following details how to determine the validity of a command corresponding to voice information based on voice information and synchronization information, specifically including the following solutions:

[0050] First, based on the voice information, determine whether the command corresponding to the voice information is valid; if the command corresponding to the voice information is invalid, then based on the synchronization information, further determine whether the command corresponding to the voice information is valid.

[0051] Specifically, the method provided by this invention determines the validity of a command corresponding to voice information through two judgments. First, a preliminary judgment on the validity of the command corresponding to the voice information is made solely based on the voice information (such as the text content after voice conversion, grammatical structure, etc.). If the preliminary judgment result is "invalid," then synchronous information (at least one of lip-reading information, body language information, and emotional information) is introduced, and the command is re-analyzed in combination with this synchronous information to finally determine whether it is valid. For example, if a driver brakes suddenly while driving and issues a voice message, this voice message may be misinterpreted as a command. If this command is executed, it will cause danger. Therefore, it is necessary to collect body language information to comprehensively judge whether this voice message represents the driver's true intention. Ultimately, after judgment, the voice message issued by the driver is not a valid command. As another example, if a driver and other people in the car chat, saying, "Go to Window of the World?", this is not actually a command, but just a chat. To avoid this chat content being mistaken for a valid command, it is necessary to simultaneously collect facial expressions, tone of voice, speech rate, etc., to comprehensively judge whether the chat content is a valid command. After final assessment, the chat content was determined to be an invalid command.

[0052] The NLU testing method provided by this invention first performs a preliminary judgment based on voice information, which can quickly filter out obviously valid commands and reduce unnecessary synchronous information analysis. Only commands initially determined to be invalid are subject to secondary verification, thereby improving testing efficiency while ensuring testing accuracy. Furthermore, since voice information is often interfered with during driving (such as in-vehicle noise, driver accents, etc.), this application, through a secondary verification mechanism, can better cope with these interference factors, making the NLU test results more consistent with actual use scenarios, thereby improving the reliability of in-vehicle voice interaction systems in practical applications.

[0053] Preferably, the following describes how to determine the validity of a command corresponding to voice information based on the voice information, specifically including the following solutions:

[0054] The preprocessed speech information is input into the NLU model to obtain semantic analysis results based on ASR text. Based on the semantic analysis results, key semantic information is extracted, including operation object information and operation action information. The extracted operation object information is matched against a preset set of operation objects to determine if any operation object matches. The extracted operation action information is matched against a preset set of operation actions to determine if any operation action matches. The semantic analysis results are checked for content that violates preset taboo rules. If both operation object and operation action matches exist and no content violating taboo rules is detected, the instruction corresponding to the speech information is deemed valid; otherwise, it is invalid.

[0055] Specifically, in the field of intelligent driving, the preset set of operational objects is a collection of various entities or functional modules predefined by the system and controllable by voice commands. For example, it may include vehicle control-related entities, such as semantic analysis results for "windows," "doors," "air conditioning," "windshield wipers," "steering wheel," "brakes," and "accelerator"; it may also cover navigation and information interaction functional modules, such as semantic analysis results for "navigation system," "music player," "telephone module," "instrument panel display," and "voice assistant"; furthermore, driving mode-related options, such as semantic analysis results for "sport mode," "economy mode," "automatic parking mode," and "cruise mode," can also be included in this set. These operational objects are entities or functional modules that the intelligent driving system can recognize and perform corresponding operations on.

[0056] The preset set of operation actions is a collection of various executable operations that can be triggered by voice commands and are pre-defined by the system. These actions must be adapted to vehicle functions and driving scenarios. Specifically, this includes vehicle control actions, such as semantic analysis results for "accelerate," "decelerate," "brake," "steering," "stop," "start," and "shift gears"; in-vehicle device control actions, such as semantic analysis results for "turn on the air conditioner," "close the windows," "adjust the volume," and "switch the navigation route"; and driving mode-related actions, such as semantic analysis results for "activate autonomous driving," "switch to manual driving," "activate cruise control," and "deactivate cruise control."

[0057] The difference between the set of operational actions and the set of operational objects lies in the following: the latter focuses on the semantic analysis result "the entity or functional module being operated on," which is the target to which the action is directed, such as semantic analysis results for "steering wheel," "brake pedal," "air conditioning system," and "navigation module," etc.; the former focuses on the semantic analysis result "the executable action itself," which is the behavior acting on the operational object, such as semantic analysis results for "turning" (corresponding to the steering wheel), "pressing down" (corresponding to the brake pedal), and "turning on" (corresponding to the air conditioning system), etc. Simply put, the operational object is the semantic analysis result "who" or "what," while the operational action is the semantic analysis result "what to do."

[0058] This invention first preprocesses the speech information (e.g., removing noise, correcting recognition errors), then inputs the preprocessed speech information into an NLU model to obtain semantic analysis results based on Automatic Speech Recognition (ASR) text. Next, key semantic information is extracted from the semantic analysis results, including operation object information and operation action information. Then, the extracted operation object information is matched against a preset set of operation objects to determine if a match exists. The extracted operation action information is then matched against a preset set of operation actions to determine if a match exists. Finally, the semantic analysis results are checked for content that violates preset taboo rules (e.g., dangerous instructions that may affect driving safety, such as "watching videos while driving"). If both operation object and operation action matches exist, and no content violating taboo rules is detected, then the instruction corresponding to the speech information is deemed valid; otherwise, if no operation object or operation action match exists, or if content violating taboo rules is detected, then it is deemed invalid. For example, when the speech information is "turn on the air conditioner," the preprocessed information is input into the NLU model to obtain the semantic analysis results, and the extracted operation object information is "air conditioner," and the operation action information is "turn on." The preset set of operation objects includes "air conditioning", the set of operation actions includes "turn on", and the instruction does not violate the taboo rules, so the instruction is deemed valid. However, if the voice message is "open the car window to throw away trash", although the extracted operation object "car window" and operation action "open" match, "throw away trash" violates the taboo rules of safe driving, so the instruction is deemed invalid.

[0059] The method provided by this invention converts voice information into analyzable semantic results, extracts key operation objects and action information, and matches it with a pre-set set of valid commands to quickly determine whether a command is within the system's responsive range. Simultaneously, taboo rule checks further filter from a safety and compliance perspective, eliminating commands that may pose risks or violate system specifications, reducing misjudgments caused by semantic ambiguity or vagueness. Compared to traditional methods that rely solely on literal meaning, this method more accurately identifies truly valid commands that meet system requirements. Furthermore, taboo rule checks specifically target commands that may affect driving safety, effectively filtering out dangerous commands and further improving driving safety from the voice interaction level.

[0060] Preferably, the following describes how to further determine whether the command corresponding to the voice information is valid based on the synchronization information, specifically including the following scheme:

[0061] The synchronization information is input into its corresponding analysis model, yielding at least one of the following: lip-reading analysis result, body movement analysis result, and emotion analysis result. Based on the semantic analysis result and at least one of the following three results, the validity of the instruction corresponding to the voice information is determined. Specifically, the lip-reading analysis result represents the degree of matching between lip shape and voice; the body movement analysis result represents the degree of matching between body movement and semantic tendency; and the emotion analysis result represents the degree of matching between emotion and semantics.

[0062] Specifically, this includes determining the weights corresponding to semantic analysis, lip-reading analysis, body language analysis, and emotion analysis; determining the comprehensive score corresponding to the voice information based on each weight and the analysis result corresponding to each weight; and validating the instruction corresponding to the voice information if the comprehensive score is greater than a certain preset threshold.

[0063] Specifically, since a driver's true intentions are often expressed not only through voice but also indirectly through lip movements, gestures, and emotions, this invention, through comprehensive judgment of multi-dimensional information, can more fully filter out valid commands, further improving the accuracy of NUL testing. Furthermore, it reduces erroneous responses to invalid commands and missed responses to valid commands, thereby enhancing the reliability of the in-vehicle voice interaction system and better meeting the needs of driving scenarios.

[0064] The following explains how to calculate the overall score:

[0065] Let the weight of the analysis result of the speech data be (w1), the weight of the analysis result of the lip movement information be (w2), the weight of the analysis result of the body movement information be (w3), and the weight of the analysis result of the emotion information be (w4), and (w1+w2+w3+w4=1).

[0066] Let the analysis result of the speech data be (S1) (range 0-1, 1 indicates the highest semantic match), the analysis result of the lip-reading information be (S2) (range 0-1, 1 indicates a perfect match between lip-reading and speech), the analysis result of the body language information be (S3) (range 0-1, 1 indicates a perfect match between body language and semantic tendency), and the analysis result of the emotion information be (S4) (range 0-1, 1 indicates a perfect match between emotion and semantic expression). Then the comprehensive score is (S=w1*S1+w2*S2+w3*S3+w4*S4).

[0067] The following examples illustrate the application of the comprehensive score calculation model in specific scenarios:

[0068] Example: Everyday conversational scenarios (voice messages dominate)

[0069] Weighting: (w1=0.4) (voice data), (w2=0.2) (lip movement information), (w3=0.2) (body movements), (w4=0.2) (emotional information)

[0070] Scores for each dimension:

[0071] The following parameters are considered: speech data matching degree (S1=0.9) (accurate ASR recognition and clear semantic understanding); lip shape and speech matching degree (S2=0.8) (lip shape changes are basically synchronized with pronunciation); body movement and semantic tendency matching degree (S3=0.7) (gestures assist in expression, with moderate semantic relevance); emotion and semantic matching degree (S4=0.8) (calm tone, consistent with the emotions of daily conversation). Therefore, the overall score S=0.4*0.9+0.2*0.8+0.2*0.7+0.2*0.8=0.36+0.16+0.14+0.16=0.82. If the threshold is set to 0.7, the instruction corresponding to the speech information is considered valid.

[0072] Preferably, when the speech information includes at least two contextual speech information, the present invention introduces how to obtain semantic analysis results, specifically including the following schemes:

[0073] First, at least two contextual speech information are preprocessed to obtain at least two preprocessed contextual speech information; second, the at least two preprocessed contextual speech information are input into the NLU model to obtain at least two contextual speech analysis results.

[0074] Calculate the cosine similarity between the vectors corresponding to the at least two context speech analysis results, and determine the final semantic analysis result corresponding to the at least two context speech information based on the cosine similarity.

[0075] Specifically, the voice information provided in this embodiment of the invention consists of at least two contextual voice information sets, which are multi-round continuous commands issued by the driver during driving. These commands typically revolve around vehicle control (such as adjusting the air conditioning, operating the navigation, controlling the windows, etc.) or driving needs (such as adjusting the route, changing music, etc.), and have temporal sequence and semantic association, such as the command set "Navigate to the company → Avoid congested sections → Depart now". This invention preprocesses each contextual voice information set separately, such as removing noise from the voice and correcting errors that occur during the speech recognition process, to obtain at least two preprocessed contextual voice information sets, providing more accurate basic data for subsequent analysis. Then, each preprocessed contextual voice information set is input into an NLU model, which performs separate semantic analysis on each command, thereby obtaining at least two contextual voice analysis results, which reflect the independent semantics of each round of commands. Finally, each obtained contextual voice analysis result is converted into a corresponding semantic vector, and the cosine similarity between these vectors is calculated. Then, combining a set threshold (e.g., ≥0.5, considering the susceptibility to noise interference in the in-vehicle environment), the final semantic analysis result corresponding to these at least two contextual speech information is determined based on the magnitude of the cosine similarity. For example, in the scenario of "navigate to the company → avoid congested sections → depart now," each round of instructions is preprocessed separately, input into the NLU model, and the analysis results for each round are obtained. After converting these results into semantic vectors, the cosine similarity is calculated. Since all three rounds of instructions revolve around the core semantic of "navigating to the company," the similarity will be higher than the threshold, and the final semantic analysis result can clearly identify the "departure" as the "company" route. However, for "turn on the air conditioner → play music," the semantic connection between the two is weaker, and the cosine similarity may be lower than the threshold. In this case, it is necessary to further verify by combining lip movements and body language and other synchronization information to determine whether it is a valid instruction for reasonable switching, and thus determine the final semantic analysis result.

[0076] The method provided by this invention obtains analysis results by inputting each contextual speech information separately into an NLU model, converts it into semantic vectors, and uses cosine similarity to quantify the degree of correlation between semantic vectors of different rounds of instructions. Higher cosine similarity indicates a more consistent semantic direction and a stronger correlation among multiple rounds of instructions; conversely, a lower similarity indicates a weaker correlation. Combined with a set threshold, it can be determined whether multiple rounds of instructions revolve around the same core intent, thereby integrating a final semantic analysis result that accurately reflects the driver's true intent, thus further improving the testing accuracy of NUL.

[0077] Figure 2 The structural block diagram of the NLU testing device provided by the present invention is as follows: Figure 2 As shown, the device includes:

[0078] The acquisition unit 201 is used to acquire voice information and corresponding synchronization information; the synchronization information includes at least one of lip-sync information, body movement information, and emotion information.

[0079] The determining unit 202 is used to determine whether the instruction corresponding to the voice information is valid based on the voice information and the synchronization information.

[0080] This invention also provides a testing device for NLU, wherein the determining unit 202 further includes:

[0081] The judgment module is used to determine whether the instruction corresponding to the voice information is valid based on the voice information; the determination module is used to further determine whether the instruction corresponding to the voice information is valid based on the synchronization information if the instruction corresponding to the voice information is invalid.

[0082] Furthermore, the judgment module provided by the present invention also includes:

[0083] The preprocessing module is used to input the preprocessed speech information into the NLU model to obtain semantic analysis results based on ASR text;

[0084] The extraction module is used to extract key semantic information based on the semantic analysis results. The key semantic information includes information about the operation object and information about the operation action.

[0085] The matching module is used to match the extracted operation object information with a preset set of operation objects to determine whether there is an operation object matching item; and to match the extracted operation action information with a preset set of operation actions to determine whether there is an operation action matching item.

[0086] The inspection module is used to check whether the semantic analysis results contain content that violates preset taboo rules;

[0087] The determination module is used to determine that the instruction corresponding to the voice information is valid if there is a matching item for the operation object and a matching item for the operation action, and no content that violates the taboo rule is detected; otherwise, it is invalid.

[0088] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logic instructions in the memory 330 to execute an NLU test method, which includes:

[0089] Acquire voice information and corresponding synchronization information; the synchronization information includes at least one of: lip movement information, body movement information, and emotional information.

[0090] Based on the voice information and synchronization information, determine whether the instruction corresponding to the voice information is valid.

[0091] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0092] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer being able to execute the NLU testing method provided by the above methods, the method comprising:

[0093] Acquire voice information and corresponding synchronization information; the synchronization information includes at least one of: lip movement information, body movement information, and emotional information.

[0094] Based on the voice information and synchronization information, determine whether the instruction corresponding to the voice information is valid.

[0095] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the NLU testing method provided by the above methods, the method comprising: acquiring speech information and synchronization information corresponding to the speech information; the synchronization information comprising at least one of lip-sync information, body movement information, and emotion information;

[0096] Based on the voice information and synchronization information, determine whether the instruction corresponding to the voice information is valid.

[0097] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0098] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An NLU testing method, characterized in that, include: Obtain the voice information and the corresponding synchronization information; The synchronization information includes at least one of the following: lip-reading information, body movement information, and emotional information; Based on the voice information and synchronization information, determine whether the instruction corresponding to the voice information is valid.

2. The NLU testing method according to claim 1, characterized in that, The step of determining whether the instruction corresponding to the voice information is valid based on the voice information and synchronization information includes: Based on the voice information, determine whether the instruction corresponding to the voice information is valid; if invalid, further determine whether the instruction corresponding to the voice information is valid based on the synchronization information.

3. The NLU testing method according to claim 2, characterized in that, The step of determining whether the instruction corresponding to the voice information is valid based on the voice information includes: The preprocessed speech information is input into the NLU model to obtain semantic analysis results based on ASR text; Based on the semantic analysis results, key semantic information is extracted, including operation object information and operation action information. The extracted operation object information is matched with a preset set of operation objects to determine whether there are any matching operation objects. The extracted operation action information is matched with a preset set of operation actions to determine whether there is an operation action matching item; Check whether the semantic analysis results contain content that violates the preset taboo rules; If there is a matching item for the operation object and a matching item for the operation action, and no content that violates the taboo rule is detected, then the instruction corresponding to the voice information is determined to be valid; otherwise, it is invalid.

4. The NLU testing method according to claim 3, characterized in that, The step of further determining whether the instruction corresponding to the voice information is valid based on the synchronization information includes: The synchronization information is input into its corresponding analysis model to obtain at least one of the following: lip movement analysis result, body movement analysis result, and emotion analysis result. Based on the semantic analysis results, and at least one of the lip-reading analysis results, body movement analysis results, and emotion analysis results, determine whether the instruction corresponding to the voice information is valid.

5. The NLU testing method according to claim 4, characterized in that, The lip-reading analysis result is the degree of matching between lip shape and speech; the body movement analysis result is the degree of matching between body movement and semantic tendency; and the emotion analysis result is the degree of matching between emotion and semantics.

6. The NLU testing method according to claim 4, characterized in that, The step of determining whether the instruction corresponding to the voice information is valid based on the semantic analysis results, as well as at least one of the lip-reading analysis results, body movement analysis results, and emotion analysis results includes: Determine the weights for semantic analysis, lip-reading analysis, body language analysis, and emotion analysis. Based on each weight and the analysis results corresponding to each weight, the comprehensive score corresponding to the voice information is determined; If the overall score is greater than a certain preset threshold, the instruction corresponding to the voice information is valid.

7. The NLU testing method according to any one of claims 3-6, characterized in that, The voice information includes: at least two contextual voice information; At least two contextual speech information are preprocessed to obtain at least two preprocessed contextual speech information; The preprocessed at least two contextual speech information are respectively input into the NLU model to obtain at least two contextual speech analysis results. Calculate the cosine similarity between the vectors corresponding to the at least two context speech analysis results, and determine the final semantic analysis result corresponding to the at least two context speech information based on the cosine similarity.

8. A testing device for NLU, characterized in that, include: The acquisition unit is used to acquire voice information and the synchronization information corresponding to the voice information; The synchronization information includes at least one of the following: lip-reading information, body movement information, and emotional information; The determining unit is used to determine whether the instruction corresponding to the voice information is valid based on the voice information and the synchronization information.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the NLU testing method as described in any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the NLU testing method as described in any one of claims 1 to 6.

Images