Method and device for displaying virtual image, vehicle, medium and program product

Abstract

The embodiment of the invention provides a method and equipment for displaying a virtual image, a medium and a program product. According to the method, the scene information and the voice information in the scene can be acquired, and the action of the virtual image is adjusted based on the scene information and the voice information, so that the virtual image is controlled in a diversified and real-time manner, and more real and more diversified actions of the virtual image are displayed for a user. Therefore, through the method provided by the invention, the sense of reality of the virtual image can be improved from two aspects of semantics of the scene and the voice information, so that the man-machine interaction experience can be improved. In addition, according to the method provided by the invention, the action of the virtual image can be automatically controlled, so that more intelligent virtual image experience can be provided.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and more particularly to a method, device, vehicle, medium, and program product for displaying a virtual image. Background Art

[0002] With the advancement of technology, more and more human-computer interaction methods have emerged. As a human-computer interaction method that can interact with people efficiently, avatars have attracted attention and popularity from all parties. Avatars are images that can show human-like expressions and actions in an anthropomorphic form. They can have human body shapes or non-human body shapes, such as animal body shapes. Since the interaction with avatars can be presented in a face-to-face form, avatars provide people with a better interactive experience.

[0003] Virtual images can be used in a variety of fields. For example, virtual image-related software can be installed in a vehicle to display the virtual image on a screen in the cockpit. The driver or passenger can interact with the virtual image to achieve functions such as vehicle control. Summary of the invention

[0004] According to an exemplary embodiment of the present disclosure, a technical solution for displaying a virtual image is provided, which can be used to adjust the action of the virtual image according to scene information and voice information in the scene, and control the action of the virtual image in a diversified manner, thereby improving the realism of the virtual image and thus improving the human-computer interaction experience. In addition, the action of the virtual image can be automatically controlled to provide a more intelligent virtual image experience.

[0005] In a first aspect of the present disclosure, a method for displaying a virtual image is provided, which may include obtaining scene information and voice information. The method also includes determining the action of the virtual image based on the scene information and the voice information in the scene. And the method also includes displaying the action of the virtual image. Implementing the method provided in the first aspect can improve the realism of the virtual image, thereby improving the human-computer interaction experience. In addition, the action of the virtual image can be automatically controlled to provide a more intelligent virtual image experience.

[0006] In a second aspect of the present disclosure, an electronic device for displaying a virtual image is provided. The electronic device includes a processor and a memory coupled to the processor, the memory having instructions stored therein, and when the instructions are executed by the electronic device, the electronic device executes the method provided by the first aspect of the present disclosure. The electronic device provided by the second aspect can improve the realism of the virtual image, thereby improving the human-computer interaction experience. In addition, the actions of the virtual image can be automatically controlled to provide a more intelligent virtual image experience.

[0007] In a third aspect of the present disclosure, a vehicle is provided, the vehicle comprising the electronic device provided in the second aspect. The vehicle provided by the third aspect of the present disclosure can improve the realism of the virtual image, thereby improving the human-computer interaction experience. In addition, the actions of the virtual image can be automatically controlled to provide a more intelligent virtual image experience.

[0008] In the fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, and when the computer-executable instructions are executed, the computer executes the method provided in the first aspect. The computer-readable storage medium provided in the fourth aspect of the present disclosure can improve the realism of the virtual image, thereby improving the human-computer interaction experience. In addition, the actions of the virtual image can be automatically controlled to provide a more intelligent virtual image experience.

[0009] In a fifth aspect of the present disclosure, there is provided a computer program product, which is tangibly stored on a computer-readable medium and comprises computer-executable instructions which, when executed, cause a computer to perform the method according to the first aspect of the present disclosure.

[0010] It can be seen from the above description that according to the solutions of the various embodiments of the present disclosure, the efficiency of configuring function flags can be improved. It should be understood that the invention summary is provided to introduce the selection of concepts in a simplified form, which will be further described in the specific implementation below. The invention summary is not intended to identify the key features or main features of the present disclosure, nor is it intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0012] Figure 1 A schematic diagram showing application scenarios according to some embodiments of the present disclosure is shown.

[0013] Figure 2 An example timing diagram according to some embodiments of the present disclosure is shown.

[0014] Figure 3 A schematic diagram showing action attributes of various dimensions according to some embodiments of the present disclosure is shown.

[0015] Figure 4 A schematic diagram of multi-entity sentiment analysis according to some embodiments of the present disclosure is shown.

[0016] Figure 5An example architecture diagram for implementing attribute calculation when the voice information is first voice information according to some embodiments of the present disclosure is shown.

[0017] Figure 6 An example architecture diagram for implementing attribute calculation when the voice information is second voice information according to some embodiments of the present disclosure is shown.

[0018] Figure 7 A schematic diagram showing the relationship between the weighted weight of each word segment and the confidence of the emotion corresponding to each word segment according to some embodiments of the present disclosure.

[0019] Figure 8 An example architecture diagram for implementing attribute calculation when voice information includes first voice information and second voice information according to some embodiments of the present disclosure is shown.

[0020] Fig. 9 A flowchart showing a method for displaying a virtual image according to some embodiments of the present disclosure; and

[0021] Fig.10 A schematic structural diagram of a device that can be used to implement an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0022] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0023] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0024] Although virtual images have been widely used in various fields, the current virtual images are not highly anthropomorphic, which makes it difficult to give people a high sense of reality, but reduces the experience of human-computer interaction. For example, for in-vehicle virtual images, if their anthropomorphism is not high, the driver or passengers may lose interest in interacting with them.

[0025] The present disclosure proposes a method, device, medium and program product for displaying a virtual image. The method for displaying a virtual image can obtain scene information and voice information, and adjust the action of the virtual image based on the scene information and the voice information in the scene, so as to control the virtual image in a diversified and real-time manner, and display more realistic and diverse actions of the virtual image to the user. Therefore, through the method provided by the present disclosure, the realism of the virtual image can be improved from the semantics of the scene and voice information, thereby improving the human-computer interaction experience. In addition, the method provided by the present disclosure can automatically control the action of the virtual image, so as to provide a more intelligent virtual image experience.

[0026] Figure 1 A schematic diagram of application scenarios according to some embodiments of the present disclosure is shown. Figure 1 There is no restriction on the application scenarios in Figure 1 The application scenario in the can be the cockpit of a vehicle, or Figure 1 The application scenarios can also be rooms, offices, etc. Figure 1 The system includes a virtual image 102, a user 104 interacting with the virtual image 102, a processor 106, a database 108, and a scene information module 110. The processor 106 is used to determine the action of the virtual image based on the information in the database 108 and the scene information module 110. The scene information module 110 is used to obtain and / or store scene information. Figure 2 right Figure 1 For example, it should be noted that the timing Figure 2 The timing sequence given in the figure is for exemplary purposes only, and the timing sequence of different steps can be changed according to actual conditions.

[0027] In some embodiments, the scene information may include at least one of time information, road condition information, vehicle operation information, or signal light information. It is understandable that the scene information module 110 may be different depending on the scene information. For example, when the scene information is only time information, the scene information module 110 may be a clock module for local calculation or networking to obtain the current time. For example, when Figure 1 When the application scenario is a cockpit, and the scene information includes road condition information and signal light information, the scene information module 110 may be a networking module that obtains the road condition information and signal light information through the Internet. At this time, the road condition information and signal light information may refer to the road condition information and signal light information near the vehicle. Alternatively, when the scene information includes time information, road condition information, and signal light information at the same time, the scene information module 110 may include a clock module and a networking module at the same time. The processor 106 may request 202 scene information from the scene information module 110, and the database sends 204 the current scene information to the processor 106.

[0028] In some embodiments, the voice information may include user voice information input by user 104 (also referred to as first voice information) and / or avatar voice information output by avatar 102 (also referred to as second voice information). For example, when the application scenario is a cockpit, processor 106 may obtain 206 user voice information through audio input module 114, and / or query database 108 or a database for storing logs of avatar 102 (not shown). Figure 1 ) to obtain the avatar voice information. It is understandable that the avatar voice information can be triggered based on the function control of the application scenario. For example, when the user 104 turns on the air conditioner, humidifier and other function modules 116 in the application scenario, the avatar 102 can output the corresponding avatar voice information.

[0029] The database 108 can be used to store different actions of the virtual image 102. The processor 106 can be used to determine the action of the virtual image 102 from the database 108 based on the scene information and the voice information for display. In some embodiments, the processor 106 can first identify the semantics of the voice information, and then query the predetermined action table in the database 108 based on the semantics and the scene information to determine the action of the virtual image. For example, when the processor 106 recognizes that the semantics of the voice information is "turn on the air conditioner" and the scene information is the time information "9AM", the action "press the remote control" can be determined as the action of the virtual image 102. For another example, when the processor 106 recognizes that the semantics of the voice information is "turn on the air conditioner" and the scene information is the time information "9PM", the action "press the remote control after stretching" can be determined as the action of the virtual image 102. Therefore, the method provided by the present disclosure can control the action of the virtual image in a diversified manner based on the voice information and the scene information at the same time, thereby improving the realism of the virtual image. In addition, the method provided by the present disclosure can automatically control the action of the virtual image, so as to provide a more intelligent virtual image experience.

[0030] In some embodiments, the database 108 can be used to store different types of action templates of the virtual image 102, and the action template can be an action video / action animation including a series of frames, and the action attributes of different dimensions of the action template can be adjusted. The processor 106 can be used to determine the action template in the database 108, that is, to determine the current action type of the virtual image 102, and further set and / or adjust the action attribute of at least one dimension of the action template based on the scene information and the voice information, and then determine the action of the virtual image 102 for display according to the action template and the determined corresponding action attribute. For example, the dimension of the action attribute can include at least two of the action motion amplitude, action frequency, or action strength. In some examples, determining the action attributes of different dimensions can be implemented based on determining the action parameters corresponding to the action attributes of different dimensions. For ease of understanding, when the action attributes are involved below, the action parameters will be used as an example for explanation.

[0031] In some embodiments, the processor 106 may query 208 an action template from the database 108 based on the current execution function of the application scenario (e.g., cockpit), and the database 108 sends 210 the queried action template to the processor 106, and then query the predetermined parameter table in the database 108 based on the scene information and the voice information, and the database sends the queried action parameters to the processor 106 to determine the action parameters of the virtual image. For example, when the current execution function of the application scenario is "music playback", the processor 106 may determine that the action template is "dancing", and then query the predetermined parameter table based on the scene information and the voice information to obtain the action parameters of "dancing", for example, the parameter values ​​of at least two of the action movement amplitude, action frequency, or action strength may be queried, and the action of the virtual image is determined based on the determined action parameters and the action template for display. Figure 3 , three different action movement amplitudes, action frequencies, and action strengths are shown in the schematic diagram, for the action movement amplitude, it can be shown as different ranges of body movement, for the action frequency, it can be shown as the length of time for performing a whole set of actions, and for the action strength, it can be shown as the change in the number of frames of some actions. It can be understood that if the processor 106 only determines a part of all action parameters corresponding to the action template, the remaining undetermined other part of the action parameters can adopt the default parameter values ​​carried by the action template. For example, when the action parameters of "dancing" include action movement amplitude, action frequency, and action strength, if the processor 106 only determines the action movement amplitude, the action frequency and action strength can adopt the default parameters carried by the action template.

[0032] Alternatively, the action template can also be determined by the following method. The action configuration request sent by the user is responded to through semantic configuration, the virtual image corresponding to the action configuration request is determined and the to-be-processed text that the virtual image needs to process is obtained, the to-be-processed text is parsed through the semantic tag annotation model and the preset semantic tag rules to obtain each semantic tag, the action tag corresponding to each semantic tag is obtained, and each action tag is composed into an action tag list, the arrangement order of each action tag in the action tag list is determined according to the priority of the semantic tag corresponding to each action tag, the action tag processing algorithm corresponding to the virtual image is called to filter out each target action tag in the action tag list, and the actions corresponding to each target action tag are inserted into the multimodal output of the virtual image in turn to determine the action template of the virtual image in real time.

[0033] In some embodiments, the actions described in the present disclosure may include body actions and facial expressions. Body actions may refer to actions produced by movements of various parts of the body trunk, and facial expressions may refer to actions produced by movements of the facial features or other parts of the face. Body actions, such as the above-mentioned "pressing the remote control", "dancing", etc., can be determined based on the current execution function of the application scenario as described above, while facial expressions can be determined based on the emotions in the voice information. For example, for facial expressions, the processor 106 can identify the emotions in the voice information, and determine the action template of the facial expression based on the preset expression table in the emotion query database. The process of determining the action parameters of the facial expression is similar to that of determining the action parameters of the body action, which will not be repeated.

[0034] It is understandable that the action parameters of the expression action may also include at least two dimensions of action movement amplitude, action frequency, or action strength. In addition, the action parameters of the expression action may be different from the action parameters of the body action. For example, when the action parameter of the expression action is the action movement amplitude (such as when the action template is "smile", the action movement amplitude may refer to the amplitude of the upward movement of the corners of the mouth), the action parameters of the body action may include the action frequency and the action strength. It should be understood that the above-mentioned processing and the processing described below can be performed on body actions and / or expression actions, and the specific actions described in the present disclosure should not be understood as limitations on the present disclosure. For the convenience of explanation and understanding, the following description takes the action as a body action as an example.

[0035] Therefore, even for the same action type, the present disclosure can adjust the action parameters based on different scene information and / or voice information, rather than performing at a preset fixed frequency, thereby achieving two-way emotional interaction from both scene and semantic aspects, making the action parameters change in real time, and improving the authenticity of the virtual image. For example, with scene information, such as time gradually moving from 9AM to 1PM and then to 9PM, even if the action template determined by the processor 106 is always "dancing", the action parameters of "dancing", such as at least one of the action movement amplitude, action frequency, or action strength, will also change in real time. Figure 1 The three states 1021, 1022 and 1023 of the virtual image 102 are shown, which respectively show the visual effects when the virtual image in the display module 112 performs the "dancing" action at the three time points of 9AM, 1PM and 9PM. It can be seen that the visual effect of the "dancing" action changes with time.

[0036] In some embodiments, in order to make the transition of body movements of different action types more natural when the currently executed function is switched, when the body movement is transitioned from the body movement of the action type (also referred to as the first action type) corresponding to the function executed before the switch (also referred to as the first execution function) to the body movement of the action type (also referred to as the second action type) corresponding to the execution function after the switch (also referred to as the second execution function) (for example, when it is determined that the user switches the current execution function of the cockpit), the action type used for the transition is determined according to the function before the switch and the function after the switch. For example, when the action type corresponding to the function before the switch is "sleeping" and the action type corresponding to the function after the switch is "dancing", "stretching" can be determined as the transition action type based on the function before the switch and the function after the switch to further improve the authenticity of the virtual image.

[0037] In some embodiments, if the voice information is only user voice information or avatar voice information, in addition to the above method of directly looking up the table, corresponding tables can be set for scene information and voice information respectively, and after determining the action parameter set 118 (also referred to as the first action attribute set) corresponding to the scene information and the action parameter set 120 (also referred to as the second action attribute set) corresponding to the voice information respectively, the action parameters for the avatar are calculated according to the action parameter set 118 and the action parameter set 120. For example, the processor 106 can query the action parameter set 118 from the database 108, and the database 108 sends the queried action parameter set 118 to the processor 106, and then the action parameter set 120 is queried from the database 108. It can be understood that the action parameter set 118 and the action parameter set 120 include action parameters of at least two dimensions, and the included action parameters are the same, for example, the action parameter set 118 and the action parameter set 120 can both include action movement amplitude, action frequency and action intensity. In some embodiments, the processor 106 may first identify the emotion in the voice information, and then query the action parameter set 120 from the database 108 based on the emotion to improve the authenticity of the action parameters. For example, when the emotion is "happy", the corresponding action parameters may visually make the action of the avatar more active, and when the emotion is "sad", the corresponding action parameters may visually make the action of the avatar less active.

[0038] In some embodiments, if the voice information is a virtual image voice information, the virtual image voice information can first be subjected to multi-entity emotion analysis to identify the core entity words in the virtual image voice information, that is, the segmentation words used to identify emotions in the virtual image voice information. Then, the core entity words are subjected to emotion judgment to obtain the emotion of each segmentation word and the action parameter set corresponding to the emotion of each segmentation word, and the action parameter set 120 is obtained by weighted calculation based on the action parameter set corresponding to the emotion of each segmentation word, so that the action parameter set 120 can be accurately obtained when the emotion in the virtual image voice information is not obvious or difficult to judge. In some embodiments, the same segmentation word may be judged as multiple emotions. At this time, in order to accurately obtain the action parameter set of the segmentation word, the action parameter set corresponding to each emotion can be weighted to obtain the action parameter set of the segmentation word.

[0039] For ease of understanding, refer to Figure 4 , Figure 4An example diagram of multi-entity emotion analysis of avatar voice information is shown. Among them, "ABCDEFGHI" is the text of the avatar voice information. First, the core entity words in "ABCDEFGHI" can be identified 402. It is assumed that the identified core entity words, that is, the segmentation words used to identify emotions are "AB" and "GH". Then the emotions of "AB" and "GH" are judged 404. Assuming that the emotions of "AB" are happy and calm, and the emotion of "GH" is calm, the action parameter set corresponding to "AB" can be obtained by weighted calculation 406 according to the action parameter sets corresponding to happiness and calm, respectively, and then the action parameter set corresponding to the calm emotion of "GH" and the action parameter set corresponding to "AB" are weighted 408 to obtain the action parameter set 120.

[0040] After obtaining the action parameter set 118 and the action parameter set 120, the same action parameters therein may be weighted respectively to obtain action parameters applied to the avatar. For example, when both the action parameter set 118 and the action parameter set 120 include action movement amplitude, action frequency, and action strength, the action movement amplitudes in the action parameter set 118 and the action parameter set 120 may be weighted to obtain the action movement amplitude applied to the avatar, the action frequencies in the action parameter set 118 and the action parameter set 120 may be weighted to obtain the action frequency applied to the avatar, and the action strengths in the action parameter set 118 and the action parameter set 120 may be weighted to obtain the action strength applied to the avatar.

[0041] In some embodiments, different scene information and voice information with different emotions can also correspond to different weights to improve the authenticity of the action parameters. For example, different weights can be set for different scene information and different emotions. When the scene information is different, or the emotion of the voice information is different, the weights in different situations can be determined. For example, when the emotion is "happy", its weight can be set to be larger, so that the action of the virtual image is more active visually. Reference Figure 5 , Figure 5, when the voice information is user voice information, an example architecture diagram that can be used to implement parameter calculation is shown. Among them, the scene information processing module 502 is used to obtain scene information, and input the action parameter set 118 corresponding to the scene information and the weighted weight of the action parameter set 118 into the parameter fusion module 506, the user voice recognition module 504 is used to identify the emotion of the user voice information, and input the action parameter set 120 corresponding to the emotion and the weighted weight of the action parameter set 120 into the parameter fusion module 506, the parameter fusion module 506 calculates based on the received action parameter set 118, action parameter set 120 and weighted weight, obtains the action parameters applied to the virtual image, and inputs the action parameters into the virtual image driving module 508, the virtual image driving module 508 is used to determine and display the action of the virtual image based on the received action parameters and action type.

[0042] In addition, reference Figure 6 , Figure 6 The diagram shows an example architecture diagram that can be used to implement parameter calculation when the voice information is the voice information of a virtual image. The scene information processing module 602 is used to obtain the scene information, and input the action parameter set 118 corresponding to the scene information and the weighted weight of the action parameter set 118 into the parameter fusion module 606. The virtual image voice recognition module 604 is used to obtain the action parameter set 120 and the weighted weight of the action parameter set 120, and input both into the parameter fusion module 606. The parameter fusion module 606 calculates based on the received action parameter set 118, action parameter set 120 and weighted weight to obtain the action parameters applied to the virtual image, and inputs the action parameters into the virtual image driving module 608. The virtual image driving module 608 is used to determine and display the action of the virtual image based on the received action parameters and action type.

[0043] In some embodiments, if the action parameter set 120 is not obtained by directly identifying the emotion of the virtual image voice information, but is calculated through the above-mentioned multi-entity emotion analysis, then the weighted weight of each participle can be calculated based on the confidence of the emotion corresponding to each participle and formula (1), and then the action parameter set 120 corresponding to the virtual image voice information is weightedly calculated according to the weighted weight and the action parameter set corresponding to the emotion of each participle, and the weighted weight of the action parameter set 120 is calculated according to the weighted weight of each participle by methods such as averaging calculation.

[0044] Y= (((X)^2*2)+4)*100 / 20000 Formula (1)

[0045] Where Y is the weighted weight of each segmentation, X is the confidence of the emotion corresponding to each segmentation, and X can be in the range of 0 to 10. The larger the X is, the more positive the emotion is. It can be understood that when a segmentation corresponds to two emotions, the final X of the segmentation can be obtained by averaging the X of the two emotions, or performing weighted calculation according to the weighted weights corresponding to different emotions (mentioned above).

[0046] Figure 7 The relationship between Y and X in formula (1) is shown. It can be seen that the more positive (larger) the emotion is, the greater the weight of each participle is, and the greater the impact on the action of the virtual image is. Compared with the direct table lookup method, the weight of each participle can be calculated more flexibly and accurately through formula (1). It should be understood that although the present disclosure only takes the multi-entity emotion analysis of the virtual image voice information as an example, the multi-entity emotion analysis of the user voice information should also be within the scope of the present disclosure.

[0047] In some embodiments, the processor 106 may match the scene information with multiple scene information sets and sub-scene information sets included in the scene information set to determine the action parameter set 118, or may simultaneously determine the action parameter set 118 and the weighted weight, thereby improving the subdivision degree of the scene information and improving the authenticity of the virtual image. For example, the processor 106 may first determine the scene information set to which the scene information belongs from multiple scene information sets, and then determine the target sub-scene information set to which the scene information belongs from multiple sub-scene information sets included in the scene information set, so that the action parameter set corresponding to the target sub-scene information set can be queried (for example, from the database 108) as the action parameter set 118, or the action parameter set corresponding to the target sub-scene information set and the weighted weight can be simultaneously determined as the action parameter set 118 and the weighted weight of the action parameter set 118. It should be noted that for the sub-scene information set, it may also have a sub-scene information set of the next layer, or a sub-scene information set of the next layer, and the more layers there are, the higher the subdivision degree of the scene information.

[0048] In order to facilitate understanding of the action parameter set 118 and the action parameter set 120 mentioned above, Table 1 and Table 2 are provided below. Table 1 exemplarily shows the action parameter sets and weighted weights corresponding to different scene information sets, different sub-scene information sets, and sub-scene information sets of the next layer of different sub-scene information sets (referred to as the third layer information set) when the scene information is time information. Table 2 shows the action parameter sets and weighted weights corresponding to different emotions, and also shows the action types of facial expressions corresponding to different emotions. However, Tables 1 and 2 should not be understood as limitations on the present disclosure.

[0049] Table 1. Correspondence table of scene information, action parameters, and weighted weights

[0050]

[0051]

[0052] Table 2 Emotion, action parameters, weighted weights, expression and action correspondence table

[0053]

[0054] When performing weighted calculation based on the weighted weights in Table 1 and Table 2, the action parameters can be normalized based on the weighted weights and then averaged to achieve weighted calculation, or weighted calculation can be achieved in other ways, which is not limited in the present disclosure.

[0055] In some embodiments, when the voice information includes both user voice information and avatar voice information, the action parameter set 120 can be calculated based on the action parameter set corresponding to the emotion of the user voice information and the action parameter set corresponding to the emotion of the avatar voice information. For example, the processor 106 can determine 212 the user emotion (also referred to as the first emotion) of the user voice information, and determine 214 the avatar emotion (also referred to as the second emotion) of the avatar voice information. Then, for the user voice information, the user emotion action parameter set (also referred to as the first emotion attribute set) corresponding to the user emotion and the weighted weight of the user emotion action parameter set are requested 216 from the database 108, and the database 108 sends 218 the user emotion action parameter set and the weighted weight of the user emotion action parameter set to the processor 106. For the avatar voice information, the processor 106 performs a multi-entity emotion analysis 220 on it, and based on the analysis results, requests 222 the emotion parameter set (i.e., action attribute set) corresponding to the emotion of each word segment from the database 108, and the database 108 sends 224 the emotion parameter set corresponding to the emotion of each word segment to the processor 106, and then the processor 106 calculates 226 the avatar action parameter set (also called the second emotion attribute set) corresponding to the avatar emotion, and the weighted weight of the avatar action parameter set. The example description of the multi-entity emotion analysis and the subsequent corresponding calculation can be referred to above and will not be repeated. Finally, the processor 106 determines 228 the action parameter set 120 based on the user emotion action parameter set, the avatar action parameter set, and the corresponding weighted weights.

[0056] refer to Figure 8 , Figure 8The present invention shows an example architecture diagram that can be used to implement parameter calculation when the voice information includes user voice information and virtual image voice information. The scene information processing module 802 is used to obtain scene information, and input the action parameter set 118 corresponding to the scene information and the weighted weight of the action parameter set 118 into the parameter fusion module 808. The user voice recognition module 804 is used to recognize the user emotion of the user voice information, and input the user emotion action parameter set corresponding to the user emotion and the weighted weight of the user emotion action parameter set into the parameter fusion module 808. The virtual image voice recognition module 806 is used to obtain the virtual image action parameter set and the weighted weight of the virtual image action parameter set, and input the two into the parameter fusion module 808. The parameter fusion module 808 calculates based on the received action parameter set 118, the user emotion action parameter set, the virtual image action parameter set and the respective weighted weights to obtain the action parameters applied to the virtual image, and inputs the action parameters into the virtual image driving module 810. The virtual image driving module 810 is used to determine and display the action of the virtual image based on the received action parameters and action type.

[0057] In addition, the present disclosure also provides a method for displaying a virtual image. Fig. 9 , Fig. 9 An example flow chart of a method is shown in FIG. At box 902, scene information and voice information in the scene are obtained. In some embodiments, the scene information may include at least one of time information, road condition information, vehicle operation information, or signal light information. For example, when the application scenario of the present method is a cockpit, the time information may be obtained from a clock built into the cockpit, or from a network through a networking module built into the cockpit, or the road condition information or signal light information may be obtained through a networking module, or the vehicle operation log module in the vehicle may be read to obtain the vehicle operation information. In some embodiments, the voice information may include user voice information input by a user and / or virtual image voice information output by a virtual image, wherein the virtual image voice information may be triggered based on the functional control of the application scenario. In some embodiments, in order to avoid excessive user voice information, so that the present method is performed multiple times and occupies the computing resources of the vehicle, the user voice information may be collected regularly, for example, the user voice information may be collected every 1 minute. Exemplarily, the scene information and voice information may be obtained by the processor 106.

[0058] At box 904, the action of the virtual image is determined based on the scene information and the voice information. In some embodiments, the action of the virtual image can be queried based on the semantics of the scene information and the voice information by looking up a table. In some embodiments, the action type of the virtual image can be first determined based on the current execution function of the application scene, and then the action attributes of multiple dimensions can be determined based on the emotions of the scene information and the voice information, wherein the action attributes of at least one dimension of the action attributes of the multiple dimensions change with the scene information and the voice information. In some examples, determining the action attributes of different dimensions can be achieved based on determining the action parameters corresponding to the action attributes of different dimensions. For example, the action parameters can be obtained by looking up the table based on the emotions of the scene information and the voice information, and the action of the virtual image can be determined based on the action type and the action parameters, thereby changing the action parameters of the action in real time to improve the authenticity of the virtual image. When the action includes an expression action and a body action, the action parameters corresponding to the expression action and the body action can be different.

[0059] In other embodiments, the respective action parameter sets can be obtained by looking up the emotion tables based on the scene information and the voice information, and then the action parameter sets are weighted to obtain the action parameters applied to the virtual image. When the voice information includes at least the virtual image voice information, in order to improve the accuracy of determining the action parameters, the action parameter set corresponding to the virtual image voice information can be determined based on the method of multi-entity emotion analysis, or the action parameter set and weighted weight corresponding to the virtual image voice information can be determined. The examples described above can be referred to and will not be repeated. In some embodiments, in order to improve the accuracy of the action parameter set 120 corresponding to the virtual image voice information and the weighted weight of the action parameter set 120, the weighted weight of each core entity word can be determined based on the confidence of the emotion of the core entity word in the virtual image voice information, and then the action parameter set 120 and the weighted weight of the action parameter set 120 can be determined based on the weighted weight of each core entity word.

[0060] Then at block 906, the action of the avatar is displayed. In some embodiments, the processor 106 may display the action of the avatar in the display module 112 in the application scenario. For example, when the application scenario is a cockpit, the display module 112 may refer to a screen in the cockpit. In some embodiments, the processor 106 may also display the current action parameters in a visual diagram in the display module 112 to provide the user with richer visual information.

[0061] According to the various embodiments of the present disclosure described above and their combined embodiments, the efficiency of configuring the functional state can be improved. Figures 1 to 9 The implementation method shown in the figure is only for reference. Figures 1 to 9The architecture or process illustrated in the figure may have other different forms, and may also include more or fewer functional modules and / or units. These modules and / or units may be partially or fully implemented as hardware modules, software modules, firmware modules or any combination thereof, and the embodiments of the present disclosure are not limited to this.

[0062] It is to be understood that the specific names and / or protocols of the various components in the system described herein are only to help those skilled in the art to better understand the concept of the present disclosure, and are not intended to limit the scope of the present disclosure in any way. Moreover, in other embodiments, more or better components may be included, and alternative components with the same or similar functions may also be included.

[0063] Fig.10 1 shows a schematic structural diagram of an example device 1000 that can be used to implement some embodiments of the present disclosure. The device 1000 can be implemented as a server or a PC, etc. The embodiments of the present disclosure do not limit the specific implementation type of the device 1000. Fig.10 As shown, the device 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 1002 or computer program instructions loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0064] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0065] The processing unit 1001 can execute the various methods and / or processes described above, for example Fig. 9 For example, in some embodiments, Fig. 9The method shown may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by CPU 1001, the above-described Fig. 9 Alternatively, in other embodiments, the CPU 1001 may be configured to execute the method in any other suitable manner (eg, by means of firmware, etc.). Fig. 9 The method shown.

[0066] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip systems (SOCs), load programmable logic devices (CPLDs), and the like.

[0067] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0068] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0069] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0070] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, and conventional procedural programming languages. The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server.

[0071] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer or other programmable data processing device, thereby producing a machine so that when these instructions are executed by the processing unit of a computer or other programmable data processing device, a device for realizing the functions / actions specified in one or more boxes in the flow chart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions make the computer, programmable data processing device and / or other equipment work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for realizing various aspects of the functions / actions specified in one or more boxes in the flow chart and / or block diagram. Computer-readable program instructions can also be loaded onto a computer, other programmable data processing device, or other device, so that a series of operating steps are performed on the computer, other programmable data processing device or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing device, or other device realize the functions / actions specified in one or more boxes in the flow chart and / or block diagram.

[0072] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0073] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the equipment, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous frames can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the functions involved. It should also be noted that each frame in the block diagram and / or flow chart, and the combination of frames in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0074] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.

[0075] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0076] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for displaying a virtual image, comprising: Obtain scene information and voice information in the scene; Determining an action of the avatar based on the scene information and the voice information; as well as The action of the virtual image is displayed.

2. The method of claim 1, wherein determining the action of the avatar comprises: determining an action type of the avatar; Determining action attributes in multiple dimensions based on the scene information and the voice information, wherein the action attribute in at least one dimension of the action attributes in multiple dimensions changes with the scene information and the voice information; as well as The action of the virtual image is determined based on the action type and the action attributes in multiple dimensions. 3 . The method according to claim 2 , wherein the action attributes in multiple dimensions include the action attributes in at least two of the following dimensions: action movement amplitude, action frequency, and action strength.

4. The method according to claim 2, wherein determining the action type of the avatar comprises: Determining the action type of the facial expression action based on the emotion of the voice information; and / or Based on the currently performed function of the cockpit, the action type of the body action is determined.

5. The method according to claim 4, wherein determining the action type of the body action further comprises: Determining that a current execution function of the cockpit is switched from a first execution function to a second execution function; as well as Based on the first execution function and the second execution function, the action type of the body action is determined to transition between a body action of a first action type corresponding to the first execution function to a body action of a second action type corresponding to the second execution function.

6. The method of claim 2, wherein determining the action attributes of multiple dimensions comprises: Determining a first action attribute set corresponding to the scene information and a second action attribute set corresponding to the voice information; as well as Based on the first action attribute set and the second action attribute set, the action attributes of multiple dimensions are determined.

7. The method according to claim 6, wherein determining the action attributes of multiple dimensions further comprises: Determining a first effect weight of the scene information and a second effect weight of the voice information; as well as The action attributes of multiple dimensions are determined based on the first effect weight, the second effect weight, the first action attribute set, and the second action attribute set.

8. The method according to claim 6, wherein determining the first action attribute set corresponding to the scene information comprises: Determining the scene information set to which the scene information belongs; Determining a target sub-scene information set to which the scene information belongs from a plurality of sub-scene information sets included in the scene information set; as well as A first action attribute set corresponding to the target sub-scene is determined.

9. The method according to claim 6, wherein the voice information includes first voice information input by a user and / or second voice information output by the virtual image.

10. The method according to claim 9, wherein determining the second action attribute set corresponding to the voice information comprises: Determine a first emotion of the first voice information or a second emotion of the second voice information; as well as Based on the first emotion or the second emotion, the second action attribute set is determined.

11. The method according to claim 9, wherein determining the second action attribute set corresponding to the voice information comprises: Determining a first emotion of the first voice information and a second emotion of the second voice information; Determining a first emotion attribute set corresponding to the first emotion and a second emotion attribute set corresponding to the second emotion; as well as Based on the first emotion attribute set and the second emotion attribute set, the second action attribute set corresponding to the voice information is determined.

12. The method of claim 11, wherein determining the second set of emotion attributes comprises: Determining the emotion of each of the plurality of segmented words in the second voice information from the second emotion; as well as The second emotion attribute set is determined based on the emotion attribute set corresponding to the emotion of each word in the multiple word segments.

13. The method according to claim 12, further comprising: Determining that there is a target participle having multiple emotions among the multiple participles; as well as Based on the emotion attribute set corresponding to each emotion in the multiple emotions, the emotion set attribute corresponding to the target word segment is determined.

14. The method according to claim 1, wherein the scene information includes at least one of time information, road condition information, vehicle operation information, or traffic light information.

15. An electronic device, comprising: processor, and A memory is coupled to the processor and stores instructions which, when executed by the processor, cause the device to perform the method according to claims 1 to 14.

16. A vehicle comprising the electronic device according to claim 15.

17. A computer-readable storage medium having computer-executable instructions stored thereon, wherein when the computer-executable instructions are executed, the computer executes the steps of the performance evaluation method for an on-board chip according to any one of claims 1 to 14.

18. A computer program product, which is tangibly stored on a non-volatile computer-readable medium and comprises machine-executable instructions, which when executed, cause a machine to perform the steps of the performance evaluation method for an on-vehicle chip according to any one of claims 1 to 14.

Images