Methods, display methods, and related equipment for obtaining operational instructions based on the cockpit area

Abstract

This disclosure provides a method, display method, and related device for obtaining operation instructions based on a cockpit area. The method includes: recognizing and processing operation commands from a user in the cockpit to obtain a first recognition result; receiving a second recognition result from a wearable electronic device of an in-vehicle device icon; filtering a target functional area corresponding to the first recognition result and / or the second recognition result from at least one pre-defined functional area; obtaining operation instruction information corresponding to the target functional area; and sending the operation instruction information to the wearable electronic device so that the wearable electronic device can prompt the user with the operation instruction information.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent driving technology, and in particular to a method, display method and related equipment for obtaining operation instructions based on the cockpit area. Background Technology

[0002] In today's society, vehicle ownership is constantly increasing. However, vehicles have numerous functional areas, and correspondingly, the operation of these areas is becoming increasingly complex. Users unfamiliar with how to operate these functional areas often find it difficult to do so. In emergency situations, users may be unable to operate the relevant functional areas, negatively impacting the user experience.

[0003] Therefore, how to determine the functional areas that users want to know and how to operate the corresponding functional areas has become an important research question. Summary of the Invention

[0004] In view of this, the purpose of this disclosure is to propose a method, display method and related equipment for obtaining operation instructions based on the cockpit area, so as to solve the problem in the prior art that users do not understand the operation mode of the functional area and cannot operate the corresponding functional area.

[0005] To achieve the above objectives, the first aspect of this disclosure proposes a method for obtaining operational instructions based on the cockpit area, comprising:

[0006] The operation commands from the user inside the cockpit are recognized and processed to obtain the first recognition result;

[0007] Receive the second recognition result of the wearable electronic device for the in-vehicle device icon;

[0008] From at least one pre-defined functional region, filter the target functional region corresponding to the first identification result and / or the second identification result;

[0009] Obtain the operation instruction information corresponding to the target functional area;

[0010] The operation instruction information is sent to the wearable electronic device so that the wearable electronic device can prompt the user with the operation instruction information.

[0011] In some embodiments, the step of filtering target functional regions corresponding to the first identification result and / or the second identification result from at least one pre-defined functional region includes:

[0012] In response to determining that the first identification result meets the preset conditions, a multimodal fusion judgment is triggered;

[0013] The first identification result and / or the second identification result are judged and processed by the multimodal fusion judgment to determine the target functional region from at least one pre-defined functional region;

[0014] The target functional area is sent to the in-vehicle virtual personal assistant for confirmation.

[0015] In some embodiments, the first recognition result includes at least one of the following: voice keyword recognition result, facial expression recognition result;

[0016] The response to determining that the first recognition result meets a preset condition and triggering a multimodal fusion judgment includes:

[0017] In response to determining that there is a voice keyword in the voice keyword recognition result that is consistent with the at least one functional area, or that there is a preset expression in the expression recognition result, a multimodal fusion judgment is triggered.

[0018] In some embodiments, the number of the first recognition results is multiple, and the first recognition results include at least two of the following: voice keyword recognition results, gaze recognition results, gesture recognition results, and facial expression recognition results;

[0019] The step of judging the first recognition result through the multimodal fusion judgment includes:

[0020] The multimodal fusion judgment process is used to judge at least two first recognition results in a pre-set priority order.

[0021] or,

[0022] The step of judging the first recognition result and the second recognition result through the multimodal fusion judgment includes:

[0023] The multimodal fusion judgment process sequentially judges at least one first recognition result and the second recognition result according to a pre-set priority order.

[0024] In some embodiments, the first recognition result includes: voice keyword recognition result, gaze recognition result, gesture recognition result, and facial expression recognition result; the second recognition result includes: vehicle device icon recognition result;

[0025] The step of judging the first recognition result and / or the second recognition result through the multimodal fusion judgment to determine the target functional region from at least one pre-defined functional region includes:

[0026] The first identification result and the second identification result are judged sequentially according to a pre-set priority order through the multimodal fusion judgment, and the target functional region is determined from at least one pre-defined functional region.

[0027] The preset priority order is the priority order of the voice keyword recognition result, the device icon recognition result, the gaze recognition result, and the gesture recognition result.

[0028] In some embodiments, the step of sequentially judging the first recognition result and the second recognition result according to a pre-set priority order through multimodal fusion to determine the target functional region from at least one pre-defined functional region includes:

[0029] In response to determining that there is a voice keyword in the voice keyword recognition result that is consistent with the keyword of the at least one functional region, the functional region that is consistent with the voice keyword is taken as the target functional region;

[0030] In response to determining that there is no voice keyword in the voice keyword recognition result that is consistent with the keyword of the at least one functional area, the vehicle device icon recognition result is judged;

[0031] In response to determining that there is a vehicle device icon with a functional area in the vehicle device icon recognition result, the functional area corresponding to the vehicle device icon is taken as the target functional area;

[0032] In response to determining that there are vehicle device icons with at least two functional areas in the vehicle device icon recognition result, the gaze recognition result is judged based on the at least two functional areas;

[0033] In response to determining that there is no vehicle device icon with a functional area in the vehicle device icon recognition result, the gaze recognition result is judged;

[0034] In response to determining that there is a functional region in which the gaze is looking in the gaze recognition result, the functional region in which the gaze is looking is taken as the target functional region;

[0035] In response to determining that there are at least two functional areas in the gaze recognition result, the gesture recognition result is judged based on the at least two functional areas in the gaze recognition result;

[0036] In response to determining that there is no functional area being looked at in the gaze recognition result, the gesture recognition result is judged;

[0037] In response to determining that there is a finger pointing to a functional area in the gesture recognition result, the finger pointing to a functional area is taken as the target functional area.

[0038] In some embodiments, before filtering the target functional region corresponding to the first identification result and / or the second identification result from at least one pre-defined functional region, the method further includes:

[0039] By analyzing and processing the gaze recognition results, the gesture recognition results, and the facial expression recognition results, the face recognition results are obtained.

[0040] By analyzing the speech keyword recognition results, the speech region location recognition result is obtained;

[0041] The face recognition result and the voice zone location recognition result are processed to obtain a matching judgment result, so as to determine that the first recognition result is the recognition result data of the same user in the cabin.

[0042] Based on the same inventive concept, a second aspect of this disclosure proposes a method for displaying operation instructions, comprising:

[0043] The icons of in-vehicle devices collected in front of the user inside the cabin are identified and processed to obtain a second identification result;

[0044] The second identification result is sent to the vehicle terminal so that the vehicle terminal can obtain the corresponding operation instruction information based on the second identification result and the first identification result obtained by the vehicle terminal;

[0045] Receive the operation instruction information from the vehicle terminal;

[0046] The operation instructions can be read aloud or overlaid on a real-world scene.

[0047] Based on the same inventive concept, a third aspect of this disclosure proposes a vehicle-mounted device, comprising:

[0048] The first recognition result acquisition module is configured to recognize and process the operation commands of the user in the cockpit to obtain the first recognition result;

[0049] The second recognition result receiving module is configured to receive the second recognition result of the wearable electronic device for the vehicle device icon;

[0050] The target functional area determination module is configured to filter target functional areas corresponding to the first identification result and / or the second identification result from at least one pre-defined functional area;

[0051] The operation instruction information acquisition module is configured to acquire operation instruction information corresponding to the target functional area;

[0052] The operation instruction information sending module is configured to send the operation instruction information to the wearable electronic device so that the wearable electronic device can prompt the user with the operation instruction information.

[0053] Based on the same inventive concept, a fourth aspect of this disclosure proposes a wearable electronic device, comprising:

[0054] The second recognition result acquisition module is configured to recognize and process the in-vehicle device icons collected in front of the user in the cabin to obtain the second recognition result;

[0055] The second identification result sending module is configured to send the second identification result to the vehicle terminal, so that the vehicle terminal can obtain corresponding operation instruction information based on the second identification result and the first identification result obtained by the vehicle terminal;

[0056] The operation instruction information receiving module is configured to receive the operation instruction information from the vehicle terminal;

[0057] The operation instruction information display module is configured to either broadcast the operation instruction information via voice or overlay it onto a real-world scene.

[0058] Based on the same inventive concept, a fifth aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method described above when executing the computer program.

[0059] Based on the same inventive concept, a sixth aspect of this disclosure provides a non-transitory computer-readable storage medium that stores computer instructions for causing a computer to perform the method described above.

[0060] Based on the same inventive concept, a seventh aspect of this disclosure proposes a vehicle, the vehicle including the vehicle end as described in the third aspect, the electronic equipment as described in the fifth aspect, or the storage medium as described in the sixth aspect.

[0061] As can be seen from the above, the method, display method, and related equipment for obtaining operation instructions based on the cockpit area provided in this disclosure filter out the corresponding target functional areas based on the recognition results of user operation commands in the cockpit and / or the recognition results of in-vehicle electronic devices. This makes the filtered target functional areas more accurate and precisely identifies the target functional areas of interest to the user. The operation instruction information corresponding to the target functional area is sent to a wearable electronic device to prompt the user, so that the user can understand the operation instruction information of the target functional area in a timely manner and operate the target functional area according to the operation instruction information, thereby improving the user experience. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in this disclosure or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a flowchart of a method for obtaining operational instructions based on the cockpit area according to an embodiment of the present disclosure;

[0064] Figure 2 A flowchart illustrating a method for displaying operation instructions according to an embodiment of this disclosure;

[0065] Figure 3 This is a schematic diagram illustrating the acquisition and display of operation instructions based on the cockpit area according to an embodiment of this disclosure;

[0066] Figure 4 This is a schematic diagram of XR glasses according to an embodiment of the present disclosure;

[0067] Figure 5 This is a schematic diagram of the vehicle end structure according to an embodiment of the present disclosure;

[0068] Figure 6 This is a schematic diagram of the structure of a wearable electronic device according to an embodiment of the present disclosure;

[0069] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0071] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0072] As mentioned above, how to determine the functional areas that users want to know and how to operate the corresponding functional areas has become an important research question.

[0073] Based on the above description, such as Figure 1 As shown in this embodiment, the method for obtaining operation instructions based on the cockpit area is applied to the vehicle end, and the method includes:

[0074] Step 101: Recognize and process the user's operation commands in the cockpit to obtain the first recognition result.

[0075] In practice, the user's operating commands in the cabin are recognized and processed by sensors on the vehicle side to obtain a first recognition result. These sensors include at least one of the following: an inertial measurement unit (IMU), a camera, and a recording device. Operating commands mainly consist of the user's voice and actions; therefore, the first recognition result primarily involves recognizing the user's voice and actions to obtain corresponding results. For example, the first recognition result could be a voice keyword recognition result, an facial expression recognition result, a gaze recognition result, or a gesture recognition result. Based on the information related to functional areas in the user's voice and actions from the first recognition result, the target functional area of ​​interest to the user can be determined.

[0076] Step 102: Receive the second recognition result of the wearable electronic device for the vehicle device icon.

[0077] In specific implementation, the wearable electronic device is equipped with sensors (e.g., a camera) capable of collecting information. Information data is collected through the sensors on the wearable electronic device, and icons on in-vehicle devices and displays near the user or in front of their line of sight are identified and processed to obtain the in-vehicle device icon recognition result as a second recognition result. The wearable electronic device can be glasses, headphones, a watch, a necklace, earrings, a brooch, etc., capable of data processing. In this embodiment, the wearable electronic device is preferably XR glasses (Extended Reality, XR for short). The second recognition result is mainly the in-vehicle device icon recognition result, which can determine the target functional area of ​​interest to the user based on the in-vehicle devices and icons collected near the user or in front of their line of sight.

[0078] Step 103: Select the target functional area corresponding to the first identification result and / or the second identification result from at least one pre-defined functional area.

[0079] In practice, functional areas are defined by dividing the vehicle cabin's various in-vehicle devices and icons according to their different functions and locations. Multiple functional areas are defined, and for different vehicle models, these areas are divided and defined based on the function and location of the in-vehicle equipment. For example, functional areas might be divided into: air conditioning area, entertainment screen, sunroof, seats, etc. Based on the first and / or second recognition results, the target functional area that the user wants to know or is interested in is selected from these multiple functional areas.

[0080] Step 104: Obtain the operation instruction information corresponding to the target functional area.

[0081] In practice, the vehicle-mounted terminal pre-stores operation instructions for each functional area. After selecting the target functional area, the corresponding operation instructions are retrieved. These operation instructions can be retrieved from a knowledge graph, which is stored on the vehicle-mounted terminal or a server. The operation instructions include at least one of the following: extended equipment maintenance information, or equipment operation animation display.

[0082] Step 105: Send the operation instruction information to the wearable electronic device so that the wearable electronic device can prompt the user with the operation instruction information.

[0083] In practice, the operation instructions corresponding to the target functional area are sent to the wearable electronic device to prompt the user, so that the user can understand the operation instructions of the target functional area in a timely manner and operate the target functional area according to the operation instructions, thereby improving the user experience.

[0084] In the above embodiments, based on the recognition results of user operation commands in the cockpit and / or the recognition results of in-vehicle electronic devices, corresponding target functional areas are selected, making the selected target functional areas more accurate and accurately identifying the target functional areas of interest to the user; the operation instruction information corresponding to the target functional area is sent to the wearable electronic device to prompt the user, so that the user can understand the operation instruction information of the target functional area in a timely manner and operate the target functional area according to the operation instruction information, thereby improving the user experience.

[0085] In some embodiments, step 103 includes:

[0086] Step 1031: In response to determining that the first recognition result meets the preset conditions, trigger multimodal fusion judgment;

[0087] Step 1032: The first identification result and / or the second identification result are judged and processed by the multimodal fusion judgment to determine the target functional region from at least one pre-defined functional region;

[0088] Step 1033: Send the target functional area to the in-vehicle virtual personal assistant for confirmation of the target functional area.

[0089] In practice, when a pre-set condition is met in the first recognition result, a multimodal fusion judgment is triggered. There can be only one first recognition result or multiple first recognition results. When there are multiple first recognition results, the multimodal fusion judgment is triggered if at least one of the first recognition results meets the pre-set condition. The pre-set condition may be the appearance of a functional region name in the user's voice keywords, or the recognition of a user's facial expression matching a preset expression (e.g., a confused expression or a frowning expression).

[0090] Multimodal fusion judgment refers to the process of judging multiple recognition results based on rules when there is a high correlation between them. Multimodal fusion judgment is used to judge the first and / or second recognition results. When a recognition result related to a functional area is identified, that functional area is designated as the target functional area of ​​interest to the user. For example, if a voice keyword related to the name of a functional area is recognized, that functional area is designated as the target functional area. The obtained target functional area is then sent to the in-vehicle virtual personal assistant (VPA) for confirmation. For example, the VPA can provide a voice prompt to the user, asking if they need detailed information about the target functional area. After the user confirms, the accurate target functional area is obtained.

[0091] In the above scheme, when the first recognition result meets the preset conditions, a multimodal fusion judgment is triggered, and the target functional area is obtained through the multimodal fusion judgment. Therefore, the multimodal fusion judgment is only triggered when the user wants to know the operation prompt information of the functional area, thus avoiding arbitrary triggering of multimodal fusion judgment and the problem of triggering judgment errors. After the target functional area is determined, it is confirmed by VPA, making the obtained target functional area more accurate.

[0092] In some embodiments, the first recognition result includes at least one of the following: voice keyword recognition result, facial expression recognition result;

[0093] Step 1031 includes:

[0094] Step 1031A: In response to determining that there is a voice keyword in the voice keyword recognition result that is consistent with the at least one functional area, or that there is a preset expression in the expression recognition result, a multimodal fusion judgment is triggered.

[0095] In practice, the first recognition result includes multiple factors. When the first recognition result includes a voice keyword recognition result or an expression recognition result, the condition for triggering multimodal fusion judgment is that the voice keyword recognition result or the expression recognition result meets a preset condition. For example, the voice keyword recognition result contains a voice keyword consistent with the functional area, or the expression recognition result contains a preset expression, where the preset expression could be a user's frowning expression.

[0096] In the above scheme, the user's voice keywords and facial expressions can more accurately determine whether the functional area the user wants to know exists. When the voice keyword recognition result or facial expression recognition result meets the preset conditions, a multimodal fusion judgment is triggered, making the triggering conditions of the multimodal fusion judgment more accurate and timely in judging the functional area of ​​interest to the user.

[0097] In some embodiments, the number of the first recognition results is multiple, and the first recognition results include at least two of the following: voice keyword recognition results, gaze recognition results, gesture recognition results, and facial expression recognition results;

[0098] Step 1032 includes:

[0099] Step 1032A: The multimodal fusion judgment is used to judge at least two first recognition results in a pre-set priority order.

[0100] or,

[0101] Step 1032B: The first identification result and the second identification result are judged sequentially according to a pre-set priority order through the multimodal fusion judgment.

[0102] In practice, when performing multimodal fusion judgment, if multiple recognition results are judged, there is a pre-set priority order among the multiple recognition results. The multiple recognition results are judged in order of priority until a unique target functional area is determined, and then the judgment ends.

[0103] There can be multiple first identification results. In the multimodal fusion judgment process, multiple identification results can be at least two first identification results, or at least one first identification result and one second identification result.

[0104] In the above scheme, during multimodal fusion judgment, multiple recognition results are judged and processed sequentially according to a pre-set priority order to obtain the target functional region. This makes the obtained target functional region more accurate. When one recognition result cannot determine the target functional region, the next recognition result can be judged until the target functional region is obtained, avoiding the problem of not being able to determine the target functional region.

[0105] In some embodiments, the first recognition result includes: voice keyword recognition result, gaze recognition result, gesture recognition result, and facial expression recognition result; the second recognition result includes: vehicle device icon recognition result;

[0106] Step 1032 includes:

[0107] Step 10321: The first identification result and the second identification result are judged sequentially according to a preset priority order through the multimodal fusion judgment to determine the target functional region from at least one pre-defined functional region;

[0108] The preset priority order is the priority order of the voice keyword recognition result, the device icon recognition result, the gaze recognition result, and the gesture recognition result.

[0109] In practice, during multimodal fusion judgment, multiple recognition results are processed sequentially according to a pre-set priority order to obtain the target functional area. When the first recognition result includes voice keyword recognition result, gaze recognition result, gesture recognition result, and facial expression recognition result, and the second recognition result includes vehicle device icon recognition result, the pre-set priority order is the priority order of voice keyword recognition result, device icon recognition result, gaze recognition result, and gesture recognition result.

[0110] In the above scheme, the pre-set priority order is sorted according to the accuracy of the identification results in judging the target functional area. The more accurate the identification result, the higher the priority, so that the target functional area is more accurate.

[0111] In some embodiments, step 10321 includes:

[0112] Step 10321a: In response to determining that there is a voice keyword in the voice keyword recognition result that is consistent with the keyword of the at least one functional region, the functional region that is consistent with the voice keyword is taken as the target functional region;

[0113] Step 10321b: In response to determining that there is no voice keyword in the voice keyword recognition result that is consistent with the keyword of the at least one functional area, the vehicle device icon recognition result is judged;

[0114] Step 10321c: In response to determining that there is a vehicle device icon with a functional area in the vehicle device icon recognition result, the functional area corresponding to the vehicle device icon is taken as the target functional area;

[0115] Step 10321d: In response to determining that there are vehicle device icons with at least two functional areas in the vehicle device icon recognition result, the gaze recognition result is judged based on the at least two functional areas;

[0116] Step 10321e: In response to determining that there is no vehicle device icon with a functional area in the vehicle device icon recognition result, the gaze recognition result is judged;

[0117] Step 10321f: In response to determining that there is a functional area in which the gaze is looking in the gaze recognition result, the functional area in which the gaze is looking is taken as the target functional area;

[0118] Step 10321g: In response to determining that there are at least two functional areas in the gaze recognition result, the gesture recognition result is judged based on the at least two functional areas in the gaze recognition result.

[0119] Step 10321h: In response to determining that there is no functional area in which the gaze is looking in the gaze recognition result, the gesture recognition result is judged;

[0120] Step 10321i: In response to determining that there is a finger-pointing functional area in the gesture recognition result, the finger-pointing functional area is taken as the target functional area.

[0121] In practice, the judgments are performed sequentially according to a pre-set priority order. First, the speech recognition results are judged. If a unique target functional area can be obtained, the target functional area is sent to the VPA for confirmation. If no target functional area can be obtained, the vehicle device icon recognition results are judged according to the priority order. If at least two target functional areas are obtained, the vehicle device icon recognition results are judged based on the priority order of the at least two target functional areas to obtain a unique target recognition result. The judgment process for other recognition results is similar to that for the speech keyword recognition results described above, and will not be repeated here.

[0122] In addition, the first and second recognition results are acquired through sensors. For example, the voice keyword recognition result is acquired through a recording device (e.g., a microphone); the gaze recognition result, gesture recognition result, and facial expression recognition result are acquired through a camera, wherein the camera includes at least one of the following: a TOF camera (Time of Flight camera, which outputs 3D point cloud information), a DMS camera (Driver Monitoring System, which collects data from the driver), or an OMS camera (Occupancy Monitoring System, which collects data from the passenger); the device icon recognition result is acquired through a camera on a wearable electronic device. Furthermore, the acquisition process of each of the above recognition results is trained using a deep neural network, making the recognition results more accurate.

[0123] In the above scheme, when performing multimodal fusion judgment on multiple recognition results according to priority order, if at least two target functional regions exist, the next recognition result is judged according to priority order until a unique target functional region is determined, ensuring that the obtained target functional region is unique and accurate. Furthermore, the acquisition process of each recognition result is trained using a deep neural network, thus guaranteeing the accuracy of each recognition result.

[0124] In some embodiments, prior to step 103, the method further includes:

[0125] Step 103A: By analyzing and processing the gaze recognition result, the gesture recognition result, and the expression recognition result, a face recognition result is obtained;

[0126] Step 103B: By performing a voice region determination on the voice keyword recognition results, a voice region location recognition result is obtained;

[0127] Step 103C: The face recognition result and the voice zone location recognition result are judged and processed to obtain a matching judgment result, so as to determine that the first recognition result is the recognition result data of the same user in the cabin.

[0128] In practice, the gaze recognition, gesture recognition, and facial expression recognition results captured by the camera are analyzed and processed to obtain the face recognition result. Based on the face recognition result, it can be determined that the above three recognition results belong to the same user. The voice keyword recognition results captured by the audio receiving device are processed to obtain the voice region location recognition result. Based on the voice region location recognition result, the user's location can be determined. The face recognition result and voice region location recognition result are processed to obtain the matching judgment result. Based on the matching judgment result, it can be determined that the first recognition result belongs to the same user.

[0129] In the above scheme, each identification result in the first identification result is analyzed and processed to determine that the first identification result belongs to the same user. This avoids the problem of identification results not belonging to the same user, which could affect the determination of the target functional area.

[0130] In the above embodiments, based on the recognition results of user operation commands in the cockpit and / or the recognition results of in-vehicle electronic devices, corresponding target functional areas are selected, making the selected target functional areas more accurate and accurately identifying the target functional areas of interest to the user; the operation instruction information corresponding to the target functional area is sent to the wearable electronic device to prompt the user, so that the user can understand the operation instruction information of the target functional area in a timely manner and operate the target functional area according to the operation instruction information, thereby improving the user experience.

[0131] like Figure 2 As shown in this embodiment, the method for displaying operation instructions is applied to a wearable electronic device, and the method includes:

[0132] Step 201: Recognize the in-vehicle device icons collected in front of the user in the cabin to obtain the second recognition result.

[0133] Step 202: Send the second recognition result to the vehicle terminal so that the vehicle terminal can obtain the corresponding operation instruction information based on the second recognition result and the first recognition result obtained by the vehicle terminal.

[0134] Step 203: Receive the operation instruction information from the vehicle terminal.

[0135] Step 204: The operation instruction information is broadcast by voice or overlaid on the real scene.

[0136] In practice, the wearable electronic device is equipped with a camera. The camera captures icons of in-vehicle devices in front of or near the user, obtaining a second recognition result. This second recognition result includes the recognition result of the in-vehicle device icon. The second recognition result is sent to the vehicle, where the vehicle can use the first and second recognition results to determine the target functional area, making the determined target functional area more accurate.

[0137] Once the vehicle-mounted device identifies the target functional area and retrieves the corresponding operation instructions, it sends these instructions to the wearable electronic device. The wearable electronic device can then either broadcast the instructions via voice or overlay them onto the real-world scene. The operation instructions include at least one of the following: extended equipment maintenance information, or an animation of equipment operation. Alternatively, the vehicle-mounted device can also directly display the operation instructions.

[0138] The wearable electronic device can be Extended Reality (XR) glasses, which include Augmented Reality (AR) glasses and Virtual Reality (VR) glasses. During use, users can interact with their environment using a camera installed on the device. By overlaying operation instructions onto corresponding target functional areas in the real-world scene using XR glasses, users can more clearly understand how to operate the corresponding target functional areas.

[0139] Extended reality (AR) refers to the use of computers to combine the real and virtual worlds, creating an interactive virtual environment. This is also a general term for various technologies such as AR and VR. By integrating the visual interaction technologies of these three technologies, it brings users a sense of immersion, allowing for a seamless transition between the virtual and real worlds.

[0140] In the above embodiments, the second recognition result collected by the wearable electronic device is combined with the first recognition result to determine the target functional area, making the selected target functional area more accurate and precisely identifying the target functional area of ​​interest to the user. The wearable electronic device will then prompt the user with the received operation instructions, allowing the user to promptly understand the operation instructions for the target functional area and operate accordingly. Furthermore, by overlaying the information onto the real-world scene, the user can clearly understand the operation method of the target functional area, improving the user experience.

[0141] It should be noted that the embodiments of this disclosure can also be further described in the following ways:

[0142] like Figure 3 As shown, Figure 3This is a schematic diagram illustrating the acquisition and display of operation instructions based on the cockpit area according to an embodiment of this disclosure.

[0143] Step 1: Multimodal interaction recognition intent (intent, i.e., target functional area)

[0144] The system identifies the target functional areas of interest to the user within the cabin. This identification can be based on multiple recognition results, such as voice keyword recognition, facial expression recognition, gaze recognition, gesture recognition, and in-vehicle device icon recognition. The system performs multimodal fusion processing on the user's current gaze, hand movements, and mentioned keywords to determine the target functional areas of interest. Sensors then perform corresponding multimodal recognition (voice keywords, facial expressions, head posture, gaze, and gestures) to obtain the corresponding results. Finally, these recognition results are processed to determine the target functional areas of interest for the same user, and operation instructions for these areas are displayed to the user. The main steps include the following:

[0145] (1) Pre-processing: Division and calibration of cockpit functional areas

[0146] The cabin is divided into different functional areas according to vehicle features, such as: air conditioning area, entertainment screen, sunroof, seats, etc. Before the sensors process each modality (i.e., the process of collecting and processing the corresponding recognition results through the sensors), the functional areas must be calibrated for different vehicle models. After processing each modality, the output is the functional area that the user is currently interested in.

[0147] (2) Preprocessing: Judgment of the identification results of the same user

[0148] 1. Face recognition results

[0149] By analyzing and processing the results of gaze recognition, gesture recognition, and facial expression recognition, it is determined whether the above recognition results belong to the same user, and the user's location is determined.

[0150] 2. Results of register location identification

[0151] By analyzing and processing the speech keyword recognition results, and judging the location of the user based on the sound field position of the sound region, the user's location is determined. Combined with the corresponding face recognition results, it is determined that the above recognition results are for the same user.

[0152] (3) Multimodal interaction recognition

[0153] In multimodal interaction recognition, sensors are primarily used to acquire the behavior and voice of users within the cabin from visual and auditory perspectives. These sensors are mainly divided into three categories: visual sensors (e.g., TOF cameras, DMS cameras, and OMS cameras), auditory sensors (e.g., microphones and other sound recording devices), and IMU sensors. Specifically, visual sensors collect user behavior and posture data; auditory sensors collect user voice data; and IMU sensors collect user motion and posture data.

[0154] 1. Speech Keyword Recognition

[0155] Based on different functional areas within the cockpit, keywords related to these functional areas are extracted from the user's speech. By recognizing the user's speech within the cockpit, the target functional areas the user might be interested in are identified, resulting in speech keyword recognition results. This speech keyword recognition process is trained using a deep neural network (e.g., an AI deep neural network).

[0156] 2. Head posture and gaze detection

[0157] Based on camera input, the system identifies the user's head posture and gaze direction, determines the target functional area that the user may be interested in, and obtains the gaze recognition result. This head posture and gaze detection process is trained using a deep neural network (e.g., an AI deep neural network), and uses labeled functional areas to determine the functional area the user is looking at as the target functional area.

[0158] 3. Gesture detection and recognition

[0159] Based on camera input, the system recognizes the user's hand gestures and, based on the functional area pointed to by the user's hand or entire arm, determines the target functional area that the user may be interested in, thus obtaining the gesture recognition result. This gesture detection and recognition process is trained using a deep neural network (e.g., an AI deep neural network), and the functional area pointed to by the user's hand is identified as the target functional area through a calibrated functional area.

[0160] 4. Facial Expression Detection and Recognition

[0161] Based on camera input, the system recognizes the user's facial expressions, obtains the expression recognition results, and provides them to the VPA virtual assistant as a trigger condition. The aforementioned expression detection and recognition process is trained using a deep neural network (e.g., an AI deep neural network).

[0162] 5. Vehicle-mounted device identification

[0163] Wearable electronic devices can utilize virtual reality display devices, which can employ XR glasses (including AR and VR glasses), such as... Figure 4 As shown, Figure 4This is a schematic diagram of an XR glasses embodiment of the present disclosure. The XR glasses are equipped with a visual sensor (e.g., a camera, which can be one or multiple cameras, depending on the XR glasses device) to collect information from the user's field of vision (e.g., collecting information from the vehicle's in-vehicle device icon in the user's forward view to obtain the vehicle device icon recognition result). In addition, the XR glasses may also have inertial sensors such as a gyroscope, accelerometer, and magnetometer. The XR glasses can establish a connection with the vehicle and obtain corresponding input and output signals.

[0164] The XR glasses have a camera that can see the user's field of view, identify in-vehicle devices and icons within that view (50% of the central area), and establish corresponding connections with the functional areas on the vehicle. This in-vehicle device recognition process is trained using a deep neural network (e.g., an AI deep neural network).

[0165] The training process can be divided into two parts: recognizing the name of the in-vehicle device, with the training input being images of the in-vehicle device from various angles and the output being the name of the in-vehicle device; and recognizing the name of the icon, with the training input being images of the icons in the function area from various angles and the output being the name of the icon.

[0166] 6. Multimodal fusion to determine the target functional region

[0167] Based on the above identification results, the target functional area of ​​interest to the user is determined through multimodal fusion and sent to the VPA virtual assistant for further confirmation of the target functional area.

[0168] Triggering conditions

[0169] When the voice keyword recognition result or the facial expression recognition result meets the preset conditions (for example, the facial expression recognition result recognizes the user's confused or frowning expression), a multimodal fusion judgment will be triggered.

[0170] ·Fusion computing interested in vehicle design L POI

[0171] L POI = voice ∩L XR ∩L sightline ∩L handpoint

[0172] Among them, L voice For speech keyword recognition results; L XR For the icon recognition results of in-vehicle devices; L sightline The result of gaze recognition; L handpoint This is the result of gesture recognition.

[0173] The intersections are calculated sequentially according to the above priority order until an accurate and unique target functional region is obtained, which is then directly fed back to the VPA. This priority order is set because the confidence level of the speech keyword recognition result is the highest, resulting in a more accurate target functional region.

[0174] The above fusion calculation process is as follows:

[0175] Determine L voice If a unique target functional area can be obtained, then the target functional area is obtained directly and fed back to the VPA; if multiple target functional areas are obtained, then based on the multiple target functional areas, the next priority L is determined. XR Make a judgment; if L voice If the target functional area cannot be obtained from L, then... XR Begin making a judgment.

[0176] For L XR Make a judgment, if based on L XR If a unique target functional area can be obtained, then the target functional area is directly obtained and fed back to the VPA; if based on L... XR If multiple target functional regions are obtained, then based on these multiple target functional regions, the next priority L is determined. sightline Make a judgment; if based on L XR If the target functional area cannot be obtained, then proceed from L. sightline Begin making a judgment.

[0177] For L sightline and L handpoint The judgment process is the same as the above L voice and L XR The judgment process is similar and will not be repeated here.

[0178] Step 2, VPA interactive confirmation of vehicle settings (vehicle settings, i.e., target functional area).

[0179] After determining the target functional area through multimodal fusion, the target functional area is sent to the VPA. The VPA can then further confirm with the user, for example, by providing a voice prompt such as, "We see you seem a bit confused and are focusing on the target functional area. Would you like more detailed help information about it?" Once the user confirms, the corresponding operation instructions for the confirmed target functional area are fed back to the wearable electronic device, which then displays the corresponding instructions (e.g., XR glasses).

[0180] Step 3: Display vehicle design content using mixed reality (vehicle design content, i.e., operation instructions corresponding to the target functional area).

[0181] The system retrieves relevant operation instructions from a knowledge graph. This knowledge graph is stored in the vehicle's infotainment system or a server database. The operation instructions can include equipment maintenance and extension information, equipment operation animations, etc. The retrieved operation instructions are then overlaid onto the target functional area of ​​the XR glasses and displayed in the real-world scene.

[0182] For example, when a user looks at the various button icons on the center console (such as windshield wipers, headlight controls, etc.), the icon content and detailed explanations are displayed; when the user looks at the chassis, animations of the motor and engine running are shown; when the user looks at the air conditioning vents, animations of the airflow are displayed; when the user looks at the sunroof and curtain, animations of the sunroof and curtain opening are overlaid, with labels next to them; when the user looks at the massage seats, arrows indicate the seat's condition buttons; when the user mentions the fragrance system, an animation simulating the fragrance system circulating inside the car is shown, following the user's gaze.

[0183] In the above embodiments, based on the recognition results of user operation commands in the cockpit and / or the recognition results of in-vehicle electronic devices, corresponding target functional areas are selected, making the selected target functional areas more accurate and accurately identifying the target functional areas of interest to the user; the operation instruction information corresponding to the target functional area is sent to the wearable electronic device to prompt the user, so that the user can understand the operation instruction information of the target functional area in a timely manner and operate the target functional area according to the operation instruction information, thereby improving the user experience.

[0184] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0185] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0186] Based on the same inventive concept, corresponding to any of the above embodiments, this disclosure also provides a vehicle-side solution.

[0187] refer to Figure 5 , Figure 5 This is a schematic diagram of the vehicle end structure according to an embodiment of the present disclosure, including:

[0188] The first recognition result acquisition module 301 is configured to recognize and process the operation commands of the user in the cockpit to obtain the first recognition result;

[0189] The second recognition result receiving module 302 is configured to receive the second recognition result of the wearable electronic device for the vehicle device icon;

[0190] The target functional area determination module 303 is configured to filter target functional areas corresponding to the first identification result and / or the second identification result from at least one pre-defined functional area;

[0191] The operation instruction information acquisition module 304 is configured to acquire operation instruction information corresponding to the target functional area;

[0192] The operation instruction information sending module 305 is configured to send the operation instruction information to the wearable electronic device so that the wearable electronic device can prompt the user with the operation instruction information.

[0193] In some embodiments, the target functional area determination module 303 includes:

[0194] The multimodal fusion judgment triggering unit is configured to trigger multimodal fusion judgment in response to determining that the first recognition result meets a preset condition;

[0195] The target functional region determination unit is configured to perform judgment processing on the first identification result and / or the second identification result through the multimodal fusion judgment, and determine the target functional region from at least one pre-defined functional region;

[0196] The target functional area confirmation unit is configured to send the target functional area to the in-vehicle virtual personal assistant for confirmation of the target functional area.

[0197] In some embodiments, the first recognition result includes at least one of the following: voice keyword recognition result, facial expression recognition result;

[0198] The multimodal fusion judgment triggering unit includes:

[0199] The multimodal fusion judgment triggering subunit is configured to trigger multimodal fusion judgment in response to determining that there is a voice keyword in the voice keyword recognition result that is consistent with the at least one functional region, or that there is a preset expression in the expression recognition result.

[0200] In some embodiments, the number of the first recognition results is multiple, and the first recognition results include at least two of the following: voice keyword recognition results, gaze recognition results, gesture recognition results, and facial expression recognition results;

[0201] The target functional area determination unit includes:

[0202] The target functional area determination subunit is configured to perform judgment processing on at least two first identification results in a pre-set priority order through the multimodal fusion judgment;

[0203] or,

[0204] The target functional area determination subunit is configured to sequentially determine at least one first identification result and the second identification result according to a pre-set priority order through the multimodal fusion judgment.

[0205] In some embodiments, the first recognition result includes: voice keyword recognition result, gaze recognition result, gesture recognition result, and facial expression recognition result; the second recognition result includes: vehicle device icon recognition result;

[0206] The target functional area determination unit includes:

[0207] The target functional region determination subunit is configured to sequentially judge the first identification result and the second identification result according to a preset priority order through the multimodal fusion judgment, and determine the target functional region from at least one pre-defined functional region.

[0208] The preset priority order is the priority order of the voice keyword recognition result, the device icon recognition result, the gaze recognition result, and the gesture recognition result.

[0209] In some embodiments, the target functional area determination subunit is specifically configured as follows:

[0210] In response to determining that there is a voice keyword in the voice keyword recognition result that matches the keyword of the at least one functional region, the functional region that matches the voice keyword is designated as the target functional region; in response to determining that there is no voice keyword in the voice keyword recognition result that matches the keyword of the at least one functional region, the vehicle device icon recognition result is judged; in response to determining that there is a vehicle device icon in the vehicle device icon recognition result that matches one functional region, the functional region corresponding to the vehicle device icon is designated as the target functional region; in response to determining that there is a vehicle device icon in the vehicle device icon recognition result that matches at least two functional regions, the gaze recognition result is judged based on the at least two functional regions. The following steps are taken: 1. In response to determining that there is no vehicle device icon with a functional area in the vehicle device icon recognition result, the gaze recognition result is judged; 2. In response to determining that there is a functional area being looked at in the gaze recognition result, the functional area being looked at is taken as the target functional area; 3. In response to determining that there are at least two functional areas being looked at in the gaze recognition result, the gesture recognition result is judged based on the at least two functional areas being looked at; 4. In response to determining that there is no functional area being looked at in the gaze recognition result, the gesture recognition result is judged; 5. In response to determining that there is a functional area being pointed at in the gesture recognition result, the functional area being pointed at is taken as the target functional area.

[0211] In some embodiments, before the target functional area determination module 303, the following is also included:

[0212] The face recognition result acquisition unit is configured to obtain the face recognition result by analyzing and processing the gaze recognition result, the gesture recognition result, and the expression recognition result;

[0213] The voice region location recognition result acquisition unit is configured to obtain the voice region location recognition result by performing voice region judgment on the voice keyword recognition result;

[0214] The matching judgment unit is configured to perform judgment processing on the face recognition result and the voice zone position recognition result to obtain a matching judgment result, so as to determine that the first recognition result is the recognition result data of the same user in the cabin.

[0215] For ease of description, the vehicle-mounted terminal is described in terms of its functions, divided into various modules. Of course, in implementing this disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0216] The vehicle-mounted terminal in the above embodiments is used to implement the corresponding method for obtaining operation instructions based on the cockpit area in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0217] Based on the same inventive concept, corresponding to any of the above-described embodiments, this disclosure also provides a wearable electronic device.

[0218] refer to Figure 6 , Figure 6 This is a schematic diagram of the structure of a wearable electronic device according to an embodiment of the present disclosure, including:

[0219] The second recognition result acquisition module 401 is configured to recognize the vehicle device icon collected in front of the user in the cabin and obtain the second recognition result.

[0220] The second identification result sending module 402 is configured to send the second identification result to the vehicle terminal so that the vehicle terminal can obtain corresponding operation instruction information based on the second identification result and the first identification result obtained by the vehicle terminal;

[0221] The operation instruction information receiving module 403 is configured to receive the operation instruction information from the vehicle end;

[0222] The operation instruction information display module 404 is configured to broadcast the operation instruction information via voice or overlay it onto a real-world scene.

[0223] For ease of description, the wearable electronic devices described above are divided into various modules based on their functions. Of course, in implementing this disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0224] The wearable electronic device described above is used to implement the display method of the corresponding operation instruction in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0225] Based on the same inventive concept, corresponding to any of the above embodiments, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the method for obtaining operation instructions based on the cockpit area or the method for displaying the operation instructions as described in any of the above embodiments.

[0226] Figure 7This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0227] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0228] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0229] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0230] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB (Universal Serial Bus), network cable, etc.) or wireless means (such as mobile network, WIFI (Wireless Fidelity), Bluetooth, etc.).

[0231] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0232] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0233] The electronic devices described above are used to implement the corresponding methods for obtaining operation instructions based on the cockpit area or the display methods for the operation instructions in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0234] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, which are used to cause the computer to execute the method for obtaining operation instructions based on the cockpit area or the method for displaying operation instructions as described in any of the above embodiments.

[0235] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0236] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the method for obtaining operation instructions based on the cockpit area or the method for displaying operation instructions as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0237] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a vehicle, including the vehicle end, or electronic device, or storage medium in the above embodiments, wherein the vehicle device implements the method for obtaining operation instructions based on the cockpit area or the method for displaying the operation instructions as described in any of the above embodiments.

[0238] The vehicle described in the above embodiments is used to implement the method for obtaining operation instructions based on the cabin area or the method for displaying operation instructions as described in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0239] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this disclosure as described above, which are not provided in detail for the sake of brevity.

[0240] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this disclosure, the provided drawings may or may not show well-known power / ground connections to integrated circuit (IC) chips and other components. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this disclosure, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this disclosure will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuitry) have been set forth to describe exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that the embodiments of this disclosure may be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0241] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0242] This disclosure is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for obtaining an operation instruction based on a cockpit area, characterized in that, The method comprises: The operation instruction of the user in the cabin is identified to obtain a first identification result; Receiving a second identification result of a vehicle-mounted device icon by a wearable electronic device; wherein the second identification result is a vehicle-mounted device icon identification result obtained by identifying the icons on the vehicle-mounted device and display screen collected by the sensors on the wearable electronic device in the vicinity of the user or in front of the line of sight of the user; From the at least one function area pre-divided by calibration, the target function area corresponding to the first identification result and / or the second identification result is screened; Obtaining operation instruction information corresponding to the target function area; The operation instruction information is sent to the wearable electronic device to prompt the user with the operation instruction information by the wearable electronic device; The target function area corresponding to the first identification result and / or the second identification result is determined from the at least one function area pre-divided by calibration, comprising: In response to determining that the first identification result meets the pre-set condition, a multi-modal fusion judgment is triggered; The first identification result and / or the second identification result is judged by the multi-modal fusion judgment to determine the target function area from the at least one function area pre-divided by calibration; The target function area is sent to the vehicle-mounted virtual personal assistant for confirmation of the target function area; The first identification result comprises at least one of the following: voice keyword identification result, expression identification result; In response to determining that the first identification result meets the pre-set condition, a multi-modal fusion judgment is triggered, comprising: In response to determining that there is a voice keyword consistent with the at least one function area in the voice keyword identification result, or there is a pre-set expression in the expression identification result, a multi-modal fusion judgment is triggered.

2. The method of claim 1, wherein, The number of the first identification result is multiple, and the first identification result comprises at least two of the following: voice keyword identification result, line of sight identification result, gesture identification result and expression identification result; The first identification result is judged by the multi-modal fusion judgment, comprising: At least two first identification results are judged by the multi-modal fusion judgment in turn according to a pre-set priority order; Or, The first identification result and the second identification result are judged by the multi-modal fusion judgment, comprising: At least one first identification result and the second identification result are judged by the multi-modal fusion judgment in turn according to a pre-set priority order.

3. The method of claim 1, wherein, The first identification result comprises: voice keyword identification result, line of sight identification result, gesture identification result and expression identification result; the second identification result comprises: vehicle-mounted device icon identification result; The first identification result and / or the second identification result is judged by the multi-modal fusion judgment to determine the target function area from the at least one function area pre-divided by calibration, comprising: The first recognition result and the second recognition result are sequentially processed according to the preset priority order through the multi-modal fusion judgment, and a target functional area is determined from at least one functional area divided by a preset calibration. The preset priority order is the priority order of the voice keyword recognition result, the device icon recognition result, the line-of-sight recognition result, and the gesture recognition result.

4. The method of claim 3, wherein, The first recognition result and the second recognition result are sequentially processed according to the preset priority order through the multi-modal fusion judgment, and a target functional area is determined from at least one functional area divided by a preset calibration. In response to determining that the voice keyword recognition result includes a voice keyword consistent with a keyword of the at least one functional area, the functional area with the consistent voice keyword is taken as the target functional area. In response to determining that the voice keyword recognition result does not include a voice keyword consistent with a keyword of the at least one functional area, the vehicle-mounted device icon recognition result is processed. In response to determining that the vehicle-mounted device icon recognition result includes a vehicle-mounted device icon of one functional area, the functional area corresponding to the vehicle-mounted device icon is taken as the target functional area. In response to determining that the vehicle-mounted device icon recognition result includes vehicle-mounted device icons of at least two functional areas, the line-of-sight recognition result is processed based on the at least two functional areas. In response to determining that the vehicle-mounted device icon recognition result does not include a vehicle-mounted device icon of a functional area, the line-of-sight recognition result is processed. In response to determining that the line-of-sight recognition result includes a functional area looked at by a line of sight, the functional area looked at by the line of sight is taken as the target functional area. In response to determining that the line-of-sight recognition result includes functional areas looked at by at least two lines of sight, the gesture recognition result is processed based on the at least two functional areas looked at by the lines of sight. In response to determining that the line-of-sight recognition result does not include a functional area looked at by a line of sight, the gesture recognition result is processed. In response to determining that the gesture recognition result includes a functional area pointed at by a finger, the functional area pointed at by the finger is taken as the target functional area.

5. The method of claim 3, wherein, Before the target functional area corresponding to the first recognition result and / or the second recognition result is filtered from the at least one functional area divided by a preset calibration, the method further includes: A face recognition result is obtained by analyzing and processing the line-of-sight recognition result, the gesture recognition result, and the expression recognition result. An audio area position recognition result is obtained by performing audio area judgment on the voice keyword recognition result. The face recognition result and the audio area position recognition result are processed to obtain a matching judgment result, which is used to determine that the first recognition result is recognition result data of a same user in the cabin.

6. A method of displaying an operation instruction, characterized by The method includes: The vehicle-mounted device icon collected in front of the user in the cabin is identified to obtain a second identification result; wherein the second identification result is obtained by collecting information data through a sensor on the wearable electronic device, identifying the icons of the vehicle-mounted device and the display screen collected near the user or in front of the user's line of sight, and obtaining the vehicle-mounted device icon identification result; The second identification result is sent to the vehicle end, so that the vehicle end obtains corresponding operation instruction information according to the second identification result and the first identification result obtained by the vehicle end according to the method of any one of claims 1 to 5; The operation instruction information of the vehicle end is received; The operation instruction information is voice broadcasted or superimposed and displayed on the real scene.

7. A vehicle end characterized by, Comprising: The first identification result acquisition module is configured to identify the operation instruction of the user in the cabin to obtain a first identification result; The second identification result receiving module is configured to receive the second identification result of the wearable electronic device on the vehicle-mounted device icon; wherein the second identification result is obtained by collecting information data through a sensor on the wearable electronic device, identifying the icons of the vehicle-mounted device and the display screen collected near the user or in front of the user's line of sight, and obtaining the vehicle-mounted device icon identification result; The target function area determination module is configured to filter the target function area corresponding to the first identification result and / or the second identification result from at least one function area divided and calibrated in advance; The operation instruction information acquisition module is configured to acquire the operation instruction information corresponding to the target function area; The operation instruction information sending module is configured to send the operation instruction information to the wearable electronic device, so that the wearable electronic device prompts the user with the operation instruction information; The target function area determination module comprises: The multi-modal fusion judgment triggering unit is configured to trigger multi-modal fusion judgment in response to determining that the first identification result meets a pre-set condition; The target function area determination unit is configured to determine the target function area from at least one function area divided and calibrated in advance by judging and processing the first identification result and / or the second identification result through the multi-modal fusion judgment; The target function area confirmation unit is configured to send the target function area to the vehicle-mounted virtual personal assistant for confirmation of the target function area; The first identification result comprises at least one of the following: voice keyword identification result, expression identification result; The multi-modal fusion judgment triggering unit comprises: The multi-modal fusion judgment triggering sub-unit is configured to trigger multi-modal fusion judgment in response to determining that there is a voice keyword consistent with the at least one function area in the voice keyword identification result, or there is a pre-set expression in the expression identification result.

8. A wearable electronic device, comprising: Comprising: The second identification result acquisition module is configured to perform identification processing on the vehicle-mounted device icons collected in front of the user in the cabin to obtain a second identification result; wherein the second identification result is a vehicle-mounted device icon identification result obtained by performing identification processing on the icons on the vehicle-mounted device and the display screen collected near the user or in front of the user's line of sight through information data collection by a sensor on the wearable electronic device; The second identification result sending module is configured to send the second identification result to the vehicle end of claim 7, so that the vehicle end obtains corresponding operation instruction information according to the second identification result and the first identification result obtained by the vehicle end; The operation instruction information receiving module is configured to receive the operation instruction information of the vehicle end; The operation instruction information display module is configured to perform voice broadcast or superimposed display on the real scene.

9. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium stores computer instructions for causing a computer to execute the method of any one of claims 1 to 6.

10. A vehicle characterized by comprising: The vehicle end of claim 7 or the storage medium of claim 9 is included.

Images