User intention recognition and model training method and device based on multiple data modalities

Abstract

The invention provides a user intention recognition and model training method and device based on multiple data modalities, and relates to the technical field of data processing, in particular to the technical field of intelligent cabins, automatic driving, computer vision and large models. According to the specific implementation scheme, the method comprises the steps of obtaining user data of a user in multiple data modes, and determining environment information of an environment where the user is located and / or behavior habit information of the user; wherein the behavior habit information represents a behavior control mode when the user operates the vehicle; determining the weight of the user data corresponding to the data modality according to the environment information and / or the behavior habit information; and determining the user intention of the user according to the user data and the weight of the user data.

Description

Technical Field

[0001] The present disclosure relates to the fields of smart cockpit, autonomous driving, computer vision and large model technology in the field of data processing technology, and in particular to a method and device for user intent recognition and model training based on multiple data modalities. Background Art

[0002] With the development of intelligent technology, it is often necessary to obtain user voice to complete the corresponding control process. For example, in a vehicle, it is necessary to obtain user voice, recognize the user voice, obtain the user's intention, and then complete the corresponding control process based on the user's intention.

[0003] Furthermore, there is an urgent need for a solution that can accurately and effectively obtain user intentions. Summary of the invention

[0004] The present invention provides a method and device for user intent recognition and model training based on multiple data modalities.

[0005] According to a first aspect of the present disclosure, a method for identifying user intention based on multiple data modalities is provided, comprising:

[0006] Acquire user data under multiple data modes of the user, and determine environmental information of the environment in which the user is located and / or behavioral habit information of the user; wherein the behavioral habit information represents the behavioral control mode of the user when operating the vehicle;

[0007] Determining a weight of user data corresponding to a data modality according to the environment information and / or the behavior habit information;

[0008] The user intention of the user is determined according to each of the user data and the weight of the user data.

[0009] According to a second aspect of the present disclosure, a model training method for user intent recognition is provided, comprising:

[0010] Acquire data to be trained; wherein the data to be trained includes user data, environmental information, and user behavior habit information under multiple data modes; wherein the behavior habit information represents the behavior control mode of the user when operating the vehicle;

[0011] Training the initial model according to the data in the data to be trained to obtain a preset recognition model;

[0012] Among them, the preset recognition model is used to identify user data under multiple data modalities as described in the first aspect above to obtain user intention.

[0013] According to a third aspect of the present disclosure, a device for user intent recognition based on multiple data modalities is provided, comprising:

[0014] An acquisition unit, used to acquire user data under multiple data modes of the user, and determine the environment information of the environment in which the user is located and / or the behavior habit information of the user; wherein the behavior habit information represents the behavior control mode of the user when operating the vehicle;

[0015] a determination unit, configured to determine a weight of user data corresponding to a data modality according to the environment information and / or the behavior habit information;

[0016] The identification unit is used to determine the user intention of the user according to each of the user data and the weight of the user data.

[0017] According to a fourth aspect of the present disclosure, a device for model training applied to user intent recognition is provided, comprising:

[0018] An acquisition unit, used to acquire data to be trained; wherein the data to be trained includes user data, environmental information and user behavior habit information under multiple data modes; wherein the behavior habit information represents the behavior control mode of the user when operating the vehicle;

[0019] A training unit, used to train the initial model according to the data in the data to be trained to obtain a preset recognition model;

[0020] Among them, the preset recognition model is used to identify user data under multiple data modalities as described in the first aspect above to obtain user intention.

[0021] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.

[0022] According to the sixth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect.

[0023] According to a seventh aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the first aspect.

[0024] The technology according to the present disclosure solves the problem of inaccurate recognition of user intention.

[0025] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0027] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0028] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0029] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0030] Figure 4 It is a schematic diagram of the key points of the finger provided by the present disclosure Figure 1 ;

[0031] Figure 5 It is a schematic diagram of the key points of the finger provided by the present disclosure Figure 2 ;

[0032] Figure 6 is a schematic diagram according to a third embodiment of the present disclosure;

[0033] Figure 7 It is a finger pointing indication provided by the present disclosure Figure 1 ;

[0034] Figure 8 It is a finger pointing indication provided by the present disclosure Figure 2 ;

[0035] Fig. 9 It is a finger pointing indication provided by the present disclosure Figure 3 ;

[0036] Fig.10 It is a finger pointing indication provided by the present disclosure Figure 4 ;

[0037] Fig.11 It is a finger pointing indication provided by the present disclosure Figure 5 ;

[0038] Fig.12 It is a finger pointing indication provided by the present disclosure Figure 6 ;

[0039] Fig.13 It is a finger pointing indication provided by the present disclosure Figure 7 ;

[0040] Fig.14 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0041] Fig.15 is a block diagram of an apparatus for implementing user intent recognition based on multiple data modalities according to an embodiment of the present disclosure;

[0042] Fig.16 is a block diagram of another device for realizing user intent recognition based on multiple data modalities according to an embodiment of the present disclosure;

[0043] Fig.17 is a block diagram of an apparatus for implementing model training for user intent recognition according to an embodiment of the present disclosure;

[0044] Fig.18 is a block diagram of another device for model training applied to user intent recognition for implementing an embodiment of the present disclosure;

[0045] Fig.19 It is a block diagram of an electronic device used to implement the method for user intent recognition based on multiple data modalities / the method for user intent recognition based on multiple data modalities of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0047] The present disclosure provides a user intention recognition and model training method and device based on multiple data modalities, which are applied to the fields of smart cockpit, autonomous driving, computer vision and large model technology in the field of data processing technology to achieve the effect of improving the accuracy of user intention recognition.

[0048] Figure 1 Schematic diagram of the first embodiment of the present disclosure. Figure 1 As shown, the user intention recognition method based on multiple data modalities provided by the first embodiment of the present disclosure includes:

[0049] Step S101: Obtain user data in multiple data modes of the user, and determine environmental information of the user's environment and / or behavioral habit information of the user.

[0050] Specifically, user data under multiple data modes of the user may be obtained.

[0051] Among them, user data under multiple data modalities refers to various types of data collected through different sensors and technical means that reflect user behavior and status. Among them, the present disclosure does not limit the type of data modality of the acquired user data. Any type of user data collected through different sensors and technical means that reflects user behavior and status can be used as the type of data modality of the user data acquired by the present disclosure. Optionally, user data under multiple data modalities may include but is not limited to at least one of physiological data, behavioral data, voice data, visual data, location and motion data, etc.

[0052] Optionally, the process of obtaining user data under multiple data modes of the user may include: first collecting through different sensors and technical means to obtain initial user data, wherein the initial user data is all user data that can be obtained by collecting through different sensors and technical means; then performing data cleaning and preprocessing on the initial user data, optionally, data cleaning includes but is not limited to at least one of the processes of removing outliers, noise, etc., optionally, preprocessing includes but is not limited to at least one of the processes of normalization, standardization, etc.

[0053] Specifically, environmental information of the environment in which the user is located and / or behavioral habit information of the user may be determined.

[0054] Among them, the environmental information of the user's environment is the external environment in which the user is currently located and the conditions of the internal environment of the vehicle. These environmental factors may affect the user's intention. Optionally, the environmental information of the user's environment includes, but is not limited to, at least one of the following environmental information: external weather conditions, traffic conditions, light conditions, road types, and in-vehicle environment. Specifically, external weather conditions include, but are not limited to, sunny days, rainy days, snowy days, and other information. Traffic conditions include, but are not limited to, road congestion, traffic density, whether there are traffic accidents, and other information. Light conditions include, but are not limited to, daytime, night, and dim environments in tunnels, and other information. Road types include, but are not limited to, highways, urban roads, and country roads, and other information. The in-vehicle environment includes, but is not limited to, information such as temperature, humidity, and noise levels.

[0055] Optionally, the process of determining the environmental information of the environment in which the user is located may include: first collecting and acquiring initial environmental information through different sensors and technical means, wherein the initial environment is all environmental information that can be acquired through different sensors and technical means; and then performing data cleaning and preprocessing on the initial environmental information. Optionally, data cleaning includes but is not limited to at least one of the processes of removing outliers and noise. Optionally, preprocessing includes but is not limited to at least one of the processes of normalization and standardization.

[0056] Among them, behavioral habit information refers to the fixed or repetitive behavioral patterns formed by users in the daily use of the vehicle, which reflect the user's personal preferences and driving style. Optionally, behavioral habit information includes but is not limited to: at least one of driving habits, entertainment preferences, temperature preferences, seat preferences, and navigation preferences. Specifically, driving habits include but are not limited to information such as whether the user tends to drive smoothly or aggressively, whether he changes lanes or overtakes frequently. Entertainment preferences include but are not limited to information such as the type of music and radio channels that the user likes, whether he likes to play music or answer calls through the car audio system. Temperature preferences include but are not limited to information such as the range to which the user usually adjusts the temperature in the car, whether he likes to use the seat heating or ventilation function. Seat preferences include but are not limited to information such as whether the user likes to adjust the seat to a specific position, angle or inclination, and whether he likes to use functions such as seat massage. Navigation preferences include but are not limited to information such as whether the user likes to use the in-vehicle navigation system, and whether he tends to choose a specific route or avoid certain sections when using navigation.

[0057] The behavior habit information represents the behavior control mode of the user when operating the vehicle. Specifically, the behavior control mode of the user when operating the vehicle refers to the way and means by which the user controls the vehicle's driving, function operation, and interaction with other vehicle systems. Optionally, the behavior control mode of the user when operating the vehicle can be determined based on the behavior habit information described above.

[0058] Optionally, the process of determining the user behavior habit information may include: first acquiring the user's historical behavior information, and then determining the user's behavior habit information from the acquired user's historical behavior information.

[0059] Step S102: Determine the weight of the user data corresponding to the data modality based on the environmental information and / or behavioral habit information.

[0060] Specifically, based on the environmental information and / or behavioral habit information determined in step S101, the weight of the user data corresponding to the data modality obtained in step S101 can be determined.

[0061] Among them, the present application does not limit the process of determining the weight of user data corresponding to the data modality based on environmental information and / or behavioral habit information.

[0062] Optionally, the process of determining the weight of user data corresponding to the data modality based on environmental information and / or behavioral habit information may correspond to the type of information based on which it is based, i.e., environmental information and / or behavioral habit information. Optionally, if the information based on which it is based is environmental information, then the above process is to determine the weight of user data corresponding to the data modality based on environmental information; optionally, if the information based on which it is based is behavioral habit information, then the above process is to determine the weight of user data corresponding to the data modality based on behavioral habit information; optionally, if the information based on which it is based is environmental information and behavioral habit information, then the above process is to determine the weight of user data corresponding to the data modality based on environmental information and behavioral habit information.

[0063] Optionally, if the type of information used is the same, the weight of the user data corresponding to the data modality can also be determined according to different confirmation processes. Optionally, if the type of information used is environmental information, the process of determining the weight of the user data corresponding to the data modality can be combined with at least one of the information such as the noise value and the ambient brightness value determined according to the environmental information; Optionally, if the type of information used is behavioral habit information, the process of determining the weight of the user data corresponding to the data modality can be combined with at least one of the information such as the number of historical operations of the user in different behavior control modes determined according to the behavioral habit information; Optionally, if the type of information used is environmental information and behavioral habit information, the process of determining the weight of the user data corresponding to the data modality can be combined with at least one of the preset weight processing model, the preset mapping relationship, and the like.

[0064] Among them, the environmental information and / or behavioral habit information used in the weight determination process corresponds to the environmental information and / or behavioral habit information obtained in step S101, that is, if only environmental information is obtained in step S101, then in this step, the weight of the user data corresponding to the data modality is determined based on the environmental information; if only behavioral habit information is obtained in step S101, then in this step, the weight of the user data corresponding to the data modality is determined based on the behavioral habit information; if both environmental information and behavioral habit information are obtained in step S101, then in this step, the weight of the user data corresponding to the data modality is determined based on the environmental information and the behavioral habit information.

[0065] Step S103: Determine the user intention of the user according to each user data and the weight of the user data.

[0066] Specifically, the user intention of the user may be determined according to the respective user data acquired in step S101 and the weight of the user data acquired in step S102 .

[0067] Among them, user intention refers to the goals, needs or desired behaviors that users hope to achieve when using the vehicle. User intention is the core of the user's interaction with the vehicle system, and it reflects the user's expectations and preferences when using the vehicle.

[0068] Optionally, after determining the user's intention, the vehicle can be controlled accordingly based on the determined user intention. For example, when the user expresses the intention to navigate to a destination, the vehicle system can automatically plan the optimal driving route and provide real-time traffic information when necessary; for example, when the user expresses the intention to adjust the temperature in the car, the vehicle system can automatically adjust the settings of the air-conditioning system to maintain a comfortable temperature in the car.

[0069] Among them, in the process of performing corresponding control processing on the vehicle according to the determined user intention, if the accuracy of identifying the user's intention is improved, the vehicle system can provide more personalized and intelligent services and responses, thereby improving the user's driving experience and satisfaction.

[0070] Among them, the present application does not limit the process of determining the user intention based on the user data and the weight of the user data. Optionally, feature extraction processing can be first performed on each user data based on a preset recognition model, and then recognition processing can be performed based on the preset recognition model according to the extracted features and the weight of the user data to obtain the user intention.

[0071] In the embodiments of the present disclosure, user data of the user under multiple data modalities is obtained, and environmental information of the user's environment and / or behavioral habit information is determined, and the weight of the user data corresponding to the data modality is determined based on the environmental information and / or behavioral habit information, and the user's user intention is determined based on each user data and the weight of the user data. Among them, the user intention is identified based on each user data under multiple data modalities, which can improve the accuracy of user intention recognition. On this basis, the weight of the user data corresponding to the data modality is determined based on the environmental information and / or behavioral habit information, and the user intention is identified based on each user data and the weight of the user data, thereby further improving the accuracy of user intention recognition.

[0072] It should be noted that the head model in this embodiment is not a head model for a specific user and cannot reflect the personal information of a specific user. It should be noted that the two-dimensional face image in this embodiment comes from a public data set.

[0073] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0074] In order to enable readers to more deeply understand the implementation principle of the present disclosure, the following Figure 2-Figure 19 right Figure 1 The illustrated embodiment is further refined.

[0075] In some embodiments, user data in multiple data modalities include: user voice, facial image, and hand image.

[0076] Among them, the description of user data under various data modes can refer to the description in step S101, which will not be repeated here.

[0077] Specifically, user voice refers to the sound information emitted by the user when using the vehicle. Such sound information may include the user's voice commands, conversation content, emotional expressions, etc. Facial images refer to the facial image information of the user captured by the vehicle-mounted camera when the user is using the vehicle. Such image information may reflect the user's facial expression, eye direction, head posture, etc. Hand images refer to the user's gesture action information captured by the vehicle-mounted camera or gesture sensor when the user is using the vehicle. Such image information may reflect the user's hand posture, movement trajectory, etc.

[0078] Among them, if the user data under multiple data modes includes user voice, facial image, and hand image, the user data under each behavior control mode can be comprehensively obtained, thereby improving the accuracy of determining the user's user intention based on the obtained user data.

[0079] In some embodiments, the process of determining the weight of user data corresponding to the data modality according to the environmental information may include: first determining a noise value according to the environmental information. If it is determined that the noise value is greater than a preset noise threshold, increasing the weight of the facial image and / or the weight of the hand image, and reducing the weight of the user's voice.

[0080] The noise value represents the noise level of the environment in which the user is located. Specifically, based on the description of the environmental information of the environment in which the user is located in step S101, the noise value can be extracted from the environmental information of the environment in which the user is located.

[0081] Among them, the preset noise threshold is a preset noise threshold for determining the weight of user data corresponding to the data modality. Optionally, the preset noise threshold can be preset according to user needs, and optionally, can also be preset according to the vehicle model.

[0082] Specifically, if it is determined that the noise value is greater than the preset noise threshold, it is determined that the noise in the environment where the user is located is relatively large, which will affect the accuracy of the user's intention determined based on the user's voice, and therefore, the weight of the facial image and / or the weight of the hand image can be increased, and the weight of the user's voice can be reduced. Conversely, if it is determined that the noise value is less than or equal to the preset noise threshold, it is determined that the noise in the environment where the user is located is relatively small, which will not affect the accuracy of the user's intention determined based on the user's voice, and therefore, the weight of the user's voice can be increased, and the weight of the facial image and / or the weight of the hand image can be reduced.

[0083] Among them, in the process of determining the weight of user data corresponding to the data modality based on environmental information, increasing or decreasing the weight of the user voice according to the determined noise value can reduce the impact of excessive noise value on the accuracy of the recognized user voice, thereby effectively improving the accuracy of determining user intentions.

[0084] In some embodiments, the process of determining the weight of user data corresponding to the data modality according to the environmental information may include: first determining the ambient brightness value according to the environmental information. If it is determined that the ambient brightness value is less than a preset brightness threshold, increasing the weight of the user's voice, and reducing the weight of the facial image and / or the weight of the hand image.

[0085] The ambient brightness value represents the brightness of the environment the user is in. Specifically, based on the description of the ambient information of the environment the user is in in step S101, the ambient brightness value can be extracted from the ambient information of the environment the user is in.

[0086] Among them, the preset ambient brightness value is a preset ambient brightness threshold for determining the weight of user data corresponding to the data modality. Optionally, the preset ambient brightness threshold can be preset according to user needs, and optionally, can also be preset according to the vehicle model.

[0087] Specifically, if it is determined that the ambient brightness value is less than the preset ambient brightness threshold, it is determined that the ambient brightness of the environment in which the user is located is relatively small, which will affect the accuracy of the user's intention determined based on the facial image and / or the hand image, and therefore, the weight of the user's voice can be increased, and the weight of the facial image and / or the weight of the hand image can be reduced. Conversely, if it is determined that the ambient brightness value is greater than or equal to the preset ambient brightness threshold, it is determined that the ambient brightness of the environment in which the user is located is relatively large, which will not affect the accuracy of the user's intention determined based on the facial image and / or the hand image, and therefore, the weight of the user's voice can be reduced, and the weight of the facial image and / or the weight of the hand image can be increased.

[0088] Among them, in the process of determining the weight of user data corresponding to the data modality based on environmental information, the weight of the facial image and / or the weight of the hand image is increased or decreased according to the determined ambient brightness value, which can reduce the impact of the low ambient brightness value on the accuracy of the recognized facial image and / or hand image, thereby effectively improving the accuracy of determining the user's intention.

[0089] In some embodiments, the process of determining the weight of user data corresponding to the data modality according to the behavior habit information may include: first determining the number of historical operations of each user in different behavior control modes according to the behavior habit information, and then increasing the weight of the user data in the behavior control mode with the largest number of historical operations, and reducing the weight of other user data.

[0090] For the description of the behavior control method, please refer to the description in step S101, which will not be repeated here. The different behavior control methods include the following behavior control methods: voice control method, facial expression control method, and gesture control method. Specifically, the voice control method is a behavior control method for controlling the vehicle by commands recognized from the user's voice; the facial expression control method is a behavior control method for controlling the vehicle by commands recognized from facial images; and the gesture control method is a behavior control method for controlling the vehicle by commands recognized from hand images.

[0091] Among them, the more times the user performs historical operations on one or more of the various behavior control modes, the greater the weight of the user data corresponding to the behavior control mode, which can improve the accuracy of user intention recognition.

[0092] Optionally, the weight of the user data in the behavior control mode with the largest number of historical operations can be increased, and the weight of other user data can be reduced. For example, if it is determined that the user has performed the largest number of historical operations in the voice control mode, the weight of the user's voice can be increased, and the weight of the facial image and / or the weight of the hand image can be reduced.

[0093] Optionally, the weight of each user data may be determined based on the size relationship between the number of historical operations of each user in different behavior control modes. For example, the weight of the user data in the behavior control mode with the largest number of historical operations may be set to the maximum value among the weights, the weight of the user data in the behavior control mode with the smallest number of historical operations may be set to the minimum value among the weights, and the weight of the user data in other behavior control modes may be set to the middle value among the weights, etc.

[0094] Among them, in the process of determining the weight of user data corresponding to the data modality based on behavioral habit information, increasing or decreasing the weight of each user data according to the determined number of historical operations of the user in different behavioral control methods can make the identified user intention closer to the user's behavioral habits, thereby improving the accuracy of user intention identification.

[0095] In some embodiments, the process of determining the weight of user data corresponding to the data modality based on the environmental information and the behavioral habit information may include: firstly performing feature fusion processing on the environmental information and the behavioral habit information based on a preset weight processing model to obtain a fused feature, and then processing the fused feature based on the preset weight processing model to obtain the weight of the user data corresponding to the data modality.

[0096] The preset weight processing model is a pre-built model for determining the weight of user data corresponding to each data modality. The present application does not limit the preset weight processing model, and any pre-built model for determining the weight of user data corresponding to each data modality can be used as the preset weight processing model provided by the present application.

[0097] Specifically, the preset weight processing model provided in the embodiment of the present disclosure first performs feature fusion processing on the environmental information and the behavioral habit information based on the preset weight processing model to obtain fusion features. The fusion features are then processed based on the preset weight processing model to obtain the weights of the user data corresponding to the data modality, so that the preset weight processing model can learn the comprehensive features of each data modality on the basis of learning the features of each data modality, thereby improving the accuracy of the weights of the user data corresponding to the obtained data modality.

[0098] Among them, in the process of determining the weight of the user data corresponding to the data modality based on the environmental information and behavioral habit information, the weight of the user data corresponding to the data modality can be accurately and efficiently determined according to the preset weight processing model. Furthermore, in the process of determining the weight of the user data corresponding to the data modality, the preset weight processing model is based on which, in the process of generating the weight of the user data corresponding to the data modality, feature fusion processing is performed, thereby further improving the accuracy of the weight of the user data corresponding to the determined data modality. Based on the above description, on the basis of improving the accuracy of the weight of the user data corresponding to the data modality, the accuracy of user intent recognition is improved.

[0099] In some embodiments, the process of determining the weight of user data corresponding to a data modality based on environmental information and behavioral habit information may include: determining the weight of user data corresponding to a data modality based on environmental information and behavioral habit information based on a preset mapping relationship.

[0100] The preset mapping relationship is a pre-built mapping relationship for determining the user data corresponding to each data modality. Optionally, the preset mapping relationship can be preset according to user needs, and optionally, can also be preset according to the vehicle model.

[0101] Among them, the preset mapping relationship represents the corresponding relationship among the weights of environmental information, behavioral habit information, and user data.

[0102] Among them, in the process of determining the weight of user data corresponding to the data modality based on environmental information and behavioral habit information, the weight of user data corresponding to the data modality can be efficiently determined according to the preset mapping relationship, thereby further improving the accuracy of user intention recognition.

[0103] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure. Figure 2 As shown, the process of determining the user intention of a user according to each user data and the weight of the user data provided by the second embodiment of the present disclosure includes:

[0104] Step S201: Perform feature extraction processing on user data of user speech based on a preset recognition model to obtain speech features; and perform feature extraction processing on user data of facial images based on a preset recognition model to obtain lip movement features.

[0105] The preset recognition model is a pre-built model that recognizes user intent based on each user's data and the weight of the user's data. The present disclosure does not limit the preset recognition model, and any pre-built model that can recognize user intent based on each user's data and the weight of the user's data can be used as the preset recognition model provided by the present application.

[0106] Specifically, the user data of the user's voice can be subjected to feature extraction processing based on a preset recognition model to obtain voice features. The present application does not limit the process of extracting features from the user data of the user's voice based on the preset recognition model to obtain voice features. Optionally, an algorithm for feature recognition of the user's voice can be configured in the preset recognition model, wherein the present application does not limit the algorithm for feature recognition of the user's voice configured in the preset recognition model, and optionally includes but is not limited to: at least one of an algorithm based on dynamic time warping (DTW), an algorithm based on a hidden Markov model (HMM), an algorithm based on vector quantization (VQ), an algorithm based on a neural network, and the like. Optionally, the embodiments of the present disclosure do not limit the extracted voice features, and optionally include but are not limited to at least one of Mel Frequency Cepstral Coefficients (MFCC for short), pitch, and other features.

[0107] Specifically, the user data of the facial image can be subjected to feature extraction processing based on the preset recognition model to obtain the lip movement features. The present application does not limit the process of extracting the user data of the facial image based on the preset recognition model to obtain the lip movement features. Optionally, an algorithm for feature recognition of the facial image can be configured in the preset recognition model. The present application does not limit the algorithm for feature recognition of the facial image configured in the preset recognition model. Optionally, it includes but is not limited to at least one of the classic algorithms for extracting local features of the image (Scale-Invariant Feature Transform, referred to as SIFT), the image feature extraction algorithm (Histogram of Oriented Gradient, referred to as HOG), and the like.

[0108] Step S202: Process the user data of the hand image to obtain the finger pointing in the hand image.

[0109] Specifically, the user data of the hand image may be processed to obtain the finger pointing in the hand image.

[0110] The finger pointing is the direction in which the finger is pointing obtained from the hand image information. The application does not limit the finger pointing, and any direction in which the finger is pointing obtained from the hand image information can be used as the finger pointing provided by the application.

[0111] Among them, the present application does not limit the process of processing user data of a hand image to obtain the finger pointing in the hand image. Optionally, the key points of the fingers among adjacent fingers in the hand image can be determined; the finger information is determined based on the key points of the fingers among the adjacent fingers; wherein the finger information includes a first distance, a second distance and a third distance, the first distance representing the distance between adjacent fingers in the X-axis direction of the hand image; the second distance represents the length of the previous finger among the adjacent fingers in the X-axis direction of the hand image; the third distance represents the length of the previous finger among the adjacent fingers in the Y-axis direction of the hand image; the finger pointing of the hand image is determined based on the position state of the top of the previous finger, the first distance, the second distance and the third distance; wherein the position state represents the distance relationship between the top of the previous finger and the bottom joint of the previous finger.

[0112] Among them, in the process of processing the user data of the hand image to obtain the finger pointing in the hand image, the finger pointing can be accurately and efficiently determined through the determined finger key points and the distances determined according to the finger key points, thereby further improving the efficiency and accuracy of user intention recognition.

[0113] Step S203: performing recognition processing based on a preset recognition model according to the voice features, lip movement features, finger pointing and weights of the user data to obtain the user intention.

[0114] Specifically, the user intention can be obtained by performing recognition processing based on a preset recognition model according to the voice features and lip movement features obtained in step S201, the finger pointing obtained in step S202, and the weight of the user data.

[0115] Among them, the present application does not limit the process of obtaining user intention by performing recognition processing based on a preset recognition model according to the weights of voice features, lip movement features, finger pointing and user data. Optionally, voice features, lip movement features and finger pointing can be recognized and processed based on a preset recognition model and the weights of user data to obtain initial intention information; the initial intention information is adjusted according to the user intention obtained last time to obtain the current user intention.

[0116] Among them, the present application does not limit the process of identifying and processing voice features, lip movement features and finger pointing based on a preset recognition model and the weight of user data to obtain initial intention information. Optionally, feature fusion processing can be performed on voice features, lip movement features and finger pointing based on the weight of user data, and then the fused features can be identified and processed based on the preset recognition model to obtain the initial intention.

[0117] Among them, the user intention at the current moment has a certain correlation with the user intention at the previous moment. Therefore, after obtaining the initial intention, the initial intention can be adjusted according to the user intention obtained last time to obtain the current user intention to improve the accuracy of the user intention.

[0118] In the process of determining the current user intention, the initial intention is adjusted according to the user intention obtained last time, which can improve the accuracy of user intention recognition.

[0119] In the embodiment of the present disclosure, feature extraction is performed on user data of user speech based on a preset recognition model to obtain speech features; and feature extraction is performed on user data of facial images based on a preset recognition model to obtain lip movement features; user data of hand images are processed to obtain finger pointing in the hand image; recognition is performed based on the speech features, lip movement features, finger pointing and weights of the user data to obtain user intention, wherein, in the process of determining the user intention of the user based on each user data and the weight of the user data, speech features and lip movement features are extracted based on the preset recognition model, and user intention is obtained, which can further improve the accuracy of user intention recognition on the basis of improving the accuracy of feature extraction, wherein, in the process of determining the finger pointing, there is no need to manually unlabel the image training set to train the model, which reduces the labor cost and time cost of identifying the finger pointing, and further reduces the labor cost and time cost of user intention recognition. Based on the above description, the embodiment of the present disclosure improves the accuracy of user intention recognition and reduces the labor cost and time cost of user intention recognition.

[0120] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure, Figure 3 As shown, the present disclosure provides a method for identifying finger pointing, the method comprising:

[0121] 301. Acquire a hand image, and determine key points of fingers among adjacent fingers in the hand image.

[0122] Exemplarily, the execution subject of this embodiment can be a controller in a vehicle, or a processor in a vehicle, or a device or apparatus for determining the position of a vehicle, or a terminal device, or a server, or other devices or apparatuses that can implement this embodiment. There is no limitation to this. The hand image is captured by a camera. The key points of the hand image are extracted or identified to obtain the key points of each finger in the hand image. Among them, if the fingers in the hand image are blocked by other fingers, the key points of the blocked fingers can be automatically deduced.

[0123] For example, a hand key point model can be used to identify the hand image to obtain the key points of each finger. If a finger in the hand image is blocked by other fingers, the key points of the blocked finger can be automatically deduced based on the hand key point model.

[0124] Figure 4 It is a schematic diagram of the key points of the finger provided by the present disclosure Figure 1 , Figure 4 As shown in FIG, after collecting the hand image, the key points of each finger in the hand image are identified. The key points are labeled as follows: Figure 4 As shown. Among them, Figure 4 The X-axis and Y-axis of the hand image are marked in FIG.

[0125] Figure 4 The key points of each finger are marked in Figure 4 As shown, the key point of the hand is key point 0; along the bottom to the top of the thumb, the key points of the thumb are key point 1, key point 2, key point 3, and key point 4; along the bottom to the top of the index finger, the key points of the index finger are key point 5, key point 6, key point 7, and key point 8; along the bottom to the top of the middle finger, the key points of the middle finger are key point 9, key point 10, key point 11, and key point 12; along the bottom to the top of the ring finger, the key points of the ring finger are key point 13, key point 14, key point 15, and key point 16; along the bottom to the top of the little finger, the key points of the little finger are key point 17, key point 18, key point 19, and key point 20.

[0126] The above key point labeling method can be used to label the same key point in different hand images (images with the hand in different states).

[0127] Figure 5 It is a schematic diagram of the key points of the finger provided by the present disclosure Figure 2 , Figure 5 As shown in FIG, after collecting the hand image, the key points of each finger in the hand image are identified. The key points are labeled as follows: Figure 5 As shown. Among them, Figure 5 The X-axis and Y-axis of the hand image are marked in FIG.

[0128] in, Figure 5 The key points are labeled in the same way as Figure 4 same, Figure 5 The figure only shows the key points to make it easier to see the key points.

[0129] Figure 4 and Figure 5 However, if the hand in the hand image is in other postures, or if the fingers in the hand image are blocked by other fingers, the key points of the blocked fingers can be automatically deduced based on the hand key point model.

[0130] In this embodiment, only the key points of the fingers in the adjacent fingers can be extracted from the obtained key points. For example, the key points of the index finger and the middle finger are extracted; or the key points of the thumb and the index finger are extracted; or the key points of the middle finger and the ring finger are extracted; or the key points of the ring finger and the little finger are extracted.

[0131] For example, the previous finger among adjacent fingers is the index finger, and the next finger among adjacent fingers is the middle finger. In the smart cockpit, when the user needs to issue a control command to the smart cockpit, the user issues a gesture; the smart cockpit collects the user's hand image, and then the smart cockpit can extract the key points of the index finger and the key points of the middle finger in the hand image.

[0132] 302. Determine finger information based on key points of fingers among adjacent fingers; wherein the finger information includes a first distance and a second distance, the first distance represents the distance between adjacent fingers in the X-axis direction of the hand image; the second distance represents the length of a previous finger among the adjacent fingers in the X-axis direction of the hand image.

[0133] Exemplarily, based on the key points of each finger in the adjacent fingers, the distance between the adjacent fingers in the X-axis direction of the hand image is calculated, thereby obtaining the first distance. Figure 4 The distance between the key point 5 of the middle index finger and the key point 10 of the middle finger in the X-axis direction of the hand image is used to obtain the first distance.

[0134] Based on the key point of the previous finger among the adjacent fingers, the length of the previous finger in the X-axis direction of the hand image is calculated, thereby obtaining the second distance. Figure 4 The distance between the key point 5 of the middle finger and the key point 8 of the index finger in the X-axis direction of the hand image is used to obtain the second distance.

[0135] Furthermore, the position state of the top of the previous finger can also be determined, wherein the position state represents the distance relationship between the top of the previous finger and the bottom joint of the previous finger. For example, the position state represents that the top of the previous finger is far away from the bottom joint of the previous finger, or the position state represents that the top of the previous finger is close to the bottom joint of the previous finger.

[0136] For example, the adjacent fingers are the index finger and the middle finger, and the preceding finger is the index finger.

[0137] For example, the top of the previous finger is a key point, and the bottom joint of the previous finger is also a key point. The Y-axis coordinate value of the top of the previous finger on the hand image and the Y-axis coordinate value of the bottom joint of the previous finger on the hand image can be determined. If it is determined that the Y-axis coordinate value of the top of the previous finger is much smaller than the Y-axis coordinate value of the bottom joint of the previous finger, then it is determined that the position state represents that the top of the previous finger is far away from the bottom joint of the previous finger; otherwise, it is determined that the position state represents that the top of the previous finger is close to the bottom joint of the previous finger.

[0138] For example, the top of the previous finger is a key point, and the bottom joint of the previous finger is also a key point. The Y-axis coordinate value of the top of the previous finger on the hand image and the Y-axis coordinate value of the bottom joint of the previous finger on the hand image can be determined; if the Y-axis coordinate value of the top of the previous finger is determined to be much greater than the Y-axis coordinate value of the bottom joint of the previous finger, then the position state is determined to represent that the top of the previous finger is far away from the bottom joint of the previous finger; otherwise, the position state is determined to represent that the top of the previous finger is close to the bottom joint of the previous finger.

[0139] For example, the top of the previous finger is a key point, and the bottom joint of the previous finger is also a key point. The X-axis coordinate value of the top of the previous finger on the hand image and the X-axis coordinate value of the bottom joint of the previous finger on the hand image can be determined; if the X-axis coordinate value of the top of the previous finger is determined to be much greater than the X-axis coordinate value of the bottom joint of the previous finger, then the position state is determined to represent that the top of the previous finger is far away from the bottom joint of the previous finger; otherwise, the position state is determined to represent that the top of the previous finger is close to the bottom joint of the previous finger.

[0140] For example, the top of the previous finger is a key point, and the bottom joint of the previous finger is also a key point. The X-axis coordinate value of the top of the previous finger on the hand image and the X-axis coordinate value of the bottom joint of the previous finger on the hand image can be determined; if the X-axis coordinate value of the top of the previous finger is much smaller than the X-axis coordinate value of the bottom joint of the previous finger, the position state is determined to represent that the top of the previous finger is far away from the bottom joint of the previous finger; otherwise, the position state is determined to represent that the top of the previous finger is close to the bottom joint of the previous finger.

[0141] 303. Determine the finger direction of the hand image according to the position state of the top of the previous finger, the first distance, and the second distance; wherein the position state represents the distance relationship between the top of the previous finger and the bottom joint of the previous finger.

[0142] Exemplarily, a comprehensive judgment is made based on the previously obtained position state of the tip of the previous finger, the first distance, and the second distance, thereby obtaining the finger pointing of the hand image.

[0143] For example, if it is determined that the position state of the previous finger represents that the top of the previous finger is away from the bottom joint of the previous finger; and the Y-axis coordinate value of the top of the previous finger is much smaller than the Y-axis coordinate value of the bottom joint of the previous finger; and it is determined that the second distance is smaller than the first distance, then it is determined that the finger is pointing upward.

[0144] If it is determined that the position state of the previous finger represents that the top of the previous finger is away from the bottom joint of the previous finger; and the Y-axis coordinate value of the top of the previous finger is much larger than the Y-axis coordinate value of the bottom joint of the previous finger; and it is determined that the second distance is smaller than the first distance, then it is determined that the finger is pointing upward.

[0145] If it is determined that the position state of the previous finger represents that the top of the previous finger is away from the bottom joint of the previous finger; and the X-axis coordinate value of the top of the previous finger is much larger than the X-axis coordinate value of the bottom joint of the previous finger; and it is determined that the second distance is greater than the first distance, then it is determined that the finger is pointing to the right.

[0146] If it is determined that the position state of the previous finger represents that the top of the previous finger is away from the bottom joint of the previous finger; and the X-axis coordinate value of the top of the previous finger is much smaller than the X-axis coordinate value of the bottom joint of the previous finger; and it is determined that the second distance is greater than the first distance, then it is determined that the finger is pointing to the left.

[0147] In this embodiment, by obtaining the key points of the fingers in the adjacent fingers in the hand image, the positional relationship of the key points of the fingers in the adjacent fingers is determined to obtain the position state of the top of the previous finger; and the distance of the key points of the fingers in the adjacent fingers is calculated to obtain the first distance (the distance between the adjacent fingers in the X-axis direction of the hand image) and the second distance (the length of the previous finger in the X-axis direction of the hand image). Based on the above position state, the first distance and the second distance, the hand direction can be obtained; it can be seen from the introduction of the above embodiment that if the position state of the previous finger represents that the top of the previous finger is far away from the bottom joint of the previous finger; and the Y-axis coordinate value of the top of the previous finger is much smaller than the Y-axis coordinate value of the bottom joint of the previous finger; and it is determined that the second distance is less than the first distance, it can be determined that the finger is upward (for the rest of the directions, see the previous introduction); referring to the previous introduction, it can be known that the user's finger direction can be quickly identified. For example, the above adjacent fingers are the index finger and the ring finger, and the previous finger is the index finger; and the solution of this embodiment can identify the direction of the index finger.

[0148] Furthermore, there is no need for the model to identify the hand image to obtain the finger pointing, and thus there is no need to manually mark the image training set (i.e., there is no need to manually mark the finger pointing in the hand image for training the model); the labor cost and time cost of identifying the finger pointing are reduced. When identifying the finger pointing, the finger pointing can be obtained based only on the above-mentioned position state, the first distance and the second distance, and there is no need to use a model to identify the finger pointing; thus, in actual application scenarios, there is no need to deploy the model in the device, thereby reducing the computing power occupied by the device and the memory consumption of the device.

[0149] Figure 6is a schematic diagram according to the third embodiment of the present disclosure, Figure 6 As shown, the present disclosure provides a method for identifying finger pointing, the method comprising:

[0150] 601. Determine key points of fingers among adjacent fingers in the hand image.

[0151] For example, the execution subject of this embodiment may be a controller in a vehicle, or a processor in a vehicle, or a device or equipment for determining the position of a vehicle, or a terminal device, or a server, or other devices or equipment that can implement this embodiment. There is no limitation on this.

[0152] This step refers to the above step 301 and will not be described in detail.

[0153] 602. Determine a first distance according to key points of fingers among the adjacent fingers, wherein the first distance represents the distance between the adjacent fingers in the X-axis direction of the hand image.

[0154] In one example, step 402 includes: determining the distance between a first key point of a front finger and a second key point of a rear finger among adjacent fingers in the X-axis direction as a first distance; wherein the first key point is the bottom joint of the front finger, and the second key point is the key point on the rear finger adjacent to the bottom joint of the rear finger.

[0155] For example, for adjacent fingers, the bottom joint of the previous finger is determined as the first key point, and the key point on the next finger adjacent to the bottom joint of the next finger is determined as the second key point. Then, the distance between the first key point and the second key point on the X-axis of the hand image is calculated, and the first distance can be obtained.

[0156] For example, see Figure 4 , the adjacent fingers are the index finger and the middle finger, that is, the previous finger among the adjacent fingers is the index finger, and the next finger among the adjacent fingers is the middle finger. The first key point is determined to be key point 5, and the second key point is key point 10, and then the distance between key point 5 and key point 10 in the X-axis direction of the hand image is calculated as the first distance. In different hand images (images with the hand in different states), for the same key point, the labels of the key point in different images can be the same. Furthermore, no matter what the posture of the hand in the hand image is, for the case where the adjacent fingers are the index finger and the middle finger, the distance between key point 5 and key point 10 in the X-axis direction of the hand image is determined to be the first distance.

[0157] For example, see Figure 4, the adjacent fingers are the middle finger and the ring finger, that is, the previous finger among the adjacent fingers is the middle finger, and the next finger among the adjacent fingers is the ring finger. The first key point is determined to be key point 9, and the second key point is determined to be key point 14, and then the distance between key point 9 and key point 14 in the X-axis direction of the hand image is calculated as the first distance. In different hand images (images with the hand in different states), for the same key point, the labels of the key point in different images can be the same. Furthermore, no matter what the posture of the hand in the hand image is, for the case where the adjacent fingers are the middle finger and the ring finger, the distance between key point 9 and key point 14 in the X-axis direction of the hand image is determined to be the first distance.

[0158] 603. Determine a second distance according to the key points of the fingers among the adjacent fingers, wherein the second distance represents the length of the previous finger among the adjacent fingers in the X-axis direction of the hand image.

[0159] In one example, step 603 includes: determining the distance between the third key point of the previous finger and the first key point of the previous finger among adjacent fingers in the X-axis direction as the second distance; wherein the first key point is the bottom joint of the previous finger, and the third key point is the top of the previous finger.

[0160] For example, for adjacent fingers, the bottom joint of the previous finger is determined as the first key point, and the top of the previous finger is determined as the third key point. Then, the distance between the first key point and the third key point on the X-axis of the hand image is calculated, and the second distance can be obtained.

[0161] For example, see Figure 4 , the adjacent fingers are the index finger and the middle finger, that is, the previous finger among the adjacent fingers is the index finger, and the next finger among the adjacent fingers is the middle finger. The first key point is determined to be key point 5, and the third key point is determined to be key point 8, and then the distance between key point 5 and key point 8 in the X-axis direction of the hand image is calculated as the second distance. In different hand images (images with the hand in different states), for the same key point, the labels of the key point in different images can be the same. Furthermore, no matter what the posture of the hand in the hand image is, for the case where the adjacent fingers are the index finger and the middle finger, the distance between key point 5 and key point 8 in the X-axis direction of the hand image is determined to be the second distance.

[0162] For example, see Figure 4, the adjacent fingers are the middle finger and the ring finger, that is, the previous finger among the adjacent fingers is the middle finger, and the next finger among the adjacent fingers is the ring finger. The first key point is determined to be key point 9, and the third key point is determined to be key point 12, and then the distance between key point 9 and key point 12 in the X-axis direction of the hand image is calculated as the second distance. In different hand images (images with the hand in different states), for the same key point, the labels of the key point in different images can be the same. Furthermore, no matter what the posture of the hand in the hand image is, for the case where the adjacent fingers are the middle finger and the ring finger, the distance between key point 9 and key point 12 in the X-axis direction of the hand image is determined to be the second distance.

[0163] 604. Determine a third distance according to the key points of the fingers among the adjacent fingers; the third distance represents the length of the previous finger among the adjacent fingers in the Y-axis direction of the hand image.

[0164] In one example, the first distance, the second distance, and the third distance constitute the above-mentioned finger information.

[0165] In one example, step 604 includes: determining the distance between the third key point of the previous finger and the first key point of the previous finger in the adjacent fingers in the Y-axis direction as the third distance; wherein the first key point is the bottom joint of the previous finger, and the third key point is the top of the previous finger.

[0166] Exemplarily, the length of a previous finger among adjacent fingers in the Y-axis direction of the hand image may also be determined as the third distance.

[0167] For adjacent fingers, the bottom joint of the previous finger is determined as the first key point, and the top of the previous finger is determined as the third key point. Then the distance between the first key point and the third key point on the Y axis of the hand image is calculated, and the third distance can be obtained.

[0168] See also Figure 4 , the adjacent fingers are the index finger and the middle finger, that is, the previous finger among the adjacent fingers is the index finger, and the next finger among the adjacent fingers is the middle finger. The first key point is determined to be key point 5, and the third key point is determined to be key point 8, and then the distance between key point 5 and key point 8 in the Y-axis direction of the hand image is calculated as the third distance. In different hand images (images with the hand in different states), for the same key point, the labels of the key point in different images can be the same. Furthermore, no matter what the posture of the hand in the hand image is, for the case where the adjacent fingers are the index finger and the middle finger, the distance between key point 5 and key point 8 in the Y-axis direction of the hand image is determined to be the third distance.

[0169] For example, see Figure 4, the adjacent fingers are the middle finger and the ring finger, that is, the previous finger among the adjacent fingers is the middle finger, and the next finger among the adjacent fingers is the ring finger. The first key point is determined to be key point 9, and the third key point is determined to be key point 12, and then the distance between key point 9 and key point 12 in the Y-axis direction of the hand image is calculated as the third distance. In different hand images (images with the hand in different states), for the same key point, the labels of the key point in different images can be the same. Furthermore, no matter what the posture of the hand in the hand image is, for the case where the adjacent fingers are the middle finger and the ring finger, the distance between key point 9 and key point 12 in the Y-axis direction of the hand image is determined to be the third distance.

[0170] 605. If it is determined that the position state of the top of the previous finger is a preset state, determine the finger direction of the hand image according to the first distance and the second distance; wherein the preset state indicates that the top of the previous finger is away from the bottom joint of the previous finger.

[0171] Exemplarily, if it is determined that the Y-axis coordinate value of the top of the previous finger is much smaller than the Y-axis coordinate value of the bottom joint of the previous finger, then it is determined that the position state represents that the top of the previous finger is far away from the bottom joint of the previous finger.

[0172] If it is determined that the Y-axis coordinate value of the top of the previous finger is much greater than the Y-axis coordinate value of the bottom joint of the previous finger, then it is determined that the position state represents that the top of the previous finger is far away from the bottom joint of the previous finger.

[0173] If it is determined that the X-axis coordinate value of the top of the previous finger is much greater than the X-axis coordinate value of the bottom joint of the previous finger, then it is determined that the position state represents that the top of the previous finger is far away from the bottom joint of the previous finger.

[0174] If it is determined that the X-axis coordinate value of the top of the previous finger is much smaller than the X-axis coordinate value of the bottom joint of the previous finger, then it is determined that the position state represents that the top of the previous finger is far away from the bottom joint of the previous finger.

[0175] Then, the finger direction of the hand image is determined according to the magnitude relationship between the first distance and the second distance.

[0176] Step 605 includes the following situations, see the following introduction.

[0177] The first case of step 605: if it is determined that the position state of the top of the previous finger is the preset state, then it is determined whether the finger is pointing upward or downward according to the first distance, the second distance and the third distance.

[0178] In one example, the first case of step 605 includes the following steps:

[0179] In the first step of the first case of step 605, if it is determined that the key point set of the previous finger meets the first preset condition, the position state of the top of the previous finger is determined to be the preset state, and when it is determined that the second distance is less than the first distance and the third distance is greater than the first distance, it is determined that the finger is pointing upward.

[0180] Among them, the key point set includes the first key point, the third key point, the fourth key point and the fifth key point; the first preset condition is that the Y-axis coordinate value of the third key point is smaller than the Y-axis coordinate value of the fifth key point, the Y-axis coordinate value of the fifth key point is smaller than the Y-axis coordinate value of the fourth key point, and the Y-axis coordinate value of the fourth key point is smaller than the Y-axis coordinate value of the first key point.

[0181] The first key point, the fourth key point, the fifth key point and the third key point are the joint points along the bottom joint point to the top of the previous finger, respectively.

[0182] The second step of the first case of step 605, if it is determined that the key point set of the previous finger meets the second preset condition, the position state of the top of the previous finger is determined to be the preset state, and when it is determined that the second distance is less than the first distance and the third distance is greater than the first distance, it is determined that the finger is pointing downward.

[0183] Among them, the key point set includes the first key point, the third key point, the fourth key point and the fifth key point; the second preset condition is that the Y-axis coordinate value of the third key point is greater than the Y-axis coordinate value of the fifth key point, the Y-axis coordinate value of the fifth key point is greater than the Y-axis coordinate value of the fourth key point, and the Y-axis coordinate value of the fourth key point is greater than the Y-axis coordinate value of the first key point.

[0184] The first key point, the fourth key point, the fifth key point and the third key point are the joint points along the bottom joint point to the top of the previous finger, respectively.

[0185] In one example, the above “determining that the second distance is less than the first distance” includes: determining that the second distance is less than a first calculated value, wherein the first calculated value is the product of the first distance and a first preset value, and the first preset value is a value less than 1.

[0186] Exemplarily, the first distance, the second distance and the third distance have been calculated according to the above introduction.

[0187] According to the previous introduction, the key points of the previous finger of the adjacent finger have been obtained, and the key points of the previous finger constitute the key point set of the previous finger. Among them, the key point set includes the first key point, the third key point, the fourth key point and the fifth key point; among them, the first key point, the fourth key point, the fifth key point and the third key point are the joint points along the bottom joint point to the top of the previous finger.

[0188] Case 1. If it is determined that the Y-axis coordinate value of the third key point is smaller than the Y-axis coordinate value of the fifth key point, the Y-axis coordinate value of the fifth key point is smaller than the Y-axis coordinate value of the fourth key point, and the Y-axis coordinate value of the fourth key point is smaller than the Y-axis coordinate value of the first key point, then it is determined that the Y-axis coordinate value of the top joint of the previous finger is much smaller than the Y-axis coordinate value of the bottom joint of the previous finger. It can be preliminarily determined that the finger of the previous finger may be pointing upward. However, other conditions are needed to assist in the determination.

[0189] Furthermore, it is also necessary to determine the size relationship between the second distance and the first distance, and the size relationship between the third distance being greater than the first distance.

[0190] If it is determined that the second distance is smaller than the first distance (i.e., the distance between the bottom joint of the previous finger and the top of the previous finger in the X-axis direction is smaller than the distance between the second key point of the next finger and the bottom joint of the previous finger in the X-axis direction).

[0191] Then, if it is determined that the third distance is greater than the first distance (i.e., the distance between the bottom joint of the previous finger and the top of the previous finger in the Y-axis direction is greater than the distance between the second key point of the next finger and the bottom joint of the previous finger in the X-axis direction).

[0192] Combining the above three conditions, it can be determined that the previous finger is pointing upward.

[0193] In case 1, the relationship between the Y-axis coordinate values ​​of each key point of the previous finger is determined, and then it is determined that the previous finger may be pointing upward. At this time, it cannot be directly determined that it is pointing upward. Then it is further determined that the second distance is less than the first distance. At this time, it can be determined that the previous finger is not tilted too much; after further determining that the third distance is greater than the first distance, it can be determined that the previous finger is not bent too much. Based on the three conditions, it is determined that the previous finger is pointing upward; and then it is conveniently and accurately determined that the previous finger is pointing upward.

[0194] Case 2. The only difference from Case 1 is how to determine that "the third distance is greater than the first distance". When it can be determined that the second distance is less than the first calculated value (the first calculated value is the product of the first distance and "a value less than 1"), it is determined that "the third distance is greater than the first distance". Furthermore, in Case 2, the relationship between the Y-axis coordinate values ​​of each key point of the previous finger is judged, and then it is determined that the finger pointing of the previous finger may be upward. At this time, it cannot be directly determined that it is upward. Then it is further determined that the second distance is less than the first calculated value. At this time, it can be determined that the inclination of the previous finger is very small; after further determining that the third distance is greater than the first distance, it can be determined that the previous finger is not bent too much. Based on the three conditions, it is determined that the previous finger is pointing upward; then it is conveniently and accurately determined that the finger pointing of the previous finger is upward.

[0195] Figure 7 It is a finger pointing indication provided by the present disclosure Figure 1 ,like Figure 7 As shown, the adjacent fingers are the index finger and the middle finger, and a key point set of the index finger is obtained, which includes: a first key point 701, a fourth key point 702, a fifth key point 703, and a third key point 704. Among them, the first key point 701 is Figure 4 The key point 5, the fourth key point 702 is Figure 4 The key point 6, the fifth key point 703 is Figure 4 The key point 7, the third key point 704 is Figure 4 The key point 8 of the middle finger can also be obtained.

[0196] The distance between the first key point 701 and the second key point 705 in the X-axis direction of the hand image is calculated to obtain a first distance a.

[0197] The distance between the third key point 704 and the first key point 701 in the X-axis direction of the hand image is calculated to obtain a first distance b.

[0198] The distance between the third key point 704 and the first key point 701 in the Y-axis direction of the hand image is calculated to obtain a third distance c.

[0199] If it is determined that the Y-axis coordinate value of the third key point 704 is less than the Y-axis coordinate value of the fifth key point 703, the Y-axis coordinate value of the fifth key point 703 is less than the Y-axis coordinate value of the fourth key point 702, and the Y-axis coordinate value of the fourth key point 702 is less than the Y-axis coordinate value of the first key point 701; and it is determined that the first distance b is less than the first distance a (or the first distance b is less than the product of the first distance a and the first preset value; wherein the first preset value is, for example, 0.8); and it is determined that the third distance c is greater than the first distance a. Then it is determined that the index finger is pointing upward.

[0200] Figure 8 It is a finger pointing indication provided by the present disclosure Figure 2 ,like Figure 8 As shown, the adjacent fingers are the index finger and the middle finger, and a key point set of the index finger is obtained, which includes: a first key point 801, a fourth key point 802, a fifth key point 803, and a third key point 804. Among them, the first key point 801 is Figure 4 The key point 5, the fourth key point 802 is Figure 4 The key point 6, the fifth key point 803 is Figure 4 The key point 7, the third key point 804 is Figure 4 Key point 8.

[0201] The second key point 805 of the middle finger can also be obtained. Among them, the second key point 805 is Figure 4 Key point 10.

[0202] The distance between the first key point 801 and the second key point 805 in the X-axis direction of the hand image is calculated to obtain a first distance a.

[0203] The distance between the third key point 804 and the first key point 801 in the X-axis direction of the hand image is calculated to obtain a first distance b.

[0204] The distance between the third key point 804 and the first key point 801 in the Y-axis direction of the hand image is calculated to obtain a third distance c.

[0205] If it is determined that the Y-axis coordinate value of the third key point 804 is less than the Y-axis coordinate value of the fifth key point 803, the Y-axis coordinate value of the fifth key point 803 is less than the Y-axis coordinate value of the fourth key point 802, and the Y-axis coordinate value of the fourth key point 802 is less than the Y-axis coordinate value of the first key point 801; and it is determined that the first distance b is less than the first distance a (or the first distance b is less than the product of the first distance a and the first preset value; wherein the first preset value is, for example, 0.8); and it is determined that the third distance c is greater than the first distance a. Then it is determined that the index finger is pointing upward.

[0206] Case 3. If it is determined that the Y-axis coordinate value of the third key point is greater than the Y-axis coordinate value of the fifth key point, the Y-axis coordinate value of the fifth key point is greater than the Y-axis coordinate value of the fourth key point, and the Y-axis coordinate value of the fourth key point is greater than the Y-axis coordinate value of the first key point, then it is determined that the Y-axis coordinate value of the top joint of the previous finger is much greater than the Y-axis coordinate value of the bottom joint of the previous finger. It can be preliminarily determined that the finger of the previous finger may be pointing downward. However, other conditions are needed to assist in the determination.

[0207] Furthermore, it is also necessary to determine the size relationship between the second distance and the first distance, and the size relationship between the third distance being greater than the first distance.

[0208] If it is determined that the second distance is smaller than the first distance (i.e., the distance between the bottom joint of the previous finger and the top of the previous finger in the X-axis direction is smaller than the distance between the second key point of the next finger and the bottom joint of the previous finger in the X-axis direction).

[0209] Then, if it is determined that the third distance is greater than the first distance (i.e., the distance between the bottom joint of the previous finger and the top of the previous finger in the Y-axis direction is greater than the distance between the second key point of the next finger and the bottom joint of the previous finger in the X-axis direction).

[0210] Combining the above three conditions, it can be determined that the previous finger is pointing downward.

[0211] In case 3, the relationship between the Y-axis coordinate values ​​of each key point of the previous finger is determined, and then it is determined that the previous finger may be pointing downward. At this time, it cannot be directly determined that it is downward. Then it is further determined that the second distance is less than the first distance. At this time, it can be determined that the previous finger is not tilted too much; after further determining that the third distance is greater than the first distance, it can be determined that the previous finger is not bent too much. Based on the three conditions, it is determined that the previous finger is pointing downward; and then it is conveniently and accurately determined that the previous finger is pointing downward.

[0212] Case 4. The only difference from Case 3 is how to determine "determine that the second distance is less than the first distance". When it can be determined that the second distance is less than the first calculated value (the first calculated value is the product of the first distance and "a value less than 1"), it is determined that "determine that the second distance is less than the first distance". Furthermore, in Case 4, the relationship between the Y-axis coordinate values ​​of each key point of the previous finger is judged, and then it is determined that the finger pointing of the previous finger may be downward. At this time, it cannot be directly determined that it is downward. Then it is further determined that the second distance is less than the first calculated value. At this time, it can be determined that the inclination of the previous finger is very small; after further determining that the third distance is greater than the first distance, it can be determined that the previous finger is not bent too much. Based on the three conditions, it is determined that the previous finger is pointing downward; then it is conveniently and accurately determined that the finger pointing of the previous finger is downward.

[0213] Fig. 9 It is a finger pointing indication provided by the present disclosure Figure 3 ,like Fig. 9 As shown, the adjacent fingers are the index finger and the middle finger, and a key point set of the index finger is obtained, which includes: a first key point 901, a fourth key point 902, a fifth key point 903, and a third key point 904. Among them, the first key point 901 is Figure 4 The key point 5, the fourth key point 902 is Figure 4 The key point 6, the fifth key point 903 is Figure 4 The key point 7, the third key point 904 is Figure 4 Key point 8.

[0214] The second key point 905 of the middle finger can also be obtained. Among them, the second key point 905 is Figure 4 Key point 10.

[0215] The distance between the first key point 901 and the second key point 905 in the X-axis direction of the hand image is calculated to obtain a first distance a.

[0216] The distance between the third key point 904 and the first key point 901 in the X-axis direction of the hand image is calculated to obtain a first distance b.

[0217] The distance between the third key point 904 and the first key point 901 in the Y-axis direction of the hand image is calculated to obtain a third distance c.

[0218] If it is determined that the Y-axis coordinate value of the third key point 904 is greater than the Y-axis coordinate value of the fifth key point 903, the Y-axis coordinate value of the fifth key point 903 is greater than the Y-axis coordinate value of the fourth key point 902, and the Y-axis coordinate value of the fourth key point 902 is greater than the Y-axis coordinate value of the first key point 901; and it is determined that the first distance b is less than the first distance a (or the first distance b is less than the product of the first distance a and the first preset value; wherein the first preset value is, for example, 0.8); and it is determined that the third distance c is greater than the first distance a. Then it is determined that the index finger is pointing downward.

[0219] The second situation of step 605 includes the following steps:

[0220] The first step of the second case of step 605 is to determine that the position state of the top of the previous finger is the preset state if it is determined that the key point set of the previous finger meets the third preset condition, and when it is determined that the second distance is greater than the first distance, determine that the finger is pointing to the right.

[0221] Among them, the key point set includes the first key point, the third key point, the fourth key point and the fifth key point; the first key point, the fourth key point, the fifth key point and the third key point are the joint points along the bottom joint point to the top of the previous finger, respectively.

[0222] The third preset condition is that the X-axis coordinate value of the third key point is greater than the X-axis coordinate value of the fifth key point, the X-axis coordinate value of the fifth key point is greater than the X-axis coordinate value of the fourth key point, and the X-axis coordinate value of the fourth key point is greater than the X-axis coordinate value of the first key point.

[0223] In the second step of the second case of step 605, if it is determined that the key point set of the previous finger meets the fourth preset condition, the position state of the top of the previous finger is determined to be the preset state, and when it is determined that the second distance is greater than the first distance, the finger is determined to be pointing to the left;

[0224] The key point set includes a first key point, a third key point, a fourth key point and a fifth key point; the first key point, the fourth key point, the fifth key point and the third key point are, in order, joint points along the bottom joint point to the top joint point on the previous finger;

[0225] The fourth preset condition is that the X-axis coordinate value of the third key point is smaller than the X-axis coordinate value of the fifth key point, the X-axis coordinate value of the fifth key point is smaller than the X-axis coordinate value of the fourth key point, and the X-axis coordinate value of the fourth key point is smaller than the X-axis coordinate value of the first key point.

[0226] In one example, the above “determining that the second distance is greater than the first distance” includes: determining that the second distance is greater than a second calculated value, wherein the second calculated value is the product of the second distance and a second preset value, and the second preset value is a value greater than 1.

[0227] Exemplarily, the first distance, the second distance and the third distance have been calculated according to the above introduction.

[0228] According to the previous introduction, the key points of the previous finger of the adjacent finger have been obtained, and the key points of the previous finger constitute the key point set of the previous finger. Among them, the key point set includes the first key point, the third key point, the fourth key point and the fifth key point; among them, the first key point, the fourth key point, the fifth key point and the third key point are the joint points along the bottom joint point to the top of the previous finger.

[0229] Case 5. If it is determined that the X-axis coordinate value of the third key point is greater than the X-axis coordinate value of the fifth key point, the X-axis coordinate value of the fifth key point is greater than the X-axis coordinate value of the fourth key point, and the X-axis coordinate value of the fourth key point is greater than the X-axis coordinate value of the first key point, then it is determined that the X-axis coordinate value of the top of the previous finger is much greater than the X-axis coordinate value of the bottom joint of the previous finger. It can be preliminarily determined that the finger of the previous finger may be pointing to the right. However, other conditions are needed to assist in the determination.

[0230] Furthermore, it is also necessary to determine the magnitude relationship between the second distance and the first distance.

[0231] If it is determined that the second distance is greater than the first distance (i.e., the distance between the bottom joint of the previous finger and the top of the previous finger in the X-axis direction is greater than the distance between the second key point of the next finger and the bottom joint of the previous finger in the X-axis direction).

[0232] Combining the above two conditions, it can be determined that the previous finger is pointing to the right.

[0233] In case 5, the relationship between the X-axis coordinate values ​​of each key point of the previous finger is determined, and then it is determined that the previous finger may be pointing to the right. At this time, it cannot be directly determined that it is pointing to the right. Then it is further determined that the second distance is greater than the first distance. At this time, it can be determined that the previous finger is not bent too much. Based on the two conditions, it is determined that the previous finger is pointing to the right; and then it is conveniently and accurately determined that the previous finger is pointing to the right.

[0234] Fig.10 It is a finger pointing indication provided by the present disclosure Figure 4 ,like Fig.10 As shown, the adjacent fingers are the index finger and the middle finger, and a key point set of the index finger is obtained, which includes: a first key point 1001, a fourth key point 1002, a fifth key point 1003, and a third key point 1004. Among them, the first key point 1001 is Figure 4 The key point 5, the fourth key point 1002 is Figure 4 The key point 6, the fifth key point 1003 is Figure 4 The key point 7, the third key point 1004 is Figure 4 Key point 8.

[0235] The second key point 1005 of the middle finger can also be obtained. Among them, the second key point 1005 is Figure 4 Key point 10.

[0236] The distance between the first key point 1001 and the second key point 1005 in the X-axis direction of the hand image is calculated to obtain a first distance a.

[0237] The distance between the third key point 1004 and the first key point 1001 in the X-axis direction of the hand image is calculated to obtain a first distance b.

[0238] If it is determined that the X-axis coordinate value of the third key point 1004 is greater than the X-axis coordinate value of the fifth key point 1003, the X-axis coordinate value of the fifth key point 1003 is greater than the X-axis coordinate value of the fourth key point 1002, and the X-axis coordinate value of the fourth key point 1002 is greater than the X-axis coordinate value of the first key point 1001; and it is determined that the first distance b is greater than the first distance a (or the first distance b is greater than the product of the first distance a and the second preset value; wherein the second preset value is, for example, 1.3), it is determined that the index finger is pointing to the right.

[0239] Fig.11 It is a finger pointing indication provided by the present disclosure Figure 5 ,like Fig.11 As shown, the adjacent fingers are the index finger and the middle finger, and a key point set of the index finger is obtained, which includes: a first key point 1101, a fourth key point 1102, a fifth key point 1103, and a third key point 1104. Among them, the first key point 1101 is Figure 4 The key point 5, the fourth key point 1102 is Figure 4 The key point 6, the fifth key point 1103 is Figure 4 The key point 7, the third key point 1104 is Figure 4 Key point 8.

[0240] The second key point 1105 of the middle finger can also be obtained. Among them, the second key point 1105 is Figure 4 Key point 10.

[0241] The distance between the first key point 1101 and the second key point 1105 in the X-axis direction of the hand image is calculated to obtain a first distance a.

[0242] The distance between the third key point 1104 and the first key point 1101 in the X-axis direction of the hand image is calculated to obtain a first distance b.

[0243] If it is determined that the X-axis coordinate value of the third key point 1104 is greater than the X-axis coordinate value of the fifth key point 1103, the X-axis coordinate value of the fifth key point 1103 is greater than the X-axis coordinate value of the fourth key point 1102, and the X-axis coordinate value of the fourth key point 1102 is greater than the X-axis coordinate value of the first key point 1101; and it is determined that the first distance b is greater than the first distance a (or the first distance b is greater than the product of the first distance a and the second preset value; wherein the second preset value is, for example, 1.3), it is determined that the index finger is pointing to the right.

[0244] Case 6. The only difference from Case 5 is how to determine that "the second distance is greater than the first distance". When it can be determined that the second distance is greater than the second calculated value (the second calculated value is the product of the first distance and "a value less than 1"), it is determined that "the second distance is greater than the first distance". Furthermore, in Case 6, the relationship between the X-axis coordinate values ​​of each key point of the previous finger is judged, and then it is determined that the finger pointing of the previous finger may be to the right. At this time, it cannot be directly determined that it is to the right. Then it is further determined that the second distance is greater than the first calculated value. At this time, it can be determined that the curvature of the previous finger is very small. Based on the two conditions, it is determined that the previous finger is pointing to the right; and then it is conveniently and accurately determined that the finger pointing of the previous finger is to the right.

[0245] Case 7. If it is determined that the X-axis coordinate value of the third key point is smaller than the X-axis coordinate value of the fifth key point, the X-axis coordinate value of the fifth key point is smaller than the X-axis coordinate value of the fourth key point, and the X-axis coordinate value of the fourth key point is smaller than the X-axis coordinate value of the first key point, then it is determined that the X-axis coordinate value of the top of the previous finger is much smaller than the X-axis coordinate value of the bottom joint of the previous finger. It can be preliminarily determined that the finger of the previous finger may be pointing to the left. However, other conditions are needed to assist in the determination.

[0246] Furthermore, it is also necessary to determine the magnitude relationship between the second distance and the first distance.

[0247] If it is determined that the second distance is greater than the first distance (i.e., the distance between the bottom joint of the previous finger and the top of the previous finger in the X-axis direction is greater than the distance between the second key point of the next finger and the bottom joint of the previous finger in the X-axis direction).

[0248] Combining the above two conditions, it can be determined that the previous finger is pointing to the left.

[0249] In case 7, the relationship between the X-axis coordinate values ​​of each key point of the previous finger is determined, and then it is determined that the finger of the previous finger may be pointing to the left. At this time, it cannot be directly determined that it is pointing to the left. Then it is further determined that the second distance is greater than the first distance. At this time, it can be determined that the previous finger is not bent too much. Based on the two conditions, it is determined that the finger of the previous finger is pointing to the left; and then it is conveniently and accurately determined that the finger of the previous finger is pointing to the left.

[0250] Case 8. The only difference from Case 7 is how to determine that "the second distance is greater than the first distance". When it can be determined that the second distance is greater than the second calculated value (the second calculated value is the product of the first distance and "a value less than 1"), it is determined that "the second distance is greater than the first distance". Furthermore, in Case 8, the relationship between the X-axis coordinate values ​​of each key point of the previous finger is judged, and then it is determined that the finger pointing of the previous finger may be to the left. At this time, it cannot be directly determined that it is to the left. Then it is further determined that the second distance is greater than the first calculated value. At this time, it can be determined that the curvature of the previous finger is very small. Based on the two conditions, it is determined that the previous finger is pointing to the left; and then it is conveniently and accurately determined that the finger pointing of the previous finger is to the left.

[0251] Fig.12 It is a finger pointing indication provided by the present disclosure Figure 6 ,like Fig.12 As shown, the adjacent fingers are the index finger and the middle finger, and a key point set of the index finger is obtained, which includes: a first key point 1201, a fourth key point 1202, a fifth key point 1203, and a third key point 1204. Among them, the first key point 1201 is Figure 4 The key point 5, the fourth key point 1202 is Figure 4 The key point 6, the fifth key point 1203 is Figure 4 The key point 7, the third key point 1204 is Figure 4 Key point 8.

[0252] The second key point 1205 of the middle finger can also be obtained. Among them, the second key point 1205 is Figure 4 Key point 10.

[0253] The distance between the first key point 1201 and the second key point 1205 in the X-axis direction of the hand image is calculated to obtain a first distance a.

[0254] The distance between the third key point 1204 and the first key point 1201 in the X-axis direction of the hand image is calculated to obtain a first distance b.

[0255] If it is determined that the X-axis coordinate value of the third key point 1204 is less than the X-axis coordinate value of the fifth key point 1203, the X-axis coordinate value of the fifth key point 1203 is less than the X-axis coordinate value of the fourth key point 1202, and the X-axis coordinate value of the fourth key point 1202 is less than the X-axis coordinate value of the first key point 1201; and it is determined that the first distance b is greater than the first distance a (or the first distance b is greater than the product of the first distance a and the second preset value; wherein the second preset value is, for example, 1.3), it is determined that the index finger is pointing to the right.

[0256] Fig.13 It is a finger pointing indication provided by the present disclosure Figure 7 ,like Fig.13 As shown, the adjacent fingers are the index finger and the middle finger, and a key point set of the index finger is obtained, which includes: a first key point 1301, a fourth key point 1302, a fifth key point 1303, and a third key point 1304. Among them, the first key point 1301 is Figure 4 The key point 5, the fourth key point 1302 is Figure 4 The key point 6, the fifth key point 1303 is Figure 4 The key point 7, the third key point 1304 is Figure 4 Key point 8.

[0257] The second key point 1305 of the middle finger can also be obtained. Among them, the second key point 1305 is Figure 4 Key point 10.

[0258] The distance between the first key point 1301 and the second key point 1305 in the X-axis direction of the hand image is calculated to obtain a first distance a.

[0259] The distance between the third key point 1304 and the first key point 1301 in the X-axis direction of the hand image is calculated to obtain a first distance b.

[0260] If it is determined that the X-axis coordinate value of the third key point 1304 is less than the X-axis coordinate value of the fifth key point 1303, the X-axis coordinate value of the fifth key point 1303 is less than the X-axis coordinate value of the fourth key point 1302, and the X-axis coordinate value of the fourth key point 1302 is less than the X-axis coordinate value of the first key point 1301; and it is determined that the first distance b is greater than the first distance a (or the first distance b is greater than the product of the first distance a and the second preset value; wherein the second preset value is, for example, 1.3), it is determined that the index finger is pointing to the right.

[0261] 606. Generate a control instruction according to the finger pointing, and control the device in the vehicle to complete the action indicated by the control instruction according to the control instruction.

[0262] For example, in the intelligent cockpit of the vehicle, after obtaining the finger pointing, an instruction corresponding to the finger pointing can be generated according to the preset correspondence between the finger pointing and the control instruction. Then the control instruction is executed to control the device in the vehicle to complete the corresponding control action.

[0263] For example, if the vehicle recognizes that the finger is pointing upward, it determines and executes the control instruction corresponding to the finger pointing, for example, the control instruction is "open the sunroof"; if the vehicle recognizes that the finger is pointing downward, it determines and executes the control instruction corresponding to the finger pointing, for example, the control instruction is "close the sunroof"; if the vehicle recognizes that the finger is pointing to the right, it determines and executes the control instruction corresponding to the finger pointing, for example, the control instruction is "open the right window, or adjust the right rearview mirror, or turn on the right headlight"; if the vehicle recognizes that the finger is pointing to the left, it determines and executes the control instruction corresponding to the finger pointing, for example, the control instruction is "open the left window, or adjust the left rearview mirror, or turn on the left headlight".

[0264] In this embodiment, through the above process, referring to the aforementioned process, it is possible to conveniently and accurately determine whether the finger pointing of the previous finger is upward or downward based on three conditions; it is possible to conveniently and accurately determine whether the finger pointing of the previous finger is left or right based on two conditions.

[0265] In addition, in this embodiment, the first preset value and the second preset value can be adjusted according to different requirements; however, it is still necessary to ensure that the first preset value is a positive number greater than 1, and the second preset value is a positive number greater than 1. When judging that "the second distance is less than the first calculated value", the first calculated value is the product of the first distance and the first preset value, and then the determination of "the second distance is less than the first calculated value" is adjusted according to different requirements, that is, whether the inclination of the previous finger is very small is determined according to different requirements. Then, according to different requirements, it is determined whether the finger of the previous finger is pointing upward or downward.

[0266] When judging that "the second distance is greater than the second calculated value", the second calculated value is the product of the first distance and the second preset value, and how to judge that "the second distance is greater than the second calculated value" is adjusted according to different requirements, that is, judging whether the bending degree of the previous finger is very small according to different requirements. And judging whether the finger of the previous finger is pointing right or downward according to different requirements.

[0267] In some embodiments, the user intent recognition method based on multiple data modalities further includes: correcting the user intent in response to a correction instruction from the user.

[0268] Specifically, after obtaining the current user intent based on the process described in the above embodiment, it can be presented to the user so that the user can confirm whether the identified user intent is correct. The correction instruction is an instruction for the user to correct the presented user intent. The correction instruction indicates to adjust the obtained user intent.

[0269] Among them, after the user's intention is identified, the user's intention is corrected by responding to the user's correction instruction, thereby improving the accuracy of the user's intention, thereby improving the accuracy of vehicle control based on the user's intention.

[0270] In some embodiments, the process of determining the user intention of a user based on the data of each user and the weight of the user data also includes: performing feature extraction processing on the user data of the user's voice based on a preset recognition model to obtain voice features; and performing feature extraction processing on the user data of the facial image based on the preset recognition model to obtain lip movement features, and performing feature extraction processing on the user data of the hand image based on the preset recognition model to obtain gesture movement features in the hand image; performing recognition processing based on the preset recognition model according to the voice features, lip movement features, gesture movement features and the weight of the user data to obtain the user intention.

[0271] Among them, feature extraction processing is performed on user data of user speech based on a preset recognition model to obtain voice features; and feature extraction processing is performed on user data of facial images based on a preset recognition model to obtain lip movement features. For description of the process, please refer to the description in step S201 and will not be repeated here.

[0272] Specifically, feature extraction processing can be performed on the user data of the hand image based on the preset recognition model to obtain the gesture action features in the hand image. Among them, the present application does not limit the process of performing feature extraction processing on the user data of the hand image based on the preset recognition model to obtain the gesture action features in the hand image. Optionally, an algorithm for feature recognition of the hand image can be configured in the preset recognition model. Optionally, the algorithm for feature recognition of the hand image can be the same as the algorithm for feature recognition of the facial image described in step S201, which will not be repeated here. Optionally, it can also be other algorithms different from the algorithm for feature recognition of facial images, for example, feature extraction algorithms, statistical model algorithms, deep learning algorithms and other algorithms.

[0273] Specifically, after obtaining the voice features, lip movement features, and gesture movement features, recognition processing can be performed based on a preset recognition model according to the voice features, lip movement features, gesture movement features, and the weights of the user data to obtain the user's intention.

[0274] Among them, the present application does not limit the process of obtaining user intent by performing recognition processing based on a preset recognition model according to the weights of voice features, lip movement features, gesture movement features and user data. Optionally, recognition processing can be performed based on a preset recognition model according to the weights of voice features, lip movement features, gesture movement features and user data to obtain the initial user intent, and then the initial intent information can be adjusted according to the user intent obtained last time to obtain the current user intent.

[0275] Among them, in the process of determining the user's intention based on each user's data and the weight of the user's data, voice features, lip movement features and gesture movement features are extracted based on a preset recognition model, and the user's intention is obtained. This can further improve the accuracy of user intention recognition on the basis of improving the accuracy of feature extraction.

[0276] Fig.14 is a schematic diagram according to the fifth embodiment of the present disclosure. Fig.14 As shown, the model training method for user intent recognition provided by the fifth embodiment of the present disclosure includes:

[0277] Step S141, obtaining data to be trained.

[0278] Specifically, data to be trained may be obtained.

[0279] The training data includes user data, environment information and user behavior information in multiple data modes. The behavior information represents the user's behavior control method when operating the vehicle.

[0280] Among them, the description of user data, environmental information and user behavior habit information under various data modes can refer to the description in the embodiment shown above, which will not be repeated here.

[0281] Among them, this application does not limit the process of obtaining the data to be trained. Optionally, data obtained during the historical driving process of the vehicle can be obtained, and user data, environmental information and user behavior habit information under multiple data modalities can be extracted from the obtained data.

[0282] Step S142: train the initial model according to the data in the data to be trained to obtain a preset recognition model.

[0283] The initial model is a pre-built initial model used to train a preset recognition model.

[0284] For the description of the preset recognition model, please refer to the description in the above embodiment, which will not be repeated here. The preset recognition model is used to recognize user data under multiple data modes of the method shown in any of the above embodiments to obtain user intention.

[0285] Specifically, the present application does not limit the process of training the initial model according to the data in the data to be trained to obtain the preset recognition model. Optionally, in the process of training the initial model according to the data in the data to be trained, the weight of the user data corresponding to the data modality can be determined according to the environmental information and / or behavioral habit information in the data to be trained. Then, based on the initial model, the user data and the weight of the user data are processed to train the preset recognition model.

[0286] In an embodiment of the present disclosure, by acquiring data to be trained, an initial model is trained according to data in the data to be trained to obtain a preset recognition model, wherein the data to be trained includes user data under multiple data modalities, so that the preset recognition model obtained by training can identify user intent based on each user data under multiple data modalities, thereby improving the accuracy of user intent recognition.

[0287] In some embodiments, the process of training the initial model according to the data in the data to be trained to obtain the preset recognition model may include: in the process of training the initial model according to the data in the data to be trained, determining the weight of the user data corresponding to the data modality according to the environmental information and / or behavioral habit information in the data to be trained. Processing each user data and the weight of the user data based on the initial model to train the preset recognition model.

[0288] Among them, the process of determining the weight of user data corresponding to the data modality based on the environmental information and / or behavioral habit information in the data to be trained can refer to the description of the process of determining the weight of user data corresponding to the data modality based on the environmental information and / or behavioral habit information in the above embodiment, and will not be repeated here.

[0289] Among them, the present application does not limit the process of processing each user data and the weight of the user data based on the initial model to train the preset recognition model. Optionally, the feature data corresponding to each user data can be obtained first, and then the feature data corresponding to each user data can be fused according to the weight of the user data to train the preset recognition model. Among them, in the process of processing each user data and the weight of the user data based on the initial model to train the preset recognition model, the feature data corresponding to the user data is fused, so that the preset recognition model obtained by training can recognize the user's intention more accurately based not only on the feature data of the individual user data, but also on the fused feature data.

[0290] In the process of training to obtain a preset recognition model, the weight of the user data corresponding to the data modality is determined based on environmental information and / or behavioral habit information, and then the user intention is identified based on the user data and the weight of the user data. In this way, on the basis of identifying the user intention based on the user data under multiple data modalities, the weight of each user data is also taken into consideration, thereby further improving the accuracy of user intention recognition.

[0291] In some embodiments, the process of processing each user data and the weight of the user data based on the initial model to train a preset recognition model may include: performing feature extraction processing on the user data of the user's voice based on the initial model to obtain voice features; and performing feature extraction processing on the user data of the facial image based on the initial model to obtain lip movement features, and performing feature extraction processing on the user data of the hand image based on the initial model to obtain gesture movement features; performing recognition processing based on the initial model according to the voice features, lip movement features, gesture movement features and the weight of the user data to obtain the user intention; and training the initial model according to the obtained user intention to obtain the preset recognition model.

[0292] Among them, feature extraction is performed on user data of user speech based on the initial model to obtain speech features; and feature extraction is performed on user data of facial images based on the initial model to obtain lip movement features, and feature extraction is performed on user data of hand images based on the initial model to obtain gesture action features; recognition is performed based on the initial model according to the speech features, lip movement features, gesture action features and the weights of user data to obtain user intention; for the description of the process, reference can be made to the above embodiment in which feature extraction is performed on user data of user speech based on a preset recognition model to obtain speech features; and feature extraction is performed on user data of facial images based on a preset recognition model to obtain lip movement features, and feature extraction is performed on user data of hand images based on the preset recognition model to obtain gesture action features in the hand image; recognition is performed based on the speech features, lip movement features, gesture action features and the weights of user data to obtain user intention, which will not be repeated here.

[0293] After obtaining the user intent, the initial model can be trained according to the obtained user intent to obtain a preset recognition model. Optionally, the initial model can be trained according to the obtained user intent and the user intent corresponding to the data to be trained, so that the user intent recognized by the trained initial model is closer to the user intent corresponding to the data to be trained.

[0294] In the process of model training, speech features, lip movement features, and gesture movement features are extracted respectively, so that the preset recognition model obtained by training can more accurately recognize user intentions.

[0295] In some embodiments, the process of processing each user data and the weight of the user data based on the initial model to train a preset recognition model may include: performing feature extraction processing on the user data of the user's voice based on the initial model to obtain voice features; and performing feature extraction processing on the user data of the facial image based on the initial model to obtain lip movement features; processing the user data of the hand image to obtain the finger pointing in the hand image; performing recognition processing based on the initial model according to the voice features, lip movement features, finger pointing and the weight of the user data to obtain the user intention; and training the initial model according to the obtained user intention to obtain the preset recognition model.

[0296] Among them, the description of the process of performing feature extraction processing on the user data of the user's voice based on the initial model to obtain the voice features; and performing feature extraction processing on the user data of the facial image based on the initial model to obtain the lip movement features can refer to the description of the process of performing feature extraction processing on the user data of the user's voice based on the preset recognition model to obtain the voice features; and performing feature extraction processing on the user data of the facial image based on the preset recognition model to obtain the lip movement features in step S201, which will not be repeated here.

[0297] For a description of the process of processing the user data of the hand image to obtain the finger pointing in the hand image, reference may be made to the description in step S202, which will not be repeated here.

[0298] Among them, for the description of the process of obtaining the user's intention by performing recognition processing based on the initial model according to the weights of the voice features, lip movement features, finger pointing and user data, please refer to the description of the process of obtaining the user's intention by performing recognition processing based on the preset recognition model according to the voice features, lip movement features, finger pointing and user data weights in step S203, which will not be repeated here.

[0299] Specifically, after obtaining the user intent, the initial model can be trained according to the obtained user intent to obtain a preset recognition model. Optionally, the initial model can be trained according to the obtained user intent and the user intent corresponding to the data to be trained, so that the user intent recognized by the trained initial model is closer to the user intent corresponding to the data to be trained.

[0300] Among them, during the model training process, voice features and lip movement features are extracted respectively, so that the preset recognition model obtained by training can more accurately recognize user intentions. Among them, during the model training process, finger pointing is not recognized by the initial model, which improves the efficiency of model training. On the basis of reducing the labor cost and time cost of identifying finger pointing, the labor cost and time cost of user intention recognition are further reduced. Based on the above description, the embodiment of the present disclosure improves the accuracy of user intention recognition based on the preset recognition model obtained by training, and also reduces the labor cost and time cost of user intention recognition.

[0301] In some embodiments, user data in multiple data modalities include: user voice, facial image, and hand image.

[0302] For the description of the user voice, facial image, and hand image, please refer to the description in the above embodiment, which will not be repeated here.

[0303] Among them, if the user data under multiple data modalities in the training data include user voice, facial images, and hand images, the user data under various behavior control modes can be comprehensively obtained, thereby improving the accuracy of the preset recognition model trained based on the training set, and further improving the accuracy of the user intention identified based on the preset recognition model.

[0304] Fig.15 is a block diagram of a device for realizing user intent recognition based on multiple data modalities according to an embodiment of the present disclosure, such as Fig.15 As shown, the device 150 for user intention recognition based on multiple data modalities provided by the sixth embodiment of the present disclosure includes:

[0305] The acquisition unit 151 is used to acquire user data under multiple data modes of the user, and determine the environment information of the environment in which the user is located and / or the user's behavior habit information; wherein the behavior habit information represents the behavior control mode of the user when operating the vehicle;

[0306] A determination unit 152, configured to determine a weight of user data corresponding to a data modality according to environmental information and / or behavioral habit information;

[0307] The identification unit 153 is used to determine the user intention of the user according to each user data and the weight of the user data.

[0308] In some embodiments, user data in multiple data modalities include: user voice, facial image, and hand image.

[0309] In some embodiments, the determination unit 152 includes a first determination module 154, which is used to determine a noise value based on environmental information; wherein the noise value represents the noise level of the environment in which the user is located; if it is determined that the noise value is greater than a preset noise threshold, the weight of the facial image and / or the weight of the hand image is increased, and the weight of the user's voice is reduced.

[0310] In some embodiments, the determination unit 152 also includes a second determination module 155, which is used to determine the ambient brightness value based on the environmental information; wherein the ambient brightness value represents the brightness of the environment in which the user is located; if it is determined that the ambient brightness value is less than a preset brightness threshold, the weight of the user's voice is increased, and the weight of the facial image and / or the weight of the hand image is reduced.

[0311] In some embodiments, the determination unit 152 also includes a third determination module 156, which is used to determine the number of historical operations of each user in different behavior control methods based on the behavior habit information; wherein the different behavior control methods include the following behavior control methods: voice control method, facial expression control method, and gesture control method; increase the weight of the user data under the behavior control method with the largest number of historical operations, and reduce the weight of other user data.

[0312] In some embodiments, the determination unit 152 also includes a fourth determination module 157, which is used to perform feature fusion processing on the environmental information and behavioral habit information based on a preset weight processing model to obtain a fused feature; and process the fused feature based on the preset weight processing model to determine the weight of the user data corresponding to the data modality.

[0313] In some embodiments, the determination unit 152 also includes a fifth determination module 158, which is used to determine the weight of the user data corresponding to the data modality based on a preset mapping relationship, according to environmental information and behavioral habit information; wherein the preset mapping relationship represents the correspondence between the environmental information, behavioral habit information, and the weight of the user data.

[0314] In some embodiments, the recognition unit includes a first processing module 159, which is used to perform feature extraction processing on user data of user voice based on a preset recognition model to obtain voice features; and perform feature extraction processing on user data of facial images based on a preset recognition model to obtain lip movement features, and perform feature extraction processing on user data of hand images based on a preset recognition model to obtain gesture action features in the hand images; the first recognition module 1510 is used to perform recognition processing based on a preset recognition model according to voice features, lip movement features, gesture action features and weights of user data to obtain user intentions.

[0315] In some embodiments, the recognition unit 153 includes a second processing module 1511, which is used to perform feature extraction processing on the user data of the user voice based on a preset recognition model to obtain voice features; and perform feature extraction processing on the user data of the facial image based on a preset recognition model to obtain lip movement features; a third processing module 1512 is used to process the user data of the hand image to obtain the finger pointing in the hand image; and a second recognition module 1513 is used to perform recognition processing based on a preset recognition model according to the voice features, lip movement features, finger pointing and weights of the user data to obtain the user intention.

[0316] In some embodiments, the third processing module 1512 is specifically used to determine the key points of fingers among adjacent fingers in a hand image; determine finger information based on the key points of fingers among adjacent fingers; wherein the finger information includes a first distance, a second distance and a third distance, the first distance representing the distance between adjacent fingers in the X-axis direction of the hand image; the second distance representing the length of the previous finger among adjacent fingers in the X-axis direction of the hand image; the third distance representing the length of the previous finger among adjacent fingers in the Y-axis direction of the hand image; determine the finger pointing of the hand image based on the position state of the top of the previous finger, the first distance, the second distance and the third distance; wherein the position state represents the distance relationship between the top of the previous finger and the bottom joint of the previous finger.

[0317] In some embodiments, the third processing module 1512 is further specifically used to determine the distance between the first key point of the previous finger and the second key point of the next finger in the adjacent fingers in the X-axis direction, which is the first distance; wherein the first key point is the bottom joint of the previous finger, and the second key point is the key point on the next finger adjacent to the bottom joint of the next finger.

[0318] In some embodiments, the third processing module 1512 is further specifically used to determine the distance between the third key point of the previous finger and the first key point of the previous finger among adjacent fingers in the X-axis direction, which is the second distance; wherein the first key point is the bottom joint of the previous finger, and the third key point is the top of the previous finger.

[0319] In some embodiments, the third processing module 1512 is further specifically used to determine the distance between the third key point of the previous finger and the first key point of the previous finger in the adjacent fingers in the Y-axis direction, which is the third distance; wherein the first key point is the bottom joint of the previous finger, and the third key point is the top of the previous finger.

[0320] In some embodiments, the second recognition module 1513 is specifically used to recognize and process voice features, lip movement features, and finger pointing based on a preset recognition model and the weight of user data to obtain initial intention information; according to the user intention obtained last time, the initial intention information is adjusted to obtain the current user intention.

[0321] Fig.16 It is a block diagram of another device for implementing user intent recognition based on multiple data modalities according to an embodiment of the present disclosure.

[0322] Fig.17 is a block diagram of a device for implementing model training for user intent recognition according to an embodiment of the present disclosure, such as Fig.17As shown, the device 170 for model training for user intent recognition provided by the seventh embodiment of the present disclosure includes:

[0323] The acquisition unit 171 is used to acquire the data to be trained; wherein the data to be trained includes user data, environmental information and user behavior habit information under multiple data modes; wherein the behavior habit information represents the behavior control mode of the user when operating the vehicle;

[0324] A training unit 172, used to train the initial model according to the data in the training data to obtain a preset recognition model;

[0325] Among them, the preset recognition model is used to identify user data under multiple data modes of any method in claims 1-12 to obtain user intention.

[0326] In some embodiments, the training unit 172 includes a first training module 173, which is used to determine the weight of the user data corresponding to the data modality according to the environmental information and / or behavioral habit information in the data to be trained during the process of training the initial model according to the data in the data to be trained; the second training module 173 is used to process each user data and the weight of the user data based on the initial model to train and obtain a preset recognition model.

[0327] In some embodiments, the second training module 173 is specifically used to perform feature extraction processing on the user data of the user voice based on the initial model to obtain voice features; and perform feature extraction processing on the user data of the facial image based on the initial model to obtain lip movement features, and perform feature extraction processing on the user data of the hand image based on the initial model to obtain gesture action features; perform recognition processing based on the initial model according to the voice features, lip movement features, gesture action features and the weight of the user data to obtain user intent; and train the initial model according to the obtained user intent to obtain a preset recognition model.

[0328] In some embodiments, the second training module 173 is specifically used to perform feature extraction processing on the user data of the user voice based on the initial model to obtain voice features; and perform feature extraction processing on the user data of the facial image based on the initial model to obtain lip movement features; process the user data of the hand image to obtain the finger pointing in the hand image; perform recognition processing based on the initial model according to the voice features, lip movement features, finger pointing and weights of the user data to obtain the user intention; and train the initial model according to the obtained user intention to obtain a preset recognition model.

[0329] In some embodiments, user data in multiple data modalities include: user voice, facial image, and hand image.

[0330] Fig.18 It is a block diagram of another device for model training applied to user intent recognition, which is used to implement the embodiment of the present disclosure.

[0331] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0332] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0333] According to an embodiment of the present disclosure, the present disclosure also provides a vehicle, comprising the device for user intention recognition based on multiple data modalities provided by any of the above embodiments.

[0334] Fig.19 It is a block diagram of an electronic device for implementing a method for user intent recognition based on multiple data modalities / a method for user intent recognition based on multiple data modalities of an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0335] like Fig.19 As shown, the device 190 includes a computing unit 191, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 192 or a computer program loaded from a storage unit 198 into a random access memory (RAM) 193. In the RAM 193, various programs and data required for the operation of the device 190 can also be stored. The computing unit 191, the ROM 192, and the RAM 193 are connected to each other via a bus 194. An input / output (I / O) interface 195 is also connected to the bus 194.

[0336] A number of components in the device 190 are connected to the I / O interface 195, including: an input unit 196, such as a keyboard, a mouse, etc.; an output unit 197, such as various types of displays, speakers, etc.; a storage unit 198, such as a disk, an optical disk, etc.; and a communication unit 199, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 199 allows the device 190 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0337] The computing unit 191 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 191 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 191 performs the various methods and processes described above, such as a user intent recognition method based on multiple data modalities or a model training method applied to user intent recognition. For example, in some embodiments, a user intent recognition method based on multiple data modalities or a model training method applied to user intent recognition may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 198. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 190 via the ROM 192 and / or the communication unit 199. When the computer program is loaded into the RAM 193 and executed by the computing unit 191, one or more steps of the user intent recognition method based on multiple data modalities or the model training method applied to user intent recognition described above may be executed. Alternatively, in other embodiments, the computing unit 191 may be configured in any other appropriate manner (e.g., by means of firmware) to execute a user intent recognition method based on multiple data modalities or a model training method applied to user intent recognition.

[0338] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0339] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0340] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0341] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0342] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0343] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0344] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0345] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A user intention recognition method based on multiple data modalities, comprising: Acquire user data under multiple data modes of the user, and determine environmental information of the environment in which the user is located and / or behavioral habit information of the user; wherein the behavioral habit information represents the behavioral control mode of the user when operating the vehicle; Determining a weight of user data corresponding to a data modality according to the environment information and / or the behavior habit information; The user intention of the user is determined according to each of the user data and the weight of the user data.

2. The method according to claim 1, wherein: The user data in the multiple data modes include: user voice, facial image, and hand image.

3. The method according to claim 2, wherein: Determining, according to the environment information, a weight of user data corresponding to the data modality, including: Determine a noise value according to the environmental information; wherein the noise value represents the noise level of the environment in which the user is located; If it is determined that the noise value is greater than a preset noise threshold, the weight of the facial image and / or the weight of the hand image is increased, and the weight of the user's voice is reduced.

4. The method according to claim 2, wherein: Determining, according to the environment information, a weight of user data corresponding to the data modality, including: Determine an environment brightness value according to the environment information; wherein the environment brightness value represents the brightness of the environment in which the user is located; If it is determined that the ambient brightness value is less than a preset brightness threshold, the weight of the user's voice is increased, and the weight of the facial image and / or the weight of the hand image is reduced.

5. The method according to claim 2, wherein: Determining the weight of the user data corresponding to the data modality according to the behavior habit information includes: According to the behavior habit information, determine the number of historical operations of each user in different behavior control modes; wherein the different behavior control modes include the following behavior control modes: voice control mode, facial expression control mode, and gesture control mode; Increase the weight of the user data under the behavior control mode with the largest number of historical operations, and reduce the weight of other user data.

6. The method according to claim 2, wherein: Determining the weight of the user data corresponding to the data modality according to the environment information and the behavior habit information includes: Performing feature fusion processing on the environmental information and the behavioral habit information based on a preset weight processing model to obtain fusion features; The fusion feature is processed based on the preset weight processing model to determine the weight of the user data corresponding to the data modality.

7. The method according to claim 2, wherein: Determining the weight of the user data corresponding to the data modality according to the environment information and the behavior habit information includes: Based on a preset mapping relationship, the weight of the user data corresponding to the data modality is determined according to the environmental information and the behavioral habit information; wherein the preset mapping relationship represents the correspondence between the environmental information, the behavioral habit information, and the weight of the user data.

8. The method according to any one of claims 1 to 7, wherein: Determining the user intention of the user according to each of the user data and the weight of the user data includes: Performing feature extraction processing on the user data of the user's voice based on a preset recognition model to obtain voice features; performing feature extraction processing on the user data of the facial image based on the preset recognition model to obtain lip movement features, and performing feature extraction processing on the user data of the hand image based on the preset recognition model to obtain gesture movement features in the hand image; According to the voice features, the lip movement features, the gesture movement features and the weight of the user data, recognition processing is performed based on the preset recognition model to obtain the user intention.

9. The method according to any one of claims 1 to 7, wherein: Determining the user intention of the user according to each of the user data and the weight of the user data includes: Based on a preset recognition model, feature extraction processing is performed on the user data of the user's voice to obtain voice features; and based on the preset recognition model, feature extraction processing is performed on the user data of the facial image to obtain lip movement features; Processing the user data of the hand image to obtain the finger pointing in the hand image; According to the voice features, the lip movement features, the finger pointing and the weight of the user data, recognition processing is performed based on the preset recognition model to obtain the user intention.

10. The method according to claim 9, wherein: The user data of the hand image is processed to obtain the finger pointing in the hand image, including: Determining key points of fingers among adjacent fingers in the hand image; Determine finger information according to key points of fingers among the adjacent fingers; wherein the finger information includes a first distance, a second distance and a third distance, wherein the first distance represents the distance between adjacent fingers in the X-axis direction of the hand image; the second distance represents the length of a previous finger among the adjacent fingers in the X-axis direction of the hand image; and the third distance represents the length of the previous finger among the adjacent fingers in the Y-axis direction of the hand image; The finger pointing of the hand image is determined based on a position state of the top of the previous finger, the first distance, the second distance and multiple information in the third distance; wherein the position state represents the distance relationship between the top of the previous finger and the bottom joint of the previous finger.

11. The method according to claim 10, wherein: Determining the first distance according to the key points of the fingers among the adjacent fingers includes: Determine the distance between a first key point of a front finger and a second key point of a rear finger in the adjacent fingers in the X-axis direction as the first distance; wherein the first key point is a bottom joint of the front finger, and the second key point is a key point on the rear finger adjacent to the bottom joint of the rear finger.

12. The method according to claim 10 or 11, wherein: Determining the second distance according to the key points of the fingers among the adjacent fingers includes: Determine the distance between the third key point of the previous finger and the first key point of the previous finger in the adjacent fingers in the X-axis direction as the second distance; wherein the first key point is the bottom joint of the previous finger, and the third key point is the top of the previous finger.

13. The method according to claim 10 or 11, wherein: Determining the third distance according to the key points of the fingers among the adjacent fingers includes: Determine the distance between the third key point of the previous finger and the first key point of the previous finger in the adjacent fingers in the Y-axis direction as the third distance; wherein the first key point is the bottom joint of the previous finger, and the third key point is the top of the previous finger.

14. The method according to any one of claims 9 to 13, wherein: According to the voice feature, the lip movement feature, the finger pointing and the weight of the user data, recognition processing is performed based on the preset recognition model to obtain the user intention, including: Based on the preset recognition model and the weight of the user data, the voice features, the lip movement features and the finger pointing are recognized and processed to obtain initial intention information; According to the user intent obtained last time, the initial intent information is adjusted to obtain the current user intent.

15. A model training method for user intent recognition, comprising: Acquire data to be trained; wherein the data to be trained includes user data, environmental information, and user behavior habit information under multiple data modes; wherein the behavior habit information represents the behavior control mode of the user when operating the vehicle; Training the initial model according to the data in the data to be trained to obtain a preset recognition model; Among them, the preset recognition model is used to identify user data under multiple data modalities described in any method of claims 1-12 to obtain user intention.

16. The method according to claim 15, wherein: The initial model is trained according to the data in the data to be trained to obtain a preset recognition model, including: In the process of training the initial model according to the data in the data to be trained, determining the weight of the user data corresponding to the data modality according to the environmental information and / or the behavioral habit information in the data to be trained; Based on the initial model, each user data and the weight of the user data are processed to train and obtain the preset recognition model.

17. The method according to claim 16, wherein: Processing each of the user data and the weight of the user data based on the initial model to train and obtain the preset recognition model includes: Based on the initial model, feature extraction processing is performed on the user data of the user's voice to obtain voice features; and based on the initial model, feature extraction processing is performed on the user data of the facial image to obtain lip movement features, and based on the initial model, feature extraction processing is performed on the user data of the hand image to obtain gesture movement features; According to the voice features, the lip movement features, the gesture movement features and the weight of the user data, recognition processing is performed based on the initial model to obtain the user intention; and the initial model is trained according to the obtained user intention to obtain the preset recognition model.

18. The method according to claim 16, wherein: Processing each of the user data and the weight of the user data based on the initial model to train and obtain the preset recognition model includes: Based on the initial model, feature extraction processing is performed on the user data of the user's voice to obtain voice features; and based on the initial model, feature extraction processing is performed on the user data of the facial image to obtain lip movement features; Processing the user data of the hand image to obtain the finger pointing in the hand image; According to the voice features, the lip movement features, the finger pointing and the weight of the user data, recognition processing is performed based on the initial model to obtain the user intention; and the initial model is trained according to the obtained user intention to obtain the preset recognition model.

19. The method according to any one of claims 15 to 18, wherein: The user data in the multiple data modes include: user voice, facial image, and hand image.

20. A device for user intention recognition based on multiple data modalities, comprising: An acquisition unit, used to acquire user data under multiple data modes of the user, and determine the environment information of the environment in which the user is located and / or the behavior habit information of the user; wherein the behavior habit information represents the behavior control mode of the user when operating the vehicle; a determination unit, configured to determine a weight of user data corresponding to a data modality according to the environment information and / or the behavior habit information; The identification unit is used to determine the user intention of the user according to each of the user data and the weight of the user data.

21. The device according to claim 20, wherein: The user data in the multiple data modes include: user voice, facial image, and hand image.

22. The device according to claim 21, wherein The determination unit includes a first determination module, which is used to determine a noise value based on the environmental information; wherein the noise value represents the noise level of the environment in which the user is located; if it is determined that the noise value is greater than a preset noise threshold, the weight of the facial image and / or the weight of the hand image is increased, and the weight of the user's voice is reduced.

23. The device according to claim 21, wherein The determination unit also includes a second determination module, which is used to determine the ambient brightness value based on the environmental information; wherein the ambient brightness value represents the brightness of the environment in which the user is located; if it is determined that the ambient brightness value is less than a preset brightness threshold, the weight of the user's voice is increased, and the weight of the facial image and / or the weight of the hand image is reduced.

24. The device according to claim 21, wherein The determination unit also includes a third determination module, which is used to determine the number of historical operations of each user in different behavior control modes based on the behavior habit information; wherein the different behavior control modes include the following behavior control modes: voice control mode, facial expression control mode, and gesture control mode; increase the weight of the user data under the behavior control mode with the largest number of historical operations, and reduce the weight of other user data.

25. The device according to claim 21, wherein The determination unit further includes a fourth determination module, which is used to perform feature fusion processing on the environmental information and the behavioral habit information based on a preset weight processing model to obtain a fusion feature; The fusion feature is processed based on the preset weight processing model to determine the weight of the user data corresponding to the data modality.

26. The device according to claim 21, wherein The determination unit also includes a fifth determination module, which is used to determine the weight of the user data corresponding to the data modality based on a preset mapping relationship, according to the environmental information and the behavioral habit information; wherein the preset mapping relationship represents the correspondence between the environmental information, behavioral habit information, and the weight of the user data.

27. The device according to any one of claims 20 to 26, wherein: The recognition unit includes a first processing module, which is used to perform feature extraction processing on the user data of the user voice based on a preset recognition model to obtain voice features; and perform feature extraction processing on the user data of the facial image based on the preset recognition model to obtain lip movement features, and perform feature extraction processing on the user data of the hand image based on the preset recognition model to obtain gesture action features in the hand image; the first recognition module is used to perform recognition processing based on the preset recognition model according to the voice features, the lip movement features, the gesture action features and the weight of the user data to obtain the user intention.

28. The device according to any one of claims 20 to 26, wherein: The recognition unit includes a second processing module for performing feature extraction processing on the user data of the user's voice based on a preset recognition model to obtain voice features; and performing feature extraction processing on the user data of the facial image based on the preset recognition model to obtain lip movement features; a third processing module for processing the user data of the hand image to obtain the finger pointing in the hand image; The second recognition module is used to perform recognition processing based on the preset recognition model according to the voice features, the lip movement features, the finger pointing and the weight of the user data to obtain the user intention.

29. The device according to claim 28, wherein The third processing module is specifically used to determine the key points of fingers among adjacent fingers in the hand image; determine finger information based on the key points of fingers among the adjacent fingers; wherein the finger information includes a first distance, a second distance and a third distance, wherein the first distance represents the distance between adjacent fingers in the X-axis direction of the hand image; the second distance represents the length of the previous finger among the adjacent fingers in the X-axis direction of the hand image; the third distance represents the length of the previous finger among the adjacent fingers in the Y-axis direction of the hand image; determine the finger pointing of the hand image based on the position state of the top of the previous finger, the first distance, the second distance and multiple information in the third distance; wherein the position state represents the distance relationship between the top of the previous finger and the bottom joint of the previous finger.

30. The device according to claim 29, wherein: The third processing module is further specifically used to determine the distance between the first key point of the previous finger and the second key point of the next finger in the adjacent fingers in the X-axis direction, which is the first distance; wherein the first key point is the bottom joint of the previous finger, and the second key point is the key point on the next finger adjacent to the bottom joint of the next finger.

31. The device according to claim 29 or 30, wherein: The third processing module is further specifically used to determine the distance between the third key point of the previous finger and the first key point of the previous finger among the adjacent fingers in the X-axis direction, which is the second distance; wherein the first key point is the bottom joint of the previous finger, and the third key point is the top of the previous finger.

32. The device according to claim 29 or 30, wherein: The third processing module is further specifically used to determine the distance between the third key point of the previous finger and the first key point of the previous finger in the adjacent fingers in the Y-axis direction, which is the third distance; wherein the first key point is the bottom joint of the previous finger, and the third key point is the top of the previous finger.

33. The device according to any one of claims 28 to 32, wherein: The second recognition module is specifically used to recognize and process the voice features, the lip movement features and the finger pointing based on the preset recognition model and the weight of the user data to obtain initial intention information; according to the user intention obtained last time, the initial intention information is adjusted to obtain the current user intention.

34. A device for model training for user intent recognition, comprising: An acquisition unit, used to acquire data to be trained; wherein the data to be trained includes user data, environmental information and user behavior habit information under multiple data modes; wherein the behavior habit information represents the behavior control mode of the user when operating the vehicle; A training unit, used to train the initial model according to the data in the data to be trained to obtain a preset recognition model; Among them, the preset recognition model is used to identify user data under multiple data modalities described in any method of claims 1-14 to obtain user intention.

35. The device according to claim 34, wherein The training unit includes a first training module, which is used to determine the weight of the user data corresponding to the data modality according to the environmental information and / or the behavioral habit information in the data to be trained during the process of training the initial model according to the data in the data to be trained; The second training module is used to process each of the user data and the weight of the user data based on the initial model to train and obtain the preset recognition model.

36. The device according to claim 35, wherein The second training module is specifically used to perform feature extraction processing on the user data of the user's voice based on the initial model to obtain voice features; perform feature extraction processing on the user data of the facial image based on the initial model to obtain lip movement features, and perform feature extraction processing on the user data of the hand image based on the initial model to obtain gesture action features; perform recognition processing based on the initial model according to the voice features, the lip movement features, the gesture action features and the weight of the user data to obtain user intention; The initial model is trained according to the obtained user intention to obtain the preset recognition model.

37. The device according to claim 35, wherein: The second training module is further specifically used to perform feature extraction processing on the user data of the user voice based on the initial model to obtain voice features; and perform feature extraction processing on the user data of the facial image based on the initial model to obtain lip movement features; process the user data of the hand image to obtain the finger pointing in the hand image; and perform recognition processing based on the initial model according to the voice features, the lip movement features, the finger pointing and the weight of the user data to obtain the user intention; The initial model is trained according to the obtained user intention to obtain the preset recognition model.

38. The device according to any one of claims 34 to 37, wherein: The user data in the multiple data modes include: user voice, facial image, and hand image.

39. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-14 or claims 15-19.

40. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-14 or claims 15-19.

41. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 14 or claims 15 to 19.

Images