Vehicle intelligent control method and device, vehicle and storage medium

Abstract

The invention discloses a vehicle intelligent control method and device, a vehicle and a storage medium, and relates to the technical field of vehicle control, and the method comprises the steps: determining voice information in the vehicle through a voice collection module, and determining a sound source position and a voice instruction according to the voice information; when the voice instruction is a target instruction, acquiring in-vehicle image data acquired by an image acquisition module based on the sound source position; performing target gesture recognition by using a preset recognition function based on the in-vehicle image data to obtain a recognition result; when the recognition result is that the target gesture is recognized, the vehicle is controlled according to the target gesture, the voice instruction and the sound source position, according to the scheme, additional depth information does not need to be newly added, gesture recognition of the user in the vehicle is carried out through a multiplex image acquisition module, the hardware cost of recognition in the vehicle is reduced, and the recognition effect is improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle control technology, and in particular to a vehicle intelligent control method, device, vehicle and storage medium. Background Art

[0002] At present, the in-car cockpit interaction is mainly based on voice control, button control and touch screen control. This design is different from real social interaction, and this difference leads to the user feeling that the interaction lacks agility during human-computer interaction. Therefore, in order to improve the user's interactive experience, TOF (Time of fly) cameras are currently used for 3D gesture recognition during vehicle control, and the recognized information is combined with voice to intelligently control the vehicle. However, this control method requires the addition of a TOF camera, which increases the hardware cost. Summary of the invention

[0003] The main purpose of this application is to provide a vehicle intelligent control method, device, vehicle and storage medium, aiming to solve the technical problem that the existing vehicle control needs to be combined with a TOF camera, resulting in high costs.

[0004] To achieve the above object, the present application proposes a vehicle intelligent control method, which is applied to a vehicle, wherein the vehicle is provided with a voice acquisition module and an image acquisition module, and the method comprises:

[0005] Determine the voice information in the vehicle through the voice collection module, and determine the sound source location and voice command according to the voice information;

[0006] When the voice command is a target command, acquiring in-vehicle image data collected by the image collection module based on the sound source position;

[0007] Based on the in-vehicle image data, a preset recognition function is used to perform target gesture recognition to obtain a recognition result;

[0008] When the recognition result is that the target gesture is recognized, the vehicle is controlled according to the target gesture, the voice command and the sound source position.

[0009] In one embodiment, when the recognition result is that the target gesture is recognized, the step of controlling the vehicle according to the target gesture, the voice command and the sound source position includes:

[0010] When the recognition result is that the target gesture is recognized, determining the gesture direction according to the target gesture;

[0011] Determine a target component according to the gesture direction and the sound source position;

[0012] The target component is controlled according to the voice command.

[0013] In one embodiment, the step of controlling the target component according to the voice command includes:

[0014] Determining a key instruction word according to the voice instruction;

[0015] Get vehicle component status;

[0016] Determine a target action for the target component according to the key instruction word and the state of the vehicle component;

[0017] The vehicle is controlled according to the target action.

[0018] In one embodiment, before the step of performing target gesture recognition based on the in-vehicle image data using a preset recognition function to obtain a recognition result, the step further includes:

[0019] Acquire historical human hand data, perform image learning through the historical human hand data, construct a two-dimensional human hand model, and convert the two-dimensional human hand model into a first function;

[0020] Obtaining historical hand gesture data according to the first function and the historical hand data, performing finger pointing gesture learning according to the historical hand gesture data, constructing a two-dimensional gesture recognition model, and converting the two-dimensional gesture recognition model into a second function;

[0021] Obtaining historical finger pointing gesture data according to the second function and the historical gesture posture data, performing pointing direction learning according to the historical finger pointing gesture data, constructing a two-dimensional pointing model, and converting the two-dimensional pointing model into a third function;

[0022] A preset recognition function is obtained through the third function.

[0023] In one embodiment, when the voice command is a target command, before the step of acquiring the in-vehicle image data collected by the image collection module based on the sound source position, the step further includes:

[0024] parsing the voice command to determine whether the voice command is a vehicle control command;

[0025] When the voice command is the vehicle control command, determining whether there is vehicle component information in the vehicle control command;

[0026] When the vehicle component information does not exist in the vehicle control instruction, determining whether the vehicle control instruction contains a preset pronoun;

[0027] When the preset pronoun exists in the vehicle control instruction, the voice instruction is determined to be a target instruction.

[0028] In one embodiment, after the step of performing target gesture recognition using a preset recognition function according to the in-vehicle image data to obtain a recognition result, the step further includes:

[0029] When the recognition result is that the target gesture is not recognized, generating a voice prompt message;

[0030] The voice prompt information is prompted by the voice collection module, and the voice collection module responds to the voice information in the car, and the steps of determining the sound source position and the voice command according to the voice information are returned.

[0031] In one embodiment, the voice collection module includes a microphone array, and the microphone array includes a plurality of microphones;

[0032] The step of determining the sound source position according to the voice information comprises:

[0033] Obtaining the time difference between the microphones in the microphone array according to the sound source localization strategy and the voice information;

[0034] Calculate the angle and distance of the sound source from the microphone array according to the time difference;

[0035] The location of the sound source is determined by the angle and the distance.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a vehicle intelligent control device, the vehicle intelligent control device comprising:

[0037] A determination module, used to determine the voice information in the vehicle through the voice collection module, and determine the sound source position and voice command according to the voice information;

[0038] A collection module, used for collecting in-vehicle image data through an image collection module based on the sound source position when the voice command is a target command;

[0039] A recognition module, used to perform target gesture recognition based on the in-vehicle image data using a preset recognition function to obtain a recognition result;

[0040] A control module is used to control the vehicle according to the target gesture, the voice command and the sound source position when the recognition result is that the target gesture is recognized.

[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes a vehicle, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the vehicle intelligent control method as described above.

[0042] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the vehicle intelligent control method described above are implemented.

[0043] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the vehicle intelligent control method described above.

[0044] One or more technical solutions proposed in the present application, a vehicle intelligent control method is applied to a vehicle, and the vehicle is provided with a voice acquisition module and an image acquisition module, and the method comprises: determining the voice information in the vehicle by the voice acquisition module, and determining the sound source position and the voice command according to the voice information; when the voice command is a target command, acquiring the in-vehicle image data acquired by the image acquisition module based on the sound source position; performing target gesture recognition based on the in-vehicle image data using a preset recognition function to obtain a recognition result; when the recognition result is that the target gesture is recognized, controlling the vehicle according to the target gesture, the voice command and the sound source position. This solution does not require the addition of additional depth information, and the gesture recognition of the user in the vehicle is performed by reusing the image acquisition module, thereby reducing the hardware cost of in-vehicle recognition and improving the recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the vehicle intelligent control method of the present application;

[0048] Figure 2 This is a schematic diagram of the distribution of in-vehicle components in an embodiment of the vehicle intelligent control method of this application;

[0049] Figure 3 This is a schematic diagram of the coordinates of the sound sources in the vehicle in an embodiment of the vehicle intelligent control method of the present application;

[0050] Figure 4 A schematic diagram of a flow chart provided for the second embodiment of the vehicle intelligent control method of the present application;

[0051] Figure 5 A schematic diagram of a process flow provided for the third embodiment of the vehicle intelligent control method of the present application;

[0052] Figure 6 A flowchart of the fourth embodiment of the vehicle intelligent control method of the present application is provided;

[0053] Figure 7 A brief flowchart of an embodiment of the vehicle intelligent control method of the present application;

[0054] Figure 8 This is a schematic diagram of the module structure of the vehicle intelligent control device according to an embodiment of the present application;

[0055] Fig. 9 A schematic diagram of the vehicle structure of the hardware operating environment involved in the vehicle intelligent control method in the embodiment of the present application.

[0056] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0057] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0058] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0059] The main solution of the embodiment of the present application is: determine the voice information in the vehicle through the voice acquisition module, and determine the sound source position and the voice command based on the voice information; when the voice command is a target command, collect the image data in the vehicle through the image acquisition module based on the sound source position; use a preset recognition function to perform target gesture recognition based on the image data in the vehicle to obtain a recognition result; when the recognition result is that the target gesture is recognized, control the vehicle according to the target gesture, the voice command and the sound source position.

[0060] According to the existing technology, when the user wakes up the voice assistant, the array microphone in the cockpit identifies the voice initiator (user) through the sound source localization algorithm, and the TOF camera is activated, focusing on identifying the voice initiator. Wait for the user to complete the voice. The TOF camera emits continuous external light pulses, and then uses a sensor to receive the light returned from the object. It detects the flight (round-trip) time of the light pulse to obtain the distance of the target object, and then uses an algorithm to combine the information of multiple distance points to construct 3D scene data and accurately identify the direction of the gesture. The need to add a Time of Fly camera increases the hardware cost, and the focus is on the newly added TOF sensor hardware, so it does not have the function of OTA upgrade for existing models.

[0061] This application provides a solution that does not require a new TOF camera, but reuses the OMS (Occupant Monitor System) camera for gesture recognition. Without adding any hardware cost, similar functions can be achieved through software, and existing vehicles can be upgraded through OTA.

[0062] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a vehicle system in a vehicle, etc. The following takes the vehicle system as an example to illustrate this embodiment and the following embodiments.

[0063] Based on this, the embodiment of the present application provides a vehicle intelligent control method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the vehicle intelligent control method of the present application.

[0064] In this embodiment, the vehicle intelligent control method is applied to a vehicle, and the vehicle is provided with a voice acquisition module and an image acquisition module. The vehicle intelligent control method includes steps S10 to S40:

[0065] Step S10: determining the voice information in the vehicle through the voice collection module, and determining the sound source position and the voice command according to the voice information.

[0066] It should be noted that the vehicle is mainly equipped with a vehicle system, a voice collection module and an in-cabin monitoring camera (OMS). The vehicle system may include an OMS control module, a voice recognition module and a vehicle control module. The OMS may be installed on the front windshield of the vehicle or at the vehicle central control system. The voice collection module includes multiple microphones to collect the voice information of the user in the vehicle. The voice collection module may include a microphone array, and the microphone array includes multiple microphones, such as Figure 2 As shown, Figure 2Schematic diagram of the distribution of in-vehicle devices, including microphones (microphone 1, microphone 2, microphone 3 and microphone 4) installed at various locations in the car, the OMS camera installed on the center console, the car computer and the T-BOX (wireless gateway).

[0067] In the specific implementation, when the vehicle is started, each microphone in the voice collection module is activated and keeps collecting audio signals until the vehicle is powered off. Each microphone collects audio signals and sends the collected audio signals to the voice recognition module in the form of A2B (Automotive AudioBus, in-vehicle audio bus technology). When the user wakes up the voice recognition function, the voice recognition module determines the location of the sound source through the internal link and sends the location of the sound source to the OMS control module.

[0068] Therefore, the microphone can be used to collect voice information in the car and analyze the voice information to determine the location of the sound source and the voice command.

[0069] In a feasible implementation manner, the step S10 of "determining the sound source position according to the voice information" may include steps A11 to A13:

[0070] Step A11: obtaining the time difference between the microphones in the microphone array according to the sound source localization strategy and the voice information.

[0071] It can be understood that the sound source localization strategy is a TDOA (Time Difference Of Arrival) sound source localization technology, which obtains the time difference between the arrival of the voice signal at two microphones through TDOA and the collected voice information.

[0072] Step A12: Calculate the angle and distance of the sound source from the microphone array according to the time difference.

[0073] In a specific implementation, the angle and distance of the sound source in the car from the microphone array may be calculated based on the time difference, which may specifically include the angle and distance of the sound source from each microphone.

[0074] Step A13: Determine the location of the sound source by using the angle and the distance.

[0075] It should be understood that the location of the sound source in the vehicle can be determined based on specific angles and distances, such as Figure 3 As shown, Figure 3 Schematic diagram of the coordinates of the sound sources in the car. This embodiment is described by taking a four-seater car as an example. The azimuth coordinates of the sound sources are X_11, X_12, X_13, and X_14 respectively.

[0076] After the sound source is localized, the OMS can be driven by the sound source localization to collect image data.

[0077] Step S20: When the voice command is a target command, the in-vehicle image data collected by the image collection module is obtained based on the sound source position.

[0078] It is understandable that after receiving the voice command, the voice command can be semantically analyzed to determine the specific meaning of the command. The target command is a vehicle control command, the command contains a pronoun and the command does not contain specific vehicle component information. The voice command can be analyzed to determine whether it is a target command.

[0079] When the voice command is not a target command, the voice command can be directly sent out, and the command can be forwarded and verified, so that the vehicle can be controlled through voice commands.

[0080] If the voice command is a target command, that is, in addition to voice control, it is also necessary to combine the user's actions so that the vehicle can be controlled based on the actions and voice. Therefore, the in-vehicle image data collected by the image acquisition module can be obtained based on the sound source position.

[0081] It should be noted that the in-vehicle image data is the image data at the sound source position collected by the image acquisition module. The image data collected by the image acquisition module can be screened according to the sound source position to obtain the in-vehicle image data.

[0082] Step S30: performing target gesture recognition based on the in-vehicle image data using a preset recognition function to obtain a recognition result.

[0083] It should be understood that the preset recognition function can be a function for identifying the hands, gestures and directions of the user in the car. After determining the in-car image data corresponding to the sound source position, it can be further determined whether the user in the image makes a gesture and the specific information of the gesture, thereby obtaining a recognition result.

[0084] It should be noted that the recognition result may be that the target gesture is recognized or not recognized, and the target gesture is a valid gesture, which is a gesture with a specific direction or an approximate direction.

[0085] It should be understood that if the target gesture is not recognized, it needs to be processed in a timely manner, such as giving a reminder or re-recognition, to avoid recognition errors.

[0086] In a feasible implementation manner, the step S30 further includes steps S31 to S32:

[0087] Step S31: when the recognition result is that the target gesture is not recognized, generate voice prompt information.

[0088] It is understandable that if the recognition result is that no valid gesture is recognized, it may be that the user did not make a gesture or the user made a gesture but it was not captured by the camera. Therefore, a voice prompt message can be generated, such as: "Your gesture was not recognized, please make the gesture again" or "Your gesture was not recognized, please confirm whether to continue the operation."

[0089] Step S32: Prompt the voice prompt information through the voice collection module, and return the step of responding to the voice information in the car through the voice collection module, and determining the sound source position and voice command according to the voice information.

[0090] In a specific implementation, the voice prompt information can be sent to the microphone array or the microphone corresponding to the sound source position in the microphone array for voice prompting, so as to determine whether the user requires to continue the operation. If the user continues the operation, the user's audio data can be collected again to obtain the sound source position and voice instructions, and further determine whether a valid gesture is recognized. If the user does not want to continue the operation, the interaction ends.

[0091] Step S40: When the recognition result is that the target gesture is recognized, the vehicle is controlled according to the target gesture, the voice command and the sound source position.

[0092] It should be noted that if the recognition result is a valid gesture of the user, the vehicle can be controlled according to the recognized valid gesture, the user's voice command and the location of the sound source. Specific components in the vehicle can be controlled, and components near the user at the sound source location can also be controlled.

[0093] For example, recognition is performed based on the user's effective gestures to determine the user's intention, and the target control component is determined based on the user's voice instructions, the user's intention, and the location of the sound source to control the target control component.

[0094] The present embodiment provides a vehicle intelligent control method, which is applied to a vehicle, wherein the vehicle is provided with a voice acquisition module and an image acquisition module, and the method comprises: determining voice information in the vehicle by means of the voice acquisition module, and determining a sound source position and a voice command according to the voice information; when the voice command is a target command, acquiring in-vehicle image data acquired by the image acquisition module based on the sound source position; performing target gesture recognition based on the in-vehicle image data using a preset recognition function to obtain a recognition result; when the recognition result is recognition of a target gesture, controlling the vehicle according to the target gesture, the voice command and the sound source position. The present solution does not require the addition of additional depth information, and gesture recognition of the user in the vehicle is performed by reusing the image acquisition module, thereby reducing the hardware cost of in-vehicle recognition and improving the recognition effect.

[0095] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 4 , step S40 includes steps S401 to S403:

[0096] Step S401: when the recognition result is that a target gesture is recognized, determining a gesture direction according to the target gesture.

[0097] It should be noted that if the recognition result is that the target gesture is recognized, it proves that the user's gesture command is captured, and the gesture direction is obtained according to the user's effective gesture, and the gesture direction is the user's gesture direction analyzed through the in-vehicle image data. Specifically, when the user's gesture is recognized through the preset recognition function, the user's gesture direction can be directly recognized.

[0098] In a specific implementation, the gesture direction may be left, right, upward, or downward at the sound source location.

[0099] Step S402: determining a target component according to the gesture direction and the sound source position.

[0100] In a specific implementation, the components in the car can be divided in advance to obtain a component set {Target_1}, where {Target_1} is all identifiable vehicle components, including but not limited to: windows in various positions, sunroofs, air conditioners, ambient lights, trunks, fragrances, seat heating, seat ventilation, seat massage, etc.

[0101] It should be noted that after the direction of the gesture is determined, the target component can be determined according to the sound source position and the direction of the gesture, and the target component is the component to be controlled.

[0102] For example, if the gesture direction is left and the sound source position is X_11, the target component is determined to be the front passenger window. For example, if the gesture direction is right and the sound source position is X_21, the target component is determined to be the rear left window, as shown in Table 1 below. Table 1 is a component table inside the car.

[0103] Table 1

[0104] {Target_1} Definition Part designation {Target_1} Definition Part designation T_11 Driver's window T_51 Trunk T_12 Passenger car window T_61 Fragrance T_13 Rear left window (driver's rear window) T_71 Heated driver seat T_14 Rear right window (co-pilot rear window) T_72 Passenger seat heating T_21 Skylight T_73 Heated rear left seat T_31 Front air conditioner lower side air outlet T_74 Rear right seat heating T_32 Rear air conditioning lower side air outlet T_81 Driver seat ventilation T_33 Driver's air conditioning vents T_82 Passenger seat ventilation T_34 Passenger air conditioning vents T_83 Rear left seat ventilation T_35 Rear left air conditioning vent T_84 Rear right seat ventilation T_36 Rear right air conditioning vents T_91 Driver seat massage T_37 Car refrigerator electric door T_92 Passenger seat massage T_41 Ambient Lighting T_93 Rear left seat massage T_42 Central reading light T_94 Rear right seat massage T_43 Rear reading lights

[0105] As shown in Table 2 and Table 3, Table 2 and Table 3 are target component tables corresponding to gesture pointing and sound source position.

[0106] Table 2

[0107]

[0108] Table 3

[0109]

[0110] Step S403: Control the target component according to the voice instruction.

[0111] It is understandable that specific components can be controlled according to the user's voice command. For example, if the voice command is to turn on, the target component is turned on, and if the voice command is to turn off, the target component is turned off.

[0112] In a feasible implementation, step S403 may include steps B11 to B14:

[0113] Step B11: Determine key instruction words according to the voice instruction.

[0114] In a specific implementation, the voice command may be analyzed to determine key command words, such as the key command words such as open, open, close, and shut.

[0115] Step B12: Obtain vehicle component status.

[0116] It should be understood that the status of all components in the vehicle is obtained. For example, if the status of a vehicle component is off, if the user intends to turn off this component, there is no need to take the next step. Therefore, the user's instructions can be responded to more quickly according to the status of the vehicle component.

[0117] Step B13: Determine a target action for the target component according to the key instruction word and the state of the vehicle component.

[0118] It should be noted that whether the target component needs to be controlled can be determined according to the specific instruction and the state of the vehicle component. For example, if the key instruction word is open, the target component is a sunroof, and the state of the sunroof in the vehicle component state is fully open, then the target action for the sunroof is no action, and a voice prompt can be generated to indicate that the sunroof is open. If the key instruction word is open, the target component is a sunroof, and the state of the sunroof in the vehicle component state is not fully open or closed, then the target action is to open the sunroof.

[0119] Step B14: Control the vehicle according to the target action.

[0120] In a specific implementation, after the target action is determined, the vehicle can be controlled according to the target action to complete the interaction with the user.

[0121] Table 4

[0122]

[0123] As shown in Table 4, Table 4 is a relationship table between key instruction words, vehicle component states, gesture pointing, sound source position, pointing component representation, target component, and target action. For example, the key instruction words are open, start, etc., the gesture pointing is left, the sound source position is X_11, and the target component is the front passenger window. If the target component state is closed, the target action is to open the front passenger window. If the target component state is that the front passenger window is not fully opened, the target action is to open the front passenger window. If the target component state is that the front passenger window is fully opened and the front passenger air conditioning outlet is open, the target action is a voice prompt that the front passenger window is open. If the target component is the front passenger air conditioning outlet, the target component state is that the front passenger window is fully opened and the front passenger air conditioning outlet is closed, and the target action is to open the front passenger air conditioning outlet.

[0124] In this embodiment, when the recognition result is that the target gesture is recognized, the gesture direction is determined according to the target gesture; the target component is determined according to the gesture direction and the sound source position; and the target component is controlled according to the voice command. The gesture direction is quickly determined through the target gesture, so that the target component to be controlled is determined according to the gesture direction and the sound source position, thereby improving the control effect.

[0125] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 5 Before step S30, the vehicle intelligent control method further includes steps S21 to S24:

[0126] Step S21: acquiring historical human hand data, performing image learning through the historical human hand data, constructing a two-dimensional human hand model, and converting the two-dimensional human hand model into a first function.

[0127] It should be noted that in order to improve the accuracy and efficiency of gesture recognition, the OMS camera can be used for visual training, so that the two-dimensional human hand model, two-dimensional gestures and two-dimensional finger pointing can be quickly recognized. The content of OMS visual training is mainly to recognize effective gestures.

[0128] Therefore, during training, historical hand data can be collected first. The historical hand data is the hand data of different age groups, different genders, and different races, so as to perform image learning on the historical hand data. Learning objectives: Be able to recognize 10,000 samples through the OMS camera, with a recognition rate of 99.5%, thereby building a two-dimensional hand model, and converting the two-dimensional hand model recognition trained by vision into the function f_hand(). The first function is the function f_hand().

[0129] Step S22: obtaining historical hand gesture data according to the first function and the historical hand data, performing finger pointing gesture learning according to the historical hand gesture data, constructing a two-dimensional gesture recognition model, and converting the two-dimensional gesture recognition model into a second function.

[0130] In the specific implementation, different gesture posture data can be identified based on the function f_hand() to obtain historical gesture posture data. The historical gesture posture data includes various gesture postures. Through image learning of various gesture postures, the focus is on image learning of finger pointing gestures. Learning goal: Be able to recognize 10,000 samples through the OMS camera, with a recognition rate of 99.5% and a false recognition rate of less than 0.5%, so as to train and build a two-dimensional gesture recognition model, and convert the two-dimensional gesture recognition model into the second function f_gesture().

[0131] Step S23: Obtain historical finger pointing gesture data according to the second function and the historical gesture posture data, perform pointing direction learning according to the historical finger pointing gesture data, construct a two-dimensional pointing model, and convert the two-dimensional pointing model into a third function.

[0132] In the specific implementation, based on the second function f_gesture(), the pointing direction of the finger pointing gesture can be learned, that is, different finger pointing gesture data is obtained according to the historical gesture posture data and the second function, so as to learn the pointing direction of the finger pointing gesture, and the learning range is limited to the four directions of up, down, left and right. Learning objectives: 10,000 samples can be recognized through the OMS camera, with a recognition rate of 99% and a false recognition rate of less than 0.5%, and a two-dimensional pointing model can be constructed, and the two-dimensional pointing model of the finger trained by vision can be converted into the third function f_2Ddirection().

[0133] Step S24: obtaining a preset recognition function through the third function.

[0134] In specific implementation, the third function can be written into the OMS control module when the vehicle computer is OTA upgraded through T-BOX. When collecting image data from the OMS camera, the collected data is compared with the function f_2Ddirection(). If it meets the requirements, it is judged as yes and enters the next judgment logic; if it does not meet the requirements, it is judged as no and a voice prompt "I did not see your gesture, please try again" is given.

[0135] This embodiment obtains historical human hand data, and conducts image learning through the historical human hand data to construct a two-dimensional human hand model, and converts the two-dimensional human hand model into a first function; obtains historical hand gesture data according to the first function and the historical human hand data, and conducts finger pointing gesture learning according to the historical hand gesture data to construct a two-dimensional gesture recognition model, and converts the two-dimensional gesture recognition model into a second function; obtains historical finger pointing gesture data according to the second function and the historical hand gesture data, and conducts pointing direction learning according to the historical finger pointing gesture data to construct a two-dimensional pointing model, and converts the two-dimensional pointing model into a third function; obtains a preset recognition function through the third function. The OMS camera is used for visual training, and the training content mainly includes three items: two-dimensional human hand model recognition, two-dimensional gesture recognition, and two-dimensional finger pointing (up, down, left, and right) recognition. Therefore, the user's gesture information can be quickly recognized through the preset recognition function, and the response speed of the control of the vehicle's internal components can be improved.

[0136] Based on the first embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be described in detail later. Figure 6 Before step S20, the vehicle intelligent control method further includes steps S11 to S14:

[0137] Step S11: Analyze the voice command to determine whether the voice command is a vehicle control command.

[0138] It should be noted that the voice command can be parsed according to certain parsing rules, such as keyword splitting, keyword extraction and other rules, so as to determine whether the voice command is a vehicle control command. The vehicle control command is a set {B_1}, that is, the voice control instructions classified as vehicle control commands. The set {A_1} can be written into the voice recognition module in advance. The set {A_1} contains all recognizable valid functional voice commands (functional voice commands include: vehicle control, multimedia, telephone, navigation, AI chat, etc.), among which the subset {B_1} of {A_1} is the voice control instructions classified as vehicle control commands.

[0139] If the voice command is not a vehicle control command, it is determined whether it is a command for other voice functions. If so, other voice functions are directly controlled according to the voice command.

[0140] Step S12: When the voice command is the vehicle control command, determine whether there is vehicle component information in the vehicle control command.

[0141] It should be understood that if the voice command is a vehicle control command, it can be further determined whether there is vehicle component information in the vehicle control command. If there is vehicle component information, the specific component can be directly controlled according to the vehicle control command without the need for subsequent gesture recognition process.

[0142] For example, if the voice command is "Please open the main driver's window", the vehicle component information in the voice command is the main driver's window, and the main driver's window can be directly controlled to open according to this voice command.

[0143] Step S13: When there is no vehicle component information in the vehicle control instruction, determining whether there is a preset pronoun in the vehicle control instruction.

[0144] It is understandable that if there is no vehicle component information in the vehicle control command, it is necessary to further determine whether the vehicle control command contains preset pronouns, which include but are not limited to: "this", "that", "here", "there", "it", etc. The set {Pr_1} can be written in advance in the speech recognition module, and Pr_1} contains all the recognizable preset pronouns.

[0145] When the voice command meets the judgment result of vehicle control command Y, the judgment logic 2 of whether there is vehicle component information in the command can be performed. As long as the voice control command meets the set {Target_1}, it is judged as N, and the specific vehicle component is controlled directly according to this voice control command. If the voice control command meets the set {Pr_1} but does not contain the set {Target_1}, it is judged as Y and enters the next judgment logic. Otherwise, it is invalid voice information, the interaction fails, and the voice end blocks the recognition of this command.

[0146] Step S14: When the preset pronoun exists in the vehicle control instruction, determining that the voice instruction is a target instruction.

[0147] In a specific implementation, if there is a preset pronoun in the vehicle control command, the voice command is determined to be a target command, and the next judgment logic can be entered, that is, whether the target gesture is recognized.

[0148] This embodiment determines whether the voice command is a vehicle control command by parsing the voice command; if the voice command is the vehicle control command, determines whether the vehicle control command contains vehicle component information; if the vehicle control command does not contain vehicle component information, determines whether the vehicle control command contains a preset reference word; if the preset reference word exists in the vehicle control command, determines that the voice command is a target command. Whether the voice command is a target command can be quickly determined according to specific recognition rules. If the voice command is not a target command, there is no need to perform subsequent gesture recognition operations, thereby improving the response speed to the user's voice command.

[0149] For example, in order to help understand the implementation process of the vehicle intelligent control method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 7 , Figure 7 A brief flow chart of a vehicle intelligent control method is provided, specifically: 01: voice wake-up, 02: audio data collection; 03: sound source location based on the collected audio data, after the sound source location is successful, 13: drive the OMS, 14: image data collection through the OMS, and after the sound source location, 04: receive the command, 05: semantic analysis, 06: enter judgment logic 1: whether it is a vehicle control command, 07: if not, identify it as other voice functions; 08: if yes, enter judgment logic 2: identify whether there is component information in the vehicle control command and whether there is a pronoun; 09: if there is component information or no pronoun, confirm the command; 10: send the vehicle control command outward; 11: Command forwarding and verification; 12: Command execution; 19: Voice prompt result; 20: Enter judgment logic 4 to see if the user requires to continue the operation; 21: If not, the interaction ends; If so, return to 02: the step of audio data collection; if there is no component information and there is a pronoun, then 15: Enter judgment logic 3; 16: Identify whether there is a valid gesture based on the collected image data, 17: If there is a valid gesture, guess the gesture direction, and determine the command based on the gesture and voice information, that is, enter step 09, so as to carry out the process of vehicle control command outbound, command forwarding, verification and command execution; 18: If no valid gesture is recognized, 19: Voice prompt that no valid gesture is recognized, 20: Enter judgment logic 4.

[0150] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the vehicle intelligent control method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0151] This application also provides a vehicle intelligent control device, please refer to Figure 8 , the vehicle intelligent control device comprises:

[0152] The determination module 10 is used to determine the voice information in the vehicle through the voice collection module, and determine the sound source position and the voice command according to the voice information.

[0153] The acquisition module 20 is used to acquire in-vehicle image data through an image acquisition module based on the sound source position when the voice instruction is a target instruction.

[0154] The recognition module 30 is used to perform target gesture recognition based on the in-vehicle image data using a preset recognition function to obtain a recognition result.

[0155] The control module 40 is used to control the vehicle according to the target gesture, the voice command and the sound source position when the recognition result is that the target gesture is recognized.

[0156] The vehicle intelligent control device provided by the present application adopts the vehicle intelligent control method in the above embodiment, which can solve the technical problem that the existing vehicle control needs to be combined with a TOF camera, resulting in high costs. Compared with the prior art, the beneficial effects of the vehicle intelligent control device provided by the present application are the same as the beneficial effects of the vehicle intelligent control method provided by the above embodiment, and the other technical features in the vehicle intelligent control device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0157] In one embodiment, the control module 40 is further used to determine the gesture direction according to the target gesture when the recognition result is recognition of the target gesture; determine the target component according to the gesture direction and the sound source position; and control the target component according to the voice command.

[0158] In one embodiment, the control module 40 is further used to determine a key instruction word according to the voice instruction; obtain the status of a vehicle component; determine a target action for the target component according to the key instruction word and the status of the vehicle component; and control the vehicle according to the target action.

[0159] In one embodiment, the recognition module 30 is also used to obtain historical human hand data, perform image learning through the historical human hand data, construct a two-dimensional human hand model, and convert the two-dimensional human hand model into a first function; obtain historical hand gesture data according to the first function and the historical human hand data, perform finger pointing gesture learning according to the historical hand gesture data, construct a two-dimensional gesture recognition model, and convert the two-dimensional gesture recognition model into a second function; obtain historical finger pointing gesture data according to the second function and the historical hand gesture data, perform pointing direction learning according to the historical finger pointing gesture data, construct a two-dimensional pointing model, and convert the two-dimensional pointing model into a third function; obtain a preset recognition function through the third function.

[0160] In one embodiment, the acquisition module 20 is also used to parse the voice command to determine whether the voice command is a vehicle control command; when the voice command is the vehicle control command, determine whether there is vehicle component information in the vehicle control command; when there is no vehicle component information in the vehicle control command, determine whether there is a preset pronoun in the vehicle control command; when the preset pronoun exists in the vehicle control command, determine that the voice command is a target command.

[0161] In one embodiment, the recognition module 30 is also used to generate voice prompt information when the recognition result is that the target gesture is not recognized; prompt the voice prompt information through the voice acquisition module, and return the step of responding to the voice information in the car through the voice acquisition module, and determining the sound source location and voice command according to the voice information.

[0162] In one embodiment, the voice collection module includes a microphone array, and the microphone array includes multiple microphones; the determination module 10 is also used to determine the sound source position according to the voice information, including the following steps: obtaining the time difference between each microphone in the microphone array according to the sound source localization strategy and the voice information; calculating the angle and distance of the sound source from the microphone array according to the time difference; and determining the sound source position through the angle and the distance.

[0163] The present application provides a vehicle, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the vehicle intelligent control method in the above-mentioned embodiment one.

[0164] Reference below Fig. 9 , which shows a schematic diagram of a structure of a vehicle suitable for implementing the embodiment of the present application. The vehicle in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The vehicle shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0165] like Fig. 9As shown, the vehicle may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for vehicle operation are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the vehicle to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a vehicle with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0166] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0167] The vehicle provided by the present application adopts the vehicle intelligent control method in the above embodiment, which can solve the technical problem that the existing vehicle control needs to be combined with a TOF camera, resulting in high costs. Compared with the prior art, the beneficial effects of the vehicle provided by the present application are the same as the beneficial effects of the vehicle intelligent control method provided by the above embodiment, and the other technical features in the vehicle are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0168] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0169] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0170] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the vehicle intelligent control method in the above-mentioned embodiment.

[0171] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0172] The computer-readable storage medium may be included in the vehicle, or may exist independently without being installed in the vehicle.

[0173] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the vehicle, the vehicle: determines the voice information in the vehicle through the voice acquisition module, and determines the sound source position and the voice command based on the voice information; when the voice command is a target command, collects the image data in the vehicle through the image acquisition module based on the sound source position; uses a preset recognition function to perform target gesture recognition based on the image data in the vehicle to obtain a recognition result; when the recognition result is that the target gesture is recognized, controls the vehicle according to the target gesture, the voice command and the sound source position.

[0174] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0175] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0176] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0177] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned vehicle intelligent control method, and can solve the technical problem that the existing vehicle control needs to be combined with a TOF camera, resulting in high costs. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the vehicle intelligent control method provided by the above-mentioned embodiment, and will not be repeated here.

[0178] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned vehicle intelligent control method when executed by a processor.

[0179] The computer program product provided by this application can solve the technical problem that the existing vehicle control needs to be combined with a TOF camera, which leads to high costs. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the vehicle intelligent control method provided in the above embodiment, which will not be repeated here.

[0180] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A vehicle intelligent control method, characterized in that: The vehicle intelligent control method is applied to a vehicle, wherein the vehicle is provided with a voice acquisition module and an image acquisition module, and the method comprises: Determine the voice information in the vehicle through the voice collection module, and determine the sound source location and voice command according to the voice information; When the voice command is a target command, acquiring in-vehicle image data collected by the image collection module based on the sound source position; Based on the in-vehicle image data, a preset recognition function is used to perform target gesture recognition to obtain a recognition result; When the recognition result is that the target gesture is recognized, the vehicle is controlled according to the target gesture, the voice command and the sound source position.

2. The method according to claim 1, characterized in that When the recognition result is that the target gesture is recognized, the step of controlling the vehicle according to the target gesture, the voice command and the sound source position includes: When the recognition result is that the target gesture is recognized, determining the gesture direction according to the target gesture; Determine a target component according to the gesture direction and the sound source position; The target component is controlled according to the voice command.

3. The method according to claim 2, characterized in that The step of controlling the target component according to the voice command comprises: Determining a key instruction word according to the voice instruction; Get vehicle component status; Determine a target action for the target component according to the key instruction word and the state of the vehicle component; The vehicle is controlled according to the target action.

4. The method according to claim 1, characterized in that Before the step of performing target gesture recognition based on the in-vehicle image data using a preset recognition function to obtain a recognition result, the step further includes: Acquire historical human hand data, perform image learning through the historical human hand data, construct a two-dimensional human hand model, and convert the two-dimensional human hand model into a first function; Obtaining historical hand gesture data according to the first function and the historical hand data, performing finger pointing gesture learning according to the historical hand gesture data, constructing a two-dimensional gesture recognition model, and converting the two-dimensional gesture recognition model into a second function; Obtaining historical finger pointing gesture data according to the second function and the historical gesture posture data, performing pointing direction learning according to the historical finger pointing gesture data, constructing a two-dimensional pointing model, and converting the two-dimensional pointing model into a third function; A preset recognition function is obtained through the third function.

5. The method according to claim 1, characterized in that Before the step of acquiring the in-vehicle image data collected by the image collection module based on the sound source position when the voice instruction is a target instruction, the method further includes: parsing the voice command to determine whether the voice command is a vehicle control command; When the voice command is the vehicle control command, determining whether there is vehicle component information in the vehicle control command; When the vehicle component information does not exist in the vehicle control instruction, determining whether the vehicle control instruction contains a preset pronoun; When the preset pronoun exists in the vehicle control instruction, the voice instruction is determined to be a target instruction.

6. The method according to claim 1, characterized in that After the step of performing target gesture recognition using a preset recognition function according to the in-vehicle image data to obtain a recognition result, the method further includes: When the recognition result is that the target gesture is not recognized, generating a voice prompt message; The voice prompt information is prompted by the voice collection module, and the voice collection module responds to the voice information in the car, and the steps of determining the sound source position and the voice command according to the voice information are returned.

7. The method according to any one of claims 1 to 6, characterized in that The voice collection module includes a microphone array, and the microphone array includes a plurality of microphones; The step of determining the sound source position according to the voice information comprises: Obtaining the time difference between the microphones in the microphone array according to the sound source localization strategy and the voice information; Calculate the angle and distance of the sound source from the microphone array according to the time difference; The location of the sound source is determined by the angle and the distance.

8. A vehicle intelligent control device, characterized in that: The device comprises: A determination module, used to determine the voice information in the vehicle through the voice collection module, and determine the sound source position and voice command according to the voice information; A collection module, used for collecting in-vehicle image data through an image collection module based on the sound source position when the voice command is a target command; A recognition module, used to perform target gesture recognition based on the in-vehicle image data using a preset recognition function to obtain a recognition result; A control module is used to control the vehicle according to the target gesture, the voice command and the sound source position when the recognition result is that the target gesture is recognized.

9. A vehicle, characterized in that: The vehicle comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the vehicle intelligent control method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the vehicle intelligent control method as described in any one of claims 1 to 7 are implemented.

Images