Multi-mode-based target identification method and device and vehicle control method and device

Abstract

The invention discloses a multi-modal-based target recognition method and device and a vehicle control method and device.The multi-modal-based target recognition method comprises the steps that a to-be-recognized image is acquired, and the to-be-recognized image comprises a hand image, a body image and a face image; inputting the to-be-recognized image into a preset anaphora model to obtain anaphora information, the anaphora information including a finger anaphora direction, a body seat position and a sight angle; three-dimensional coordinates of the reference information are obtained, and human body multi-modal information is obtained; and identifying a target reference object according to the human body multi-modal information and preset in-vehicle layout information. According to the invention, through three modes of the hand, the face and the body, a more specific target object referred by fingers can be accurately identified, and the accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of automobile technology, and in particular to a multi-modal target recognition method and device, and a vehicle control method and device. Background Art

[0002] With the continuous development of automobile technology, the use of equipment in the car is becoming more and more convenient. At present, by collecting finger image information, the finger's pointing direction can be identified, and combined with the voice operation command and the setting position of the microphone, the finger direction can be identified to indicate the corresponding area, thereby controlling the equipment in the corresponding area. For example, when the identified pointing direction is upward and the voice operation command is open, it means opening the sunroof; when the identified pointing direction is left and the voice operation command is close, it means closing the window, etc.

[0003] However, during the process of implementing the invention, the inventors discovered that the finger image information collected by the existing pointing method is single-frame information, and can only output pointing directions, including pointing up, pointing down, pointing left, pointing right, pointing forward and pointing backward. In addition, the positions and number of microphones installed in the car are limited, and thus a more specific target object cannot be output. For example, if the identified pointing direction is pointing up, it can only be considered to be referring to the sunroof. When other devices are arranged on the top of the car, such as reading lights, it is impossible to identify whether the finger is pointing to the sunroof or the reading light, which is prone to misidentification and has low accuracy. Summary of the invention

[0004] The purpose of this application is to overcome the shortcomings of the prior art in-vehicle reference method that cannot identify more specific targets, is prone to misidentification, and has low accuracy, and to provide a multi-modal target recognition method and device, and a vehicle control method and device.

[0005] The technical solution of the present application provides a multi-modal target recognition method, including:

[0006] Acquire an image to be identified, wherein the image to be identified includes a hand image, a body image, and a face image;

[0007] Input the image to be identified into a preset reference model to obtain reference information, wherein the reference information includes the finger pointing direction, the body seat position and the sight angle;

[0008] Acquire the three-dimensional coordinates of the reference information to obtain multimodal information of the human body;

[0009] The target referent is identified according to the multimodal information of the human body and the preset in-vehicle layout information.

[0010] Furthermore, the preset reference model includes a classification model, and the step of inputting the image to be identified into the preset reference model to obtain the reference information includes:

[0011] The hand image is input into the classification model to obtain the finger pointing direction.

[0012] Furthermore, the preset reference model includes a body position calculation model, and the step of inputting the image to be identified into the preset reference model to obtain the reference information includes:

[0013] The body image is input into the body occupancy calculation model to obtain the body seat position.

[0014] Furthermore, the preset reference model includes a sight angle calculation model, and the step of inputting the image to be identified into the preset reference model to obtain the reference information includes:

[0015] The facial image is input into the sight line angle calculation model to obtain the sight line angle.

[0016] Furthermore, the acquiring of the three-dimensional coordinates of the reference information to obtain multimodal information of the human body includes:

[0017] Acquire the three-dimensional coordinates of the reference information to generate multimodal three-dimensional coordinate information, wherein the multimodal three-dimensional coordinate information includes the three-dimensional coordinates of the hand, the three-dimensional coordinates of the body, and the three-dimensional coordinates of the face;

[0018] Matching the hand image and the body image according to the preset first key point of the hand three-dimensional coordinates, the hand three-dimensional coordinates and the body three-dimensional coordinates to generate hand-body multimodal information, and determining the hand type, wherein the hand type includes left hand and right hand;

[0019] matching the facial image and the body image according to a preset second key point of the facial image, the facial three-dimensional coordinates, and the body three-dimensional coordinates to generate facial-body multimodal information;

[0020] The hand type is used as reference information, and the hand-body multimodal information and the face-body multimodal information are merged to generate the human body multimodal information.

[0021] Further, the hand type is used as reference information to merge the hand-body multimodal information and the face-body multimodal information to generate the human body multimodal information, and then further includes:

[0022] The overlapping finger pointing directions are deleted according to the hand three-dimensional coordinates, the body three-dimensional coordinates and / or the face three-dimensional coordinates.

[0023] Further, the hand type is used as reference information to merge the hand-body multimodal information and the face-body multimodal information to generate the human body multimodal information, and then further includes:

[0024] The human body multimodal information is inserted into a sliding window according to a preset window length. If the number of frames of the same pointing direction in the sliding window meets a preset frame number threshold, the same pointing direction is the finger pointing direction.

[0025] The technical solution of the present invention also provides a vehicle control method, comprising:

[0026] Acquire the voice operation instruction and the target referent based on the multimodal target recognition method as described above;

[0027] The target object is controlled according to the voice operation instruction.

[0028] The technical solution of the present invention also provides a multi-modal target recognition device, comprising:

[0029] A first acquisition unit, configured to acquire an image to be recognized, wherein the image to be recognized includes a hand image, a body image, and a face image;

[0030] A reference information generating unit, used for inputting the image to be identified into a preset reference model to obtain reference information, wherein the reference information includes a finger pointing direction, a body seat position and a sight angle;

[0031] A human body multimodal information generating unit, used to obtain the three-dimensional coordinates of the reference information to obtain the human body multimodal information;

[0032] The target referent identification unit is used to identify the target referent according to the multimodal information of the human body and the preset in-vehicle layout information.

[0033] The technical solution of the present invention also provides a vehicle control device, comprising:

[0034] A second acquisition unit, used to acquire the voice operation instruction and the target referent based on the multimodal target recognition method as described above;

[0035] A control unit is used to control the target referent according to the voice operation instruction.

[0036] The technical solution of the present invention also provides a storage medium, which stores computer instructions. When a computer executes the computer instructions, it is used to execute all steps of the multi-modal target recognition method or the vehicle control method as described above.

[0037] The technical solution of the present invention further provides an electronic device, comprising:

[0038] at least one processor; and,

[0039] a memory communicatively connected to the at least one processor; wherein,

[0040] The memory stores instructions that can be executed by the one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multi-modal target recognition method as described above.

[0041] The technical solution of the present invention further provides an electronic device, characterized in that it includes:

[0042] at least one processor; and,

[0043] a memory communicatively connected to the at least one processor; wherein,

[0044] The memory stores instructions that can be executed by the one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the vehicle control method as described above.

[0045] After adopting the above technical solution, the following beneficial effects are achieved: by acquiring the image to be identified, the image to be identified includes a hand image, a body image and a face image, the image to be identified is input into a preset reference model to obtain reference information, and the three-dimensional coordinates of the reference information are obtained to obtain human multimodal information, and the target reference object is identified according to the human multimodal information and the preset in-vehicle layout information, and the more specific target object referred to by the finger can be accurately identified through the three modes of hand, face and body, thereby improving accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The disclosure of the present application will become easier to understand with reference to the accompanying drawings. It should be understood that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present application. In the drawings:

[0047] Figure 1 This is a workflow diagram of a multi-modal target recognition method provided in Example 1 of the present invention;

[0048] Figure 2 This is a workflow diagram of a multi-modal target recognition method provided in Embodiment 2 of the present invention;

[0049] Figure 3 is a working flow chart of a vehicle control method provided in Embodiment 3 of the present invention;

[0050] Figure 4 is a schematic structural diagram of a multi-modal target recognition device provided in Embodiment 4 of the present invention;

[0051] Figure 5 is a structural schematic diagram of a vehicle control device provided by Embodiment 5 of the present invention;

[0052] Figure 6 Schematic diagram of the hardware structure of an electronic device for multimodal target recognition according to Embodiment 7 of the present invention;

[0053] Figure 7 It is a schematic diagram of the hardware structure of an electronic device for vehicle control provided in Embodiment 8 of the present invention. DETAILED DESCRIPTION

[0054] The specific implementation of the present application is further described below with reference to the accompanying drawings.

[0055] It is easy to understand that according to the technical solution of the present application, without changing the essential spirit of the present application, a variety of structural modes and implementation modes that can be replaced by those skilled in the art can be replaced with each other. Therefore, the following specific implementation modes and drawings are only exemplary descriptions of the technical solution of the present application, and should not be regarded as the entirety of the present application or as a limitation or restriction on the technical solution of the application.

[0056] The directional terms such as up, down, left, right, front, back, front, back, top, bottom, etc. mentioned or may be mentioned in this specification are defined relative to the structures shown in the drawings. They are relative concepts and may change accordingly according to their different positions and different usage conditions. Therefore, these or other directional terms should not be interpreted as restrictive terms. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0057] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above-mentioned in this application can be understood according to the specific circumstances.

[0058] The reference in this application refers to the target object that is positioned by finger pointing instead of positioning, and the multimodal information of the human body refers to the image information of multiple modes of the human body, including three modal image information of hands, faces and human bodies.

[0059] Embodiment 1

[0060] like Figure 1 As shown, Figure 1This is a workflow diagram of a multimodal target recognition method provided by the first embodiment of the present invention, including:

[0061] Step S101: acquiring an image to be identified, the image to be identified including a hand image, a body image and a face image;

[0062] Step S102: inputting the image to be recognized into a preset reference model to obtain reference information, which includes the finger pointing direction, body seat position and sight angle;

[0063] Step S103: Acquire the three-dimensional coordinates of the reference information to obtain multimodal information of the human body;

[0064] Step S104: identifying the target referent according to the multimodal information of the human body and the preset in-vehicle layout information.

[0065] Specifically, when an image to be recognized is detected by a camera installed in the vehicle, the controller executes step S101 to obtain the image to be recognized, which includes a hand image, a body image and a face image; then executes step S102 to input the image to be recognized into a preset reference model to obtain reference information; then executes step S103 to obtain the three-dimensional coordinates of the reference information through a time of flight (TOF) camera or a binocular camera, and obtain multimodal information of the human body; finally, executes step S104 to combine the multimodal information of the human body with the preset in-vehicle layout information to identify the target reference object, so as to accurately identify the more specific target object referred to by the finger, such as the left window, the right window, the front sunroof, the rear sunroof, the second row left reading lights, the second row right reading lights, and other devices in the vehicle that can be automatically controlled to open or close.

[0066] The preset reference model may adopt an existing reference model. Preferably, the preset reference model includes a classification model, a sight angle calculation model and a body position calculation model.

[0067] Among them, the in-vehicle layout information refers to the location information of devices inside the vehicle that can be automatically controlled to open or close, such as seats, windows, sunroofs, ambient lights, refrigerators, seat screens, reading lights, etc. The in-vehicle layout information is related to the size and type of the vehicle. The in-vehicle layout information can be determined when the vehicle leaves the factory, and the in-vehicle layout information can be stored in the controller by setting a configuration file.

[0068] The controller includes a micro control unit, an electronic control unit, a body controller, etc. The controller is preferably a vehicle control unit.

[0069] In this embodiment, by acquiring an image to be identified, the image to be identified includes a hand image, a body image and a face image, the image to be identified is input into a preset reference model to obtain reference information, and the three-dimensional coordinates of the reference information are obtained to obtain multimodal information of the human body, and the target reference object is identified according to the multimodal information of the human body and the preset in-vehicle layout information, and the more specific target object referred to by the finger can be accurately identified through the three modalities of hand, face and body, thereby improving accuracy.

[0070] Embodiment 2

[0071] like Figure 2 As shown, Figure 2 A workflow diagram of a multimodal target recognition method provided in Embodiment 2 of the present invention includes:

[0072] Step S201: obtaining an image to be recognized;

[0073] Step S202: input the hand image into the classification model to obtain the finger pointing direction;

[0074] Step S203: inputting the body image into the body occupancy calculation model to obtain the body seat position;

[0075] Step S204: inputting the facial image into the sight line angle calculation model to obtain the sight line angle;

[0076] Step S205: acquiring the three-dimensional coordinates of the reference information, and generating multimodal three-dimensional coordinate information, wherein the multimodal three-dimensional coordinate information includes the three-dimensional coordinates of the hand, the three-dimensional coordinates of the body, and the three-dimensional coordinates of the face;

[0077] Step S206: matching the hand image and the body image according to the preset first key point of the hand three-dimensional coordinates, the hand three-dimensional coordinates and the body three-dimensional coordinates, generating hand-body multimodal information, and determining the hand type, which includes left hand and right hand;

[0078] Step S207: matching the facial image and the body image according to the preset second key point of the facial image, the facial three-dimensional coordinates and the body three-dimensional coordinates to generate face-body multimodal information;

[0079] Step S208: using the hand type as reference information, merging the hand-body multimodal information and the face-body multimodal information to generate human body multimodal information;

[0080] Step S209: Identify the target referent according to the multimodal information of the human body and the preset in-vehicle layout information.

[0081] Specifically, when an image to be identified is detected by a camera installed in the car, the controller first executes step S201 to obtain the image to be identified. If the image to be identified is a hand image, step S202 is executed to input the hand image into the classification model to identify the pointing direction of the finger, including upward, upper left, upper right, downward, lower left, lower right, left, left front, left rear, right, right front, right rear and other directions; if the image to be identified is a body image, step S203 is executed to input the body image into the body occupancy calculation model to calculate the seat in the car where the body is located. The seat position of the body is obtained by using the TOF camera or the binocular camera to obtain the seat position of the body; if the image to be recognized is a face image, step S204 is executed to input the face image into the sight angle calculation model to calculate the sight angle; secondly, step S205 is executed to obtain the three-dimensional coordinates of the reference information through the TOF camera or the binocular camera to generate multimodal three-dimensional coordinate information, and the multimodal three-dimensional coordinate information includes the three-dimensional coordinates of the hand, the three-dimensional coordinates of the body and the three-dimensional coordinates of the face; thirdly, step S206 is executed to use the three-dimensional coordinates of the preset first key point as a reference, and the hand image and the body image are converted into a three-dimensional coordinate system according to the hand three-dimensional coordinates. The three-dimensional coordinates of the face and the three-dimensional coordinates of the body are matched to generate hand-body multimodal information, so that the hand is bound to the body and the hand type is determined to be the left hand or the right hand. For example, when the right hand of the passenger on the right side of the second row is identified to point to the lower right, it is considered to be a misidentification. Since the left hand usually points to the right (including the upper right, lower right, and right) and the right hand points to the left (including the upper left, lower left, and left), misidentification is prevented and the accuracy is further improved. Next, step S207 is executed to match the facial image and the body image with the three-dimensional coordinates of the preset second key point as a reference. For example, matching is performed based on the three-dimensional coordinates of the face and the three-dimensional coordinates of the body to generate face-body multimodal information, so that the face and the body are bound; then, step S208 is executed to use the hand type as reference information to merge the hand-body multimodal information and the face-body multimodal information to generate human body multimodal information, forming three modal information of hand-face-body; finally, step S209 is executed to identify the target referent according to the human body multimodal information and the preset in-vehicle layout information, so as to accurately identify the more specific target object referred to by the finger.

[0082] Among them, the classification model, body occupancy calculation model and sight angle calculation model are all existing technologies, and their working principles are not described here in detail.

[0083] Among them, the order of steps S202-S204 can be interchanged or performed simultaneously. When a hand image is acquired, step S202 is executed, when a face image is acquired, step S203 is executed, and when a body image is acquired, step S204 is executed. This does not affect the actual effect of this embodiment.

[0084] Among them, the preset first key point is preferably the wrist, and the preset second key point is preferably the head.

[0085] In this embodiment, by acquiring an image to be identified, the image to be identified includes a hand image, a body image and a face image, the image to be identified is input into a preset reference model to obtain reference information, and the three-dimensional coordinates of the reference information are obtained to obtain multimodal information of the human body, and the target reference object is identified according to the multimodal information of the human body and the preset in-vehicle layout information, and the more specific target object referred to by the finger is accurately identified through the three modalities of hand, face and body; at the same time, by merging the hand, face and body, multimodal information of the human body is generated to prevent misrecognition and further improve accuracy.

[0086] In one embodiment, step S208 further includes:

[0087] The overlapping finger pointing directions are deleted according to the hand 3D coordinates, the face 3D coordinates and / or the body 3D coordinates.

[0088] Specifically, whether there is overlap is determined based on the three-dimensional coordinates of the hand, the three-dimensional coordinates of the face and / or the three-dimensional coordinates of the body. For example, when the hand is close to the human body or face, it means there is overlap, and the overlapping finger directions are deleted, thereby further preventing misrecognition and improving accuracy.

[0089] In one embodiment, step S208 further includes:

[0090] Inserting the multimodal information of the human body into the sliding window according to the preset window length, if the number of frames with the same pointing direction in the sliding window meets the preset frame number threshold, the same pointing direction is the finger pointing direction;

[0091] Specifically, the preset window length refers to the window length of the sliding window. The human multimodal information is split according to the preset window length and inserted into the sliding window. It is determined whether the number of frames with the same reference direction in the sliding window meets the preset frame number threshold. If not, it means that the reference is invalid. If it meets, it means that the reference is valid. The same reference direction is the finger reference direction, which further prevents misidentification and improves accuracy. Taking the preset window length of 5 and the preset frame number threshold of 3 as an example, the most recent 5 frames of human multimodal information are inserted into the first to fifth window positions in the sliding window. When 2 frames are detected to be upward and 3 frames to the left, it means that the reference is valid and the finger reference direction is to the left, and so on.

[0092] Embodiment 3

[0093] like Figure 3 As shown, Figure 3 This is a working flow chart of a vehicle control method provided by Embodiment 3 of the present invention, comprising:

[0094] Step S301: Acquire the voice operation instruction and the target referent based on the multimodal target recognition method as described above;

[0095] Step S302: Control the target object according to the voice operation instruction.

[0096] Specifically, when a voice operation instruction and an identified target object are received, the controller controls the target object according to the voice operation instruction. For example, when the voice operation instruction is to open and the identified target object is the left window, the controller opens the left window; when the voice operation instruction is to close and the identified target object is the left window, the controller closes the left window, thereby being able to accurately control the more specific target object indicated by the finger, prevent misoperation, and improve accuracy.

[0097] In this embodiment, by acquiring voice operation instructions and the target referent based on the multimodal target recognition method as described above, the target referent is controlled according to the voice operation instructions, so that a more specific target object pointed by the finger can be accurately controlled, thereby preventing misoperation and improving accuracy.

[0098] Embodiment 4

[0099] like Figure 4 As shown, Figure 4 : is a schematic diagram of a structure of a multi-modal target recognition device provided by Embodiment 4 of the present invention, comprising:

[0100] A first acquisition unit 401 is used to acquire an image to be recognized, where the image to be recognized includes a hand image, a body image, and a face image;

[0101] The reference information generating unit 402 is used to input the image to be recognized into a preset reference model to obtain reference information, which includes the finger pointing direction, the body seat position and the sight angle;

[0102] A human body multimodal information generating unit 403 is used to obtain the three-dimensional coordinates of the reference information to obtain the human body multimodal information;

[0103] The target referent identification unit 404 is used to identify the target referent according to the multimodal information of the human body and the preset in-vehicle layout information.

[0104] In this embodiment, by acquiring an image to be identified, the image to be identified includes a hand image, a body image and a face image, the image to be identified is input into a preset reference model to obtain reference information, and the three-dimensional coordinates of the reference information are obtained to obtain multimodal information of the human body, and the target reference object is identified according to the multimodal information of the human body and the preset in-vehicle layout information, and the more specific target object referred to by the finger can be accurately identified through the three modalities of hand, face and body, thereby improving accuracy.

[0105] Optionally, the preset reference model includes a classification model, and the reference information generating unit 402 is further used to:

[0106] The hand image is input into the classification model to obtain the finger pointing direction.

[0107] Optionally, the preset reference model includes a body occupancy calculation model, and the reference information generating unit 402 is further used to:

[0108] The body image is input into the body occupancy calculation model to obtain the body seat position.

[0109] Optionally, the preset reference model includes a sight angle calculation model, and the reference information generating unit 402 is further used to:

[0110] The facial image is input into the gaze angle calculation model to obtain the gaze angle.

[0111] Optionally, the human body multimodal information generating unit 403 is further used for:

[0112] Acquire the three-dimensional coordinates of the reference information and generate multimodal three-dimensional coordinate information, the multimodal three-dimensional coordinate information including the three-dimensional coordinates of the hand, the three-dimensional coordinates of the body and the three-dimensional coordinates of the face;

[0113] Matching the hand image and the body image according to the preset first key point of the hand three-dimensional coordinates, the hand three-dimensional coordinates and the body three-dimensional coordinates, generating hand-body multimodal information, and determining the hand type, which includes left hand and right hand;

[0114] Matching the facial image and the body image according to the preset second key point of the facial image, the facial three-dimensional coordinates and the body three-dimensional coordinates to generate facial-body multimodal information;

[0115] Taking the hand type as reference information, the hand-body multimodal information and the face-body multimodal information are merged to generate human body multimodal information.

[0116] Optionally, the human body multimodal information generating unit 403 is further used for:

[0117] Overlapping finger pointing directions are deleted according to the hand 3D coordinates, the body 3D coordinates and / or the face 3D coordinates.

[0118] Optionally, the human body multimodal information generating unit 403 is further used for:

[0119] The multimodal information of the human body is inserted into the sliding window according to the preset window length. If the number of frames with the same pointing direction in the sliding window meets the preset frame number threshold, the same pointing direction is the finger pointing direction.

[0120] Embodiment 5

[0121] like Figure 5 As shown, Figure 5: is a structural schematic diagram of a vehicle control device provided by Embodiment 5 of the present invention, comprising:

[0122] A second acquisition unit 501 is used to acquire the voice operation instruction and the target referent based on the multimodal target recognition method as described above;

[0123] The control unit 502 is used to control the target referent according to the voice operation instruction.

[0124] In this embodiment, by acquiring voice operation instructions and the target referent based on the multimodal target recognition method as described above, the target referent is controlled according to the voice operation instructions, so that a more specific target object pointed by the finger can be accurately controlled, thereby preventing misoperation and improving accuracy.

[0125] Embodiment 6

[0126] The technical solution of the present application also provides a storage medium, which stores computer instructions. When a computer executes the computer instructions, it is used to execute all steps of the multimodal target recognition method or vehicle control method in any of the aforementioned embodiments.

[0127] Embodiment 7

[0128] Figure 6 An electronic device of the present application is shown, comprising:

[0129] at least one processor 601; and,

[0130] A memory 602 is communicatively connected to the at least one processor 601; wherein,

[0131] The memory 602 stores instructions that can be executed by the at least one processor 601, and the instructions are executed by the at least one processor 601 so that the at least one processor 601 can execute all steps of the multimodal target recognition method in any of the aforementioned method embodiments.

[0132] The electronic device is preferably an on-board electronic control unit (ECU), and further a microcontroller unit (MCU) in the on-board electronic control unit.

[0133] Figure 6 Take a processor 601 as an example:

[0134] The electronic device may further include: an input device 603 and an output device 604 .

[0135] The processor 601, the memory 602, the input device 603 and the output device 604 may be connected via a bus or other means, and the figure takes the connection via a bus as an example.

[0136] The memory 602 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the multimodal target recognition method in the embodiment of the present application, for example, Figure 1 or Figure 2 The processor 601 executes various functional applications and data processing by running the non-volatile software programs, instructions and modules stored in the memory 602, that is, the multi-modal target recognition method in the above embodiment is implemented.

[0137] The memory 602 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created based on the use of a multimodal target recognition method, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 602 may optionally include a memory remotely arranged relative to the processor 601, and these remote memories may be connected to a device for executing a multimodal target recognition method via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0138] The input device 603 can receive user clicks and generate signal inputs related to user settings and function controls based on the multimodal object recognition method. The output device 604 can include display devices such as display screens.

[0139] The one or more modules are stored in the memory 602, and when executed by the one or more processors 601, the multi-modal target recognition method in any of the above method embodiments is executed.

[0140] Embodiment 8

[0141] Figure 7 An electronic device of the present application is shown, comprising:

[0142] at least one processor 701; and,

[0143] A memory 702 that is communicatively connected to the at least one processor 701; wherein,

[0144] The memory 702 stores instructions that can be executed by the at least one processor 701, and the instructions are executed by the at least one processor 701 so that the at least one processor 701 can execute all steps of the vehicle control method in any of the aforementioned method embodiments.

[0145] The electronic device is preferably an on-board electronic control unit (ECU), and further a microcontroller unit (MCU) in the on-board electronic control unit.

[0146] Figure 7 Take a processor 701 as an example:

[0147] The electronic device may further include: an input device 703 and an output device 704 .

[0148] The processor 701, the memory 702, the input device 703 and the output device 704 may be connected via a bus or other means, and the figure takes the connection via a bus as an example.

[0149] The memory 702 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the vehicle control method in the embodiment of the present application, for example, Figure 3 The processor 701 executes various functional applications and data processing by running the non-volatile software programs, instructions and modules stored in the memory 702, that is, implements the vehicle control method in the above embodiment.

[0150] The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created according to the use of the vehicle control method, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 702 may optionally include a memory remotely arranged relative to the processor 701, and these remote memories may be connected to the device for executing the vehicle control method via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0151] The input device 703 can receive input user clicks and generate signal inputs related to user settings and function controls of the vehicle control method. The output device 704 can include display devices such as display screens.

[0152] The one or more modules are stored in the memory 702 , and when run by the one or more processors 701 , the vehicle control method in any of the above method embodiments is executed.

[0153] The above description is only the principle and preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this field, the implementation methods obtained by appropriately combining the technical solutions disclosed in different embodiments are also included in the technical scope of the present invention. On the basis of the principle of the present application, several other variations can be made, which should also be regarded as the protection scope of the present application.

Claims

1. A multimodal target recognition method, characterized in that: include: Acquire an image to be identified, wherein the image to be identified includes a hand image, a body image, and a face image; Input the image to be identified into a preset reference model to obtain reference information, wherein the reference information includes the finger pointing direction, the body seat position and the sight angle; Acquire the three-dimensional coordinates of the reference information to obtain multimodal information of the human body; The target referent is identified according to the multimodal information of the human body and the preset in-vehicle layout information.

2. The multimodal target recognition method according to claim 1, characterized in that: The preset reference model includes a classification model, and the inputting of the image to be identified into the preset reference model to obtain the reference information includes: The hand image is input into the classification model to obtain the finger pointing direction.

3. The multimodal target recognition method according to claim 1, characterized in that: The preset reference model includes a body position calculation model, and the inputting of the image to be identified into the preset reference model to obtain reference information includes: The body image is input into the body occupancy calculation model to obtain the body seat position.

4. The multimodal target recognition method according to claim 1, characterized in that: The preset reference model includes a sight angle calculation model, and the inputting of the image to be identified into the preset reference model to obtain the reference information includes: The facial image is input into the sight line angle calculation model to obtain the sight line angle.

5. The multimodal target recognition method according to claim 1, characterized in that: The step of acquiring the three-dimensional coordinates of the reference information to obtain multimodal information of the human body includes: Acquire the three-dimensional coordinates of the reference information to generate multimodal three-dimensional coordinate information, wherein the multimodal three-dimensional coordinate information includes the three-dimensional coordinates of the hand, the three-dimensional coordinates of the body, and the three-dimensional coordinates of the face; Matching the hand image and the body image according to the preset first key point of the hand three-dimensional coordinates, the hand three-dimensional coordinates and the body three-dimensional coordinates to generate hand-body multimodal information, and determining the hand type, wherein the hand type includes left hand and right hand; matching the facial image and the body image according to a preset second key point of the facial image, the facial three-dimensional coordinates, and the body three-dimensional coordinates to generate facial-body multimodal information; The hand type is used as reference information, and the hand-body multimodal information and the face-body multimodal information are merged to generate the human body multimodal information.

6. The multimodal target recognition method according to claim 5, characterized in that: The method further comprises: taking the hand type as reference information, merging the hand-body multimodal information and the face-body multimodal information to generate the human body multimodal information; and then: The overlapping finger pointing directions are deleted according to the hand three-dimensional coordinates, the body three-dimensional coordinates and / or the face three-dimensional coordinates.

7. The multimodal target recognition method according to claim 5 or 6, characterized in that: The method further comprises: taking the hand type as reference information, merging the hand-body multimodal information and the face-body multimodal information to generate the human body multimodal information; and then: The human body multimodal information is inserted into a sliding window according to a preset window length. If the number of frames of the same pointing direction in the sliding window meets a preset frame number threshold, the same pointing direction is the finger pointing direction.

8. A vehicle control method, characterized in that: include: Acquire the voice operation instruction and the target referent based on the multimodal target recognition method according to any one of claims 1 to 7; The target object is controlled according to the voice operation instruction.

9. A multi-modal target recognition device, characterized in that: include: A first acquisition unit, configured to acquire an image to be recognized, wherein the image to be recognized includes a hand image, a body image, and a face image; A reference information generating unit, used for inputting the image to be identified into a preset reference model to obtain reference information, wherein the reference information includes a finger pointing direction, a body seat position and a sight angle; A human body multimodal information generating unit, used to obtain the three-dimensional coordinates of the reference information to obtain the human body multimodal information; The target referent identification unit is used to identify the target referent according to the multimodal information of the human body and the preset in-vehicle layout information.

10. A vehicle control device, characterized in that: include: A second acquisition unit, configured to acquire the voice operation instruction and the target referent according to the multimodal target recognition method as described in any one of claims 1 to 7; A control unit is used to control the target referent according to the voice operation instruction.

11. A storage medium, characterized in that: The storage medium stores computer instructions, which, when executed by a computer, are used to execute all steps of the multimodal target recognition method as described in any one of claims 1 to 7 or the vehicle control method as described in claim 8.

12. An electronic device for multimodal target recognition, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multi-modal target recognition method as described in any one of claims 1-7.

13. An electronic device for vehicle control, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the vehicle control method as claimed in claim 8 .

Images