Intelligent cabin interaction method, system and device based on holographic projection and medium

Abstract

The invention discloses an intelligent cabin interaction method, system and device based on holographic projection and a medium, and the method comprises the steps: obtaining the position information and orientation information of a target person in response to a voice triggering instruction / gesture triggering instruction of the target person, determining an interaction plane according to the position information and the orientation information, a preset cabin interaction interface is displayed on the interaction plane through a holographic projection technology; and in response to a voice control instruction / gesture control instruction of the target person, identifying corresponding interaction information according to the voice control instruction / gesture control instruction, and further adjusting a cabin interaction interface according to the interaction information, or controlling a cabin system according to the interaction information. The method improves the driving safety and the driving experience of the user, and can be widely applied to the technical field of intelligent cabins.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent cockpits, and in particular to an intelligent cockpit interaction method, system, device and medium based on holographic projection. Background Art

[0002] Currently, vehicle display devices mainly consist of flat displays such as LCD, LED, and HUD, and related control operations are mostly completed through buttons and touch. However, 2D flat displays lack a three-dimensional sense, resulting in a poor user experience, and button-based and touch-based operations cannot achieve remote control, causing rear passengers to be unable to operate the central control screen to interact with the cockpit system, while the driver needs to divert their line of sight from the road ahead to operate the central control screen when driving, affecting driving safety and the user's driving and riding experience. Summary of the Invention

[0003] The purpose of the present invention is to solve at least to a certain extent one of the technical problems existing in the prior art.

[0004] To this end, one object of the embodiments of the present invention is to provide an intelligent cockpit interaction method based on holographic projection, which improves driving safety and the user's driving and riding experience.

[0005] Another object of the embodiments of the present invention is to provide an intelligent cockpit interaction system based on holographic projection.

[0006] To achieve the above technical objectives, the technical solutions adopted by the embodiments of the present invention include:

[0007] In a first aspect, the embodiments of the present invention provide an intelligent cockpit interaction method based on holographic projection, including the following steps:

[0008] In response to a voice trigger instruction / gesture trigger instruction of a target person, obtain the position information and orientation information of the target person, determine an interaction plane based on the position information and the orientation information, and then display a preset cockpit interaction interface on the interaction plane through holographic projection technology;

[0009] In response to the voice control instruction / gesture control instruction of the target person, identify the corresponding interaction information according to the voice control instruction / gesture control instruction, and then adjust the cockpit interaction interface according to the interaction information, or control the cockpit system according to the interaction information.

[0010] Further, in an embodiment of the present invention, in response to a voice trigger instruction of a target person, obtaining the position information and orientation information of the target person specifically includes:

[0011] Obtain the first voice information and the first image information of each driver and passenger through the voice collection device and the image collection device set in each cockpit area;

[0012] Perform voice recognition on the first voice information to obtain a first voice command, and match the first voice command with a preset voice trigger command;

[0013] When the matching is successful, determine the corresponding driver and passenger as the target person, and determine the first image information of the target person as the target image information;

[0014] Perform human key point detection on the target image information to obtain multiple human key points of the target person, and then determine the position information and the orientation information according to the human key points.

[0015] Further, in an embodiment of the present invention, in response to a gesture trigger command of a target person, obtain the position information and the orientation information of the target person, which specifically includes:

[0016] Obtain the first image information of each driver and passenger through the image collection device set in each cockpit area;

[0017] Perform gesture recognition on the first image information to obtain a first gesture command, and match the first gesture command with a preset gesture trigger command;

[0018] When the matching is successful, determine the corresponding driver and passenger as the target person, and determine the corresponding first image information as the target image information;

[0019] Perform human key point detection on the target image information to obtain multiple human key points of the target person, and then determine the position information and the orientation information according to the human key points.

[0020] Further, in an embodiment of the present invention, determine an interaction plane according to the position information and the orientation information, and then display a preset cockpit interaction interface on the interaction plane through holographic projection technology, which specifically includes:

[0021] Determine the target person coordinates according to the position information, and determine the target direction according to the orientation information;

[0022] Obtain a preset interaction distance, and determine the interaction plane according to the target person coordinates, the target direction and the interaction distance, so that the distance from the target person coordinates to the interaction plane is equal to the interaction distance, and the normal vector of the interaction plane is consistent with the target direction;

[0023] Obtain the holographic image of the preset cockpit interaction interface, and project the holographic image on the interaction plane through a plurality of laser light sources arranged in the cockpit area corresponding to the target person.

[0024] Further, in an embodiment of the present invention, in response to the voice control instruction of the target person, identify the corresponding interaction information according to the voice control instruction, and then adjust the cockpit interaction interface according to the interaction information, or control the cockpit system according to the interaction information, which specifically includes:

[0025] Obtain the second voice information of the target person through a voice collection device arranged in the cockpit area corresponding to the target person;

[0026] Perform voice recognition on the second voice information to obtain the voice control instruction, and match the voice control instruction with a preset voice interaction database to obtain the corresponding interaction information, and then determine the interaction type of the interaction information;

[0027] When the interaction type is an interface operation type, determine the corresponding interaction interface operation, and then adjust the cockpit interaction interface according to the interaction interface operation;

[0028] When the interaction type is a cockpit control type, determine the corresponding cockpit control operation, and then control the cockpit system according to the cockpit control operation.

[0029] Further, in an embodiment of the present invention, in response to the gesture control instruction of the target person, identify the corresponding interaction information according to the gesture control instruction, and then adjust the cockpit interaction interface according to the interaction information, or control the cockpit system according to the interaction information, which specifically includes:

[0030] Obtain the second image information of the target person through an image collection device arranged in the cockpit area corresponding to the target person;

[0031] Perform gesture recognition on the second image information to obtain the gesture control instruction, and perform human key point detection on the second image information to obtain the positions of multiple hand key points;

[0032] Determine the interface elements at the corresponding positions on the cockpit interaction interface according to the positions of the hand key points, and determine the interaction information and the interaction type of the interaction information according to the interface elements and the gesture control instruction;

[0033] When the interaction type is an interface operation type, determine the corresponding interaction interface operation, and then adjust the cockpit interaction interface according to the interaction interface operation;

[0034] When the interaction type is a cockpit control type, determine the corresponding cockpit control operation, and then control the cockpit system according to the cockpit control operation.

[0035] Further, in an embodiment of the present invention, gesture recognition is performed on the first image information or the second image information through a pre-trained gesture recognition model, and the gesture recognition model is trained through the following steps:

[0036] Obtain the first sample gesture image of the tester interacting with the cockpit interaction interface in the test cockpit environment, and determine the sample label of the first sample gesture image according to the interaction result;

[0037] Obtain the second sample gesture image of the tester with no operation intention in the test cockpit environment, and determine the sample label of the second sample gesture image as 0;

[0038] Use the first sample gesture image as the positive sample and the second sample gesture image as the negative sample to obtain a training sample set;

[0039] Input the training sample set into a pre-constructed convolutional neural network to obtain a gesture recognition result;

[0040] Determine the loss value according to the gesture recognition result and the sample label;

[0041] Update the parameters of the convolutional neural network according to the loss value to obtain the trained gesture recognition model.

[0042] In a second aspect, an embodiment of the present invention provides an intelligent cockpit interaction system based on holographic projection, including:

[0043] A holographic projection trigger module, configured to respond to a voice trigger instruction / gesture trigger instruction of a target person, obtain the position information and orientation information of the target person, determine an interaction plane according to the position information and the orientation information, and then display a preset cockpit interaction interface on the interaction plane through holographic projection technology;

[0044] An interaction control module, configured to respond to a voice control instruction / gesture control instruction of the target person, identify corresponding interaction information according to the voice control instruction / gesture control instruction, and then adjust the cockpit interaction interface according to the interaction information, or control the cockpit system according to the interaction information.

[0045] In a third aspect, an embodiment of the present invention provides an intelligent cockpit interaction device based on holographic projection, including:

[0046] At least one processor;

[0047] At least one memory for storing at least one program;

[0048] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned intelligent cockpit interaction method based on holographic projection.

[0049] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, in which a program executable by a processor is stored. The program executable by the processor is used to execute the above-mentioned intelligent cockpit interaction method based on holographic projection when executed by the processor.

[0050] The advantages and beneficial effects of the present invention will be partially given in the following description, partially become apparent from the following description, or be understood through the practice of the present invention:

[0051] In an embodiment of the present invention, in response to a voice trigger instruction / gesture trigger instruction of a target person, the position information and orientation information of the target person are acquired, an interaction plane is determined according to the position information and orientation information, and then a preset cockpit interaction interface is displayed on the interaction plane through holographic projection technology. In response to a voice control instruction / gesture control instruction of the target person, the corresponding interaction information is recognized according to the voice control instruction / gesture control instruction, and then the cockpit interaction interface is adjusted according to the interaction information, or the cockpit system is controlled according to the interaction information. In an embodiment of the present invention, the interaction plane is determined according to the position information and orientation information of the target person who triggers the holographic projection, and the cockpit interaction interface is displayed on the interaction plane through holographic projection technology, and then the interaction information is recognized according to the voice control instruction / gesture control instruction of the target person, so that the cockpit interaction interface is adjusted or the cockpit system is controlled according to the interaction information, enabling the passengers in the cockpit to interact with the cockpit system more conveniently, avoiding the driver from operating the central control screen and thus causing the line of sight to deviate from the road surface ahead, and improving driving safety and the riding experience of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following introduces the drawings required to be used in the embodiments of the present invention. It should be understood that the drawings introduced below are only for conveniently and clearly presenting some embodiments of the technical solutions in the present invention, and those skilled in the art can also obtain other drawings based on these drawings without creative efforts.

[0053] Figure 1 It is a flowchart of the steps of an intelligent cockpit interaction method based on holographic projection provided by an embodiment of the present invention;

[0054] Figure 2 It is a structural block diagram of an intelligent cockpit interaction system based on holographic projection provided by an embodiment of the present invention;

[0055] Figure 3 This is a structural block diagram of an intelligent cockpit interaction device based on holographic projection provided by an embodiment of the present invention. Specific embodiments

[0056] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation of the present invention. For the step numbers in the following embodiments, they are only set for the convenience of explanation and illustration, and no limitation is imposed on the order between the steps. The execution order of each step in the embodiments can be adjusted adaptively according to the understanding of those skilled in the art.

[0057] In the description of the present invention, the meaning of "a plurality" is two or more. If the first and second are described, it is only for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features or implicitly indicating the sequence of the indicated technical features. In addition, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which this technology belongs.

[0058] Referring to Figure 1 , an embodiment of the present invention provides an intelligent cockpit interaction method based on holographic projection, which specifically includes the following steps:

[0059] S101. In response to a voice trigger instruction / gesture trigger instruction of a target person, obtain the position information and orientation information of the target person, determine an interaction plane according to the position information and orientation information, and then display a preset cockpit interaction interface on the interaction plane through holographic projection technology;

[0060] S102. In response to a voice control instruction / gesture control instruction of the target person, identify corresponding interaction information according to the voice control instruction / gesture control instruction, and then adjust the cockpit interaction interface according to the interaction information, or control the cockpit system according to the interaction information.

[0061] Specifically, the embodiment of the present invention determines an interaction plane according to the position information and orientation information of the target person who triggers the holographic projection, displays a cockpit interaction interface on this interaction plane through holographic projection technology, and then identifies interaction information according to the voice control instruction / gesture control instruction of the target person, so as to adjust the cockpit interaction interface or control the cockpit system according to this interaction information, enabling the passengers in the cockpit to interact with the cockpit system more conveniently, avoiding the driver from operating the central control screen and thus causing the line of sight to deviate from the road surface ahead, and improving driving safety and the riding experience of users.

[0062] In the embodiments of the present invention, the triggering of the holographic projection can be achieved through voice and gestures. The driver and passengers can choose voice triggering or gesture triggering. The following will separately describe the two triggering methods in detail.

[0063] Further as an optional implementation manner, in response to the voice trigger instruction of the target person, the position information and orientation information of the target person are obtained, which specifically includes:

[0064] Obtain the first voice information and the first image information of each driver and passenger through the voice collection devices and image collection devices set in each cockpit area;

[0065] Perform voice recognition on the first voice information to obtain the first voice instruction, and match the first voice instruction with the preset voice trigger instruction;

[0066] When the matching is successful, determine the corresponding driver and passenger as the target person, and determine the first image information of the target person as the target image information;

[0067] Perform human key point detection on the target image information to obtain multiple human key points of the target person, and then determine the position information and orientation information according to the human key points.

[0068] Specifically, voice collection devices and image collection devices are pre-set in each cockpit area to collect the voice information and image information of the driver and passengers in the corresponding cockpit area; perform voice recognition on the collected voice information to obtain the voice instruction, and then match it with the preset voice trigger instruction; if the matching is successful, the driver and passenger are the target person triggering the holographic projection, and at the same time, determine the image information of the target person as the target image information; perform human key point detection on the target image information to obtain multiple human key points of the target person, such as head key points, torso key points, leg key points, etc., and the position information and orientation information of the target person can be determined based on the human key points.

[0069] Further as an optional implementation manner, in response to the gesture trigger instruction of the target person, the position information and orientation information of the target person are obtained, which specifically includes:

[0070] Obtain the first image information of each driver and passenger through the image collection devices set in each cockpit area;

[0071] Perform gesture recognition on the first image information to obtain the first gesture instruction, and match the first gesture instruction with the preset gesture trigger instruction;

[0072] When the matching is successful, determine the corresponding driver and passenger as the target person, and determine the corresponding first image information as the target image information;

[0073] Perform human key point detection on the target image information to obtain multiple human key points of the target person, and then determine the position information and orientation information based on the human key points.

[0074] Specifically, image acquisition devices are pre-set in each cockpit area to collect the image information of the occupants in the corresponding cockpit area; gesture recognition is performed on the collected image information to obtain gesture commands, and then the gesture commands are matched with the pre-set gesture trigger commands; if the match is successful, the occupant is the target person who triggers the holographic projection. At the same time, the image information of the target person is determined as the target image information; human key point detection is performed on the target image information to obtain multiple human key points of the target person, such as head key points, torso key points, leg key points, etc. Based on the human key points, the position information and orientation information of the target person can be determined.

[0075] Further as an optional implementation manner, an interaction plane is determined according to the position information and orientation information, and then a pre-set cockpit interaction interface is displayed on the interaction plane through holographic projection technology, which specifically includes:

[0076] Determine the target person coordinates according to the position information, and determine the target direction according to the orientation information;

[0077] Obtain the pre-set interaction distance, and determine the interaction plane according to the target person coordinates, target direction and interaction distance, so that the distance from the target person coordinates to the interaction plane is equal to the interaction distance, and the normal vector of the interaction plane is consistent with the target direction;

[0078] Obtain the holographic image of the pre-set cockpit interaction interface, and project the holographic image on the interaction plane through multiple laser light sources arranged in the cockpit area corresponding to the target person.

[0079] Specifically, according to the position information, orientation information, and a preset interaction distance, a holographic image of the cockpit interaction interface is displayed at a position in front of the target person. Holographic projection technology uses the interference principle to record the light wave information of an object. The object to be photographed forms a diffused object light beam under laser irradiation. Another part of the laser serves as a reference light beam and is projected onto the holographic film, where it is superimposed with the object light beam to produce interference. The phase and amplitude of each point on the object light wave are converted into intensity that varies in space, so that all the information of the object light wave is recorded using the contrast and interval between the interference fringes. Then, the diffraction principle is used to reproduce the object light wave information. A hologram is like a complex grating. Under the illumination of coherent laser light, the diffracted light wave of a linearly recorded sinusoidal hologram generally gives two images, namely the original image (also known as the initial image) and the conjugate image. The reproduced image has a strong three-dimensional sense and a real visual effect. Each part of the hologram records the light information of each point on the object. Therefore, in principle, each part of it can reproduce the entire image of the original object. Multiple different images can also be recorded on the same film through multiple exposures and can be displayed separately without interference.

[0080] In the embodiments of the present invention, the content that needs to be displayed by the cockpit system is pre-constructed into a holographic image, and a laser light source is constructed inside the vehicle to project and display the holographic image.

[0081] In some optional embodiments, in combination with a light quantity sensor, the brightness of the holographic image is reduced during the night and increased during the day to ensure a better user experience.

[0082] In the embodiments of the present invention, after the holographic projection of the cockpit interaction interface is triggered, the driver and passengers can also choose voice interaction or gesture interaction. The following will separately elaborate on the two interaction methods in detail.

[0083] Further as an optional implementation manner, in response to the voice control command of the target person, the corresponding interaction information is identified according to the voice control command, and then the cockpit interaction interface is adjusted according to the interaction information, or the cockpit system is controlled according to the interaction information. Specifically, it includes:

[0084] The second voice information of the target person is obtained through a voice acquisition device arranged in the cockpit area corresponding to the target person;

[0085] The second voice information is subjected to voice recognition to obtain a voice control command, and the voice control command is matched with a preset voice interaction database to obtain the corresponding interaction information, and then the interaction type of the interaction information is determined;

[0086] When the interaction type is an interface operation type, the corresponding interaction interface operation is determined, and then the cockpit interaction interface is adjusted according to the interaction interface operation;

[0087] When the interaction type is a cockpit control type, determine the corresponding cockpit control operation, and then control the cockpit system according to the cockpit control operation.

[0088] Specifically, after triggering the holographic projection of the cockpit interaction interface, continue to obtain the second voice information of the target person through the voice collection device set in the cockpit area corresponding to the target person, then perform voice recognition to obtain a voice control command, match the voice control command with the preset voice interaction database to obtain the corresponding interaction information, and at the same time, the interaction type of the interaction information can be determined. For example, if the voice control command is "Open the navigation page", the corresponding interaction information is "Open the navigation page on the cockpit interaction interface", and the corresponding interaction type is the interface operation type. Another example is that if the voice control command is "Turn on voice navigation", the corresponding interaction information is "Turn on the voice navigation function of the cockpit system", and the corresponding interaction type is the cockpit control type. When the interaction type is the interface operation type, determine the corresponding interaction interface operation, and then adjust the cockpit interaction interface according to the interaction interface operation; when the interaction type is the cockpit control type, determine the corresponding cockpit control operation, and then control the cockpit system according to the cockpit control operation.

[0089] Further as an optional implementation manner, in response to the gesture control command of the target person, identify the corresponding interaction information according to the gesture control command, and then adjust the cockpit interaction interface according to the interaction information, or control the cockpit system according to the interaction information, which specifically includes:

[0090] Obtain the second image information of the target person through the image collection device set in the cockpit area corresponding to the target person;

[0091] Perform gesture recognition on the second image information to obtain a gesture control command, and perform human key point detection on the second image information to obtain the positions of multiple hand key points;

[0092] Determine the interface elements at the corresponding positions on the cockpit interaction interface according to the positions of the hand key points, and determine the interaction information and the interaction type of the interaction information according to the interface elements and the gesture control command;

[0093] When the interaction type is the interface operation type, determine the corresponding interaction interface operation, and then adjust the cockpit interaction interface according to the interaction interface operation;

[0094] When the interaction type is the cockpit control type, determine the corresponding cockpit control operation, and then control the cockpit system according to the cockpit control operation.

[0095] Specifically, after triggering the holographic projection of the cockpit interaction interface, the image acquisition device arranged in the cockpit area corresponding to the target person continues to acquire the second image information of the target person, and then gesture recognition is performed to obtain a gesture control instruction (such as drag, pull, click, zoom, long press, etc.). At the same time, human key point detection is performed on the second image information to obtain the position of the hand key points. According to the position of the hand key points, it can be determined that the interface element at the corresponding position on the cockpit interaction interface is the interface element currently interacted with by the target person. According to this interface element and the recognized gesture control instruction, the interaction information and the interaction type can be determined. For example, if the interface element is a navigation APP icon and the gesture control instruction is a single click, the corresponding interaction information is "open the navigation page on the cockpit interaction interface", and the corresponding interaction type is the interface operation type. If the interface element is the border of the navigation page and the gesture control instruction is zoom, the corresponding interaction information is "adjust the size of the navigation page", and the corresponding interaction type is the interface operation type. Another example is that if the interface element is the voice navigation start button on the navigation page and the gesture control instruction is a single click, the corresponding interaction information is "turn on the voice navigation function of the cockpit system", and the corresponding interaction type is the cockpit control type. When the interaction type is the interface operation type, the corresponding cockpit interaction operation is determined, and then the cockpit interaction interface is adjusted according to the cockpit interaction operation. When the interaction type is the cockpit control type, the corresponding cockpit control operation is determined, and then the cockpit system is controlled according to the cockpit control operation.

[0096] Further as an optional implementation manner, gesture recognition is performed on the first image information or the second image information through a pre-trained gesture recognition model. The gesture recognition model is trained through the following steps:

[0097] Obtain the first sample gesture image of the test person interacting with the cockpit interaction interface in the test cockpit environment, and determine the sample label of the first sample gesture image according to the interaction result;

[0098] Obtain the second sample gesture image of the test person with no operation intention in the test cockpit environment, and determine the sample label of the second sample gesture image as 0;

[0099] Take the first sample gesture image as the positive sample and the second sample gesture image as the negative sample to obtain a training sample set;

[0100] Input the training sample set into a pre-constructed convolutional neural network to obtain a gesture recognition result;

[0101] Determine the loss value according to the gesture recognition result and the sample label;

[0102] Update the parameters of the convolutional neural network according to the loss value to obtain a trained gesture recognition model.

[0103] Specifically, the input data of the gesture recognition model includes the first sample gesture images of the tester interacting with the cockpit interaction interface in the test cockpit environment, and the second sample gesture images of the tester having no operation intention in the test cockpit environment. The model architecture can adopt a convolutional neural network, and the model output is the gesture recognition result.

[0104] Obtain the first sample gesture images of the tester interacting with the cockpit interaction interface in the test cockpit environment, and determine the sample labels of the first sample gesture images according to the interaction results. For example, if the interaction result is "open the navigation page on the cockpit interaction interface", the corresponding sample label is click; another example, if the interaction result is "adjust the size of the navigation page", the corresponding sample label is zoom. Obtain the second sample gesture images of the tester having no operation intention in the test cockpit environment, and determine the sample label of the second sample gesture images as 0. It should be noted that the gestures corresponding to the second sample gesture images will not trigger the holographic projection of the cockpit interaction interface, nor will they interact with the cockpit interaction interface.

[0105] Taking the first sample gesture image as a positive sample and the second sample gesture image as a negative sample, input them into a convolutional neural network for training. After inputting the training samples into the initialized convolutional neural network, the recognition result output by the model, that is, the gesture recognition result, can be obtained. The accuracy of the model recognition can be evaluated according to the gesture recognition result and the aforementioned sample labels, so as to update the parameters of the model. For a gesture recognition model, the accuracy of the model recognition result can be measured by a loss function. The loss function is defined on a single training data and is used to measure the prediction error of a training data. Specifically, the loss value of the training data is determined by the label of the single training data and the prediction result of the model for this training data. During actual training, there are many training data in a training data set. Therefore, generally, a cost function is used to measure the overall error of the training data set. The cost function is defined on the entire training data set and is used to calculate the average value of the prediction errors of all training data, which can better measure the prediction effect of the model. For a general machine learning model, based on the aforementioned cost function, plus a regularization term that measures the complexity of the model, it can be used as the objective function for training. Based on this objective function, the loss value of the entire training data set can be obtained. There are many types of commonly used loss functions. For example, the 0-1 loss function, square loss function, absolute loss function, logarithmic loss function, cross-entropy loss function, etc. can all be used as the loss function of the machine learning model, which will not be elaborated one by one here. In the embodiments of the present invention, any one of the loss functions can be selected to determine the loss value of training. Based on the loss value of training, the backpropagation algorithm is used to update the parameters of the model, and after several iterations, a trained gesture recognition model can be obtained. Specifically, the number of iterations can be preset in advance, or it is considered that the training is completed when the accuracy requirement is met in the test set.

[0106] Based on the trained gesture recognition model, the recognition of gesture trigger commands and gesture control commands can be realized.

[0107] The method steps of the embodiments of the present invention are described above. It can be understood that the embodiments of the present invention determine the interaction plane according to the position information and orientation information of the target person who triggers the holographic projection, display the cockpit interaction interface on this interaction plane through holographic projection technology, and then recognize the interaction information according to the voice control command / gesture control command of the target person, so as to adjust the cockpit interaction interface or control the cockpit system according to this interaction information, enabling the passengers in the cockpit to interact with the cockpit system more conveniently, avoiding the driver from operating the central control screen and thus causing the line of sight to deviate from the road surface ahead, and improving driving safety and the riding experience of users.

[0108] Referring to Figure 2 , the embodiments of the present invention provide an intelligent cockpit interaction system based on holographic projection, including:

[0109] A holographic projection trigger module, which is configured to respond to a voice trigger instruction / gesture trigger instruction of a target person, obtain the position information and orientation information of the target person, determine an interaction plane according to the position information and orientation information, and then display a preset cockpit interaction interface on the interaction plane through holographic projection technology;

[0110] An interaction control module, which is configured to respond to a voice control instruction / gesture control instruction of a target person, identify corresponding interaction information according to the voice control instruction / gesture control instruction, and then adjust the cockpit interaction interface according to the interaction information, or control the cockpit system according to the interaction information.

[0111] The content in the above method embodiments is applicable to the system embodiments of the present invention. The functions specifically implemented by the system embodiments of the present invention are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0112] Referring to Figure 3 , an intelligent cockpit interaction device based on holographic projection is provided in an embodiment of the present invention, including:

[0113] At least one processor;

[0114] At least one memory for storing at least one program;

[0115] When the above at least one program is executed by the above at least one processor, the above at least one processor implements the above-mentioned intelligent cockpit interaction method based on holographic projection.

[0116] The content in the above method embodiments is applicable to the device embodiments of the present invention. The functions specifically implemented by the device embodiments of the present invention are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0117] An embodiment of the present invention further provides a computer-readable storage medium, in which a processor-executable program is stored. The processor-executable program is used to execute the above-mentioned intelligent cockpit interaction method based on holographic projection when executed by a processor.

[0118] A computer-readable storage medium according to an embodiment of the present invention can execute an intelligent cockpit interaction method based on holographic projection provided by an embodiment of the method of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0119] An embodiment of the present invention also discloses a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figure 1 the method shown.

[0120] In some alternative embodiments, the functions / operations mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two consecutive blocks shown may actually be executed substantially simultaneously or the above-mentioned blocks can sometimes be executed in the reverse order. In addition, the embodiments presented and described in the flowcharts of the present invention are provided by way of example for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated, in which the order of various operations is changed and the sub-operations described as part of a larger operation are executed independently.

[0121] In addition, although the present invention has been described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the above functions and / or features may be integrated in a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It can also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More precisely, considering the attributes, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the modules will be understood within the ordinary skills of an engineer. Therefore, those skilled in the art can implement the present invention as set forth in the claims without undue experimentation. It can also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0122] If the above functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the above methods in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0123] The logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in combination with an instruction execution system, apparatus, or device.

[0124] More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection part (electronic device) having one or more wirings, a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, a computer-readable medium can even be paper or other suitable media on which the above program can be printed, because the above program can be obtained electronically, for example, by optically scanning the paper or other media, then editing, interpreting, or processing it in other suitable ways as necessary, and then storing it in a computer memory.

[0125] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), and the like.

[0126] In the above description of this specification, the descriptions referring to the terms "one embodiment / example", "another embodiment / example", or "certain embodiments / examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0127] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the claims and their equivalents.

[0128] The above has specifically described the preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art can also make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A smart cockpit interaction method based on holographic projection, characterized in that: The following steps are involved: In response to a voice trigger command / gesture trigger command of a target person, the position information and orientation information of the target person are acquired, an interaction plane is determined according to the position information and the orientation information, and a preset cockpit interaction interface is displayed on the interaction plane through holographic projection technology; In response to the voice control command / gesture control command of the target person, corresponding interaction information is identified according to the voice control command / gesture control command, and then the cockpit interaction interface is adjusted according to the interaction information, or the cockpit system is controlled according to the interaction information.

2. The intelligent cockpit interaction method based on holographic projection according to claim 1 is characterized in that: In response to the voice trigger command of the target person, the location information and orientation information of the target person are obtained, which specifically includes: Acquiring first voice information and first image information of each driver and passenger through a voice acquisition device and an image acquisition device arranged in each cockpit area; Performing voice recognition on the first voice information to obtain a first voice instruction, and matching the first voice instruction with a preset voice trigger instruction; When the match is successful, the corresponding driver and passenger is determined to be the target person, and the first image information of the target person is determined to be the target image information; The target image information is subjected to human body key point detection to obtain a plurality of human body key points of the target person, and then the position information and the orientation information are determined according to the human body key points.

3. The intelligent cockpit interaction method based on holographic projection according to claim 1 is characterized in that: In response to a gesture trigger instruction of a target person, the location information and orientation information of the target person are obtained, which specifically includes: Acquiring first image information of each driver and passenger by means of image acquisition devices arranged in each cockpit area; Performing gesture recognition on the first image information to obtain a first gesture instruction, and matching the first gesture instruction with a preset gesture trigger instruction; When the match is successful, the corresponding driver and passenger is determined to be the target person, and the corresponding first image information is determined to be the target image information; The target image information is subjected to human body key point detection to obtain a plurality of human body key points of the target person, and then the position information and the orientation information are determined according to the human body key points.

4. The intelligent cockpit interaction method based on holographic projection according to claim 1 is characterized in that: The step of determining an interaction plane according to the position information and the orientation information, and then displaying a preset cockpit interaction interface on the interaction plane by using holographic projection technology, specifically includes: Determine the coordinates of the target person according to the position information, and determine the target direction according to the orientation information; Obtaining a preset interaction distance, and determining the interaction plane according to the target person coordinates, the target direction, and the interaction distance, so that the distance from the target person coordinates to the interaction plane is equal to the interaction distance, and the normal vector of the interaction plane is consistent with the target direction; A preset holographic image of the cockpit interaction interface is obtained, and the holographic image is projected onto the interaction plane through a plurality of laser light sources arranged in the cockpit area corresponding to the target person.

5. The intelligent cockpit interaction method based on holographic projection according to claim 1 is characterized in that: In response to the voice control command of the target person, corresponding interaction information is identified according to the voice control command, and then the cockpit interaction interface is adjusted according to the interaction information, or the cockpit system is controlled according to the interaction information, which specifically includes: Acquiring second voice information of the target person through a voice collection device disposed in a cockpit area corresponding to the target person; Performing voice recognition on the second voice information to obtain the voice control instruction, and matching the voice control instruction with a preset voice interaction database to obtain the corresponding interaction information, and then determining the interaction type of the interaction information; When the interaction type is an interface operation type, determining a corresponding interaction interface operation, and then adjusting the cockpit interaction interface according to the interaction interface operation; When the interaction type is a cockpit control type, a corresponding cockpit control operation is determined, and then the cockpit system is controlled according to the cockpit control operation.

6. The intelligent cockpit interaction method based on holographic projection according to claim 1 is characterized in that: In response to the gesture control instruction of the target person, corresponding interaction information is identified according to the gesture control instruction, and then the cockpit interaction interface is adjusted according to the interaction information, or the cockpit system is controlled according to the interaction information, which specifically includes: Acquiring second image information of the target person by using an image acquisition device disposed in a cockpit area corresponding to the target person; Performing gesture recognition on the second image information to obtain the gesture control instruction, and performing human body key point detection on the second image information to obtain multiple hand key point positions; Determining an interface element at a corresponding position on the cockpit interaction interface according to the position of the key point of the hand, and determining the interaction information and the interaction type of the interaction information according to the interface element and the gesture control instruction; When the interaction type is an interface operation type, determining a corresponding interaction interface operation, and then adjusting the cockpit interaction interface according to the interaction interface operation; When the interaction type is a cockpit control type, a corresponding cockpit control operation is determined, and then the cockpit system is controlled according to the cockpit control operation.

7. The intelligent cockpit interaction method based on holographic projection according to claim 3 or 6, characterized in that: Performing gesture recognition on the first image information or the second image information by using a pre-trained gesture recognition model, wherein the gesture recognition model is trained by the following steps: Acquire a first sample gesture image of a tester interacting with a cockpit interaction interface in a test cockpit environment, and determine a sample label of the first sample gesture image according to an interaction result; Acquire a second sample gesture image of the tester without operation intention in the test cockpit environment, and determine that the sample label of the second sample gesture image is 0; Taking the first sample gesture image as a positive sample and the second sample gesture image as a negative sample to obtain a training sample set; Inputting the training sample set into a pre-built convolutional neural network to obtain a gesture recognition result; Determine a loss value according to the gesture recognition result and the sample label; The parameters of the convolutional neural network are updated according to the loss value to obtain the trained gesture recognition model.

8. An intelligent cockpit interaction system based on holographic projection, characterized in that: include: A holographic projection trigger module is used to respond to a voice trigger command / gesture trigger command of a target person, obtain the position information and orientation information of the target person, determine an interaction plane according to the position information and the orientation information, and then display a preset cockpit interaction interface on the interaction plane through holographic projection technology; An interactive control module is used to respond to the voice control instructions / gesture control instructions of the target person, identify corresponding interactive information according to the voice control instructions / gesture control instructions, and then adjust the cockpit interactive interface according to the interactive information, or control the cockpit system according to the interactive information.

9. An intelligent cockpit interaction device based on holographic projection, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the smart cockpit interaction method based on holographic projection as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to execute the smart cockpit interaction method based on holographic projection as described in any one of claims 1 to 7 when executed by the processor.

Images