Driving operation system, method, device, vehicle, storage medium and program product

Abstract

The invention discloses a driving operation system, method and equipment, a vehicle, a storage medium and a program product, relates to the technical field of automobiles, and is used for realizing intelligent interaction between a user and a holographic image. The driving operation system comprises an acquisition device, a control device and a holographic projection device, the holographic projection device is used for forming a holographic image in a preset display space; the acquisition device is used for acquiring interaction information between a user and the holographic image; the control device is configured to identify an instruction of the user based on the interaction information and execute the instruction of the user.

Description

Technical Field

[0001] The present invention relates to the field of automobile technology, and in particular to a driving operation system, method, device, vehicle, storage medium and program product. Background Art

[0002] With the rapid development of automobile technology, the application of holographic projection technology in vehicles has made progress. At present, 3D holographic projection intelligent robots can be realized in the car through holographic projection technology to interact and entertain users and enhance the user's driving experience. However, the 3D holographic projection intelligent robots currently realized in the car have limited presentation effects and can only provide some relaxation and entertainment companionship, and the sense of interaction is poor or cannot be realized. Therefore, there is still a lot of room for improvement in the application of holographic projection technology in the automotive field. Summary of the invention

[0003] The object of the present invention is to provide a driving operation system, method, device, vehicle, storage medium and program product for realizing intelligent interaction between a user and a holographic image.

[0004] In order to achieve the above object, the present invention adopts the following technical scheme:

[0005] In a first aspect, the present invention provides a driving operating system, comprising: an acquisition device, a control device and a holographic projection device; the holographic projection device is used to form a holographic image in a preset display space; the acquisition device is used to acquire interaction information between a user and the holographic image; the control device is configured to: identify the user's instructions based on the interaction information, and execute the user's instructions.

[0006] It is understandable that the driving operating system provided by the embodiment of the present application forms a holographic image in a preset display space, which can reduce the occlusion of the user's line of sight by the holographic image and improve the user's driving safety. In addition, the present application realizes intelligent interaction between the user and the holographic image by collecting the interaction information between the user and the holographic image, identifying the user's instructions based on the interaction information, and executing the user's instructions, thereby improving the user's experience.

[0007] In some embodiments, when the user's instruction is an imaging instruction, the control device is configured to execute the user's instruction, including: controlling the holographic projection device to project a holographic image corresponding to the imaging instruction in a preset display space.

[0008] In some embodiments, when the user's instruction is an operation instruction, the control device is configured to execute the user's instruction, including: executing the operation corresponding to the operation instruction; or sending the operation instruction to the corresponding operation unit.

[0009] In some embodiments, the interaction information includes at least one of the following: sound information, action information, facial expression information, and eye contact information.

[0010] In some embodiments, the action information is a hand image including the user's hand actions; when the interaction information includes action information, the control device is configured to identify the user's instructions based on the interaction information, including: processing the hand image to extract the user's hand area image; extracting the user's hand contour and the position information of each fingertip of the hand from the user's hand area image; determining the user's hand action based on the user's hand contour and the position information of each fingertip of the hand; determining the user's instructions based on the hand action and a first preset instruction library; the first preset instruction library is used to reflect the instructions corresponding to multiple preset hand actions.

[0011] In some embodiments, the action information is a hand image including the user's hand movements; when the interaction information includes action information, the control device is configured to identify the user's instructions based on the interaction information, including: processing the hand image to extract the user's hand area image; extracting the user's hand contour and the position information of each fingertip of the hand from the user's hand area image; if the distance between one of the user's fingertips and the virtual key fingertip is less than or equal to a preset distance, determining that the user's instruction is an instruction corresponding to the virtual key; wherein the virtual key is a virtual key in the holographic image displayed in a preset display space.

[0012] In some embodiments, the facial expression information includes a facial image of a user; when the interaction information includes facial expression information, the control device is configured to identify the user's instructions based on the interaction information, including: identifying the user's facial expression based on the facial image; determining the user's instructions based on the user's facial expression and a second preset instruction library; the second preset instruction library is used to reflect instructions corresponding to multiple preset facial expressions.

[0013] In some embodiments, the eye information includes a head image of the user and an eye image of the user; when the interaction information includes eye information, the control device is configured to identify the user's instructions based on the interaction information, including: extracting the user's head coordinates from the head image; determining the user's line of sight direction based on the user's head coordinates and the user's eye image; determining the target object that the user is looking at based on the user's line of sight direction; the target object is an object in the holographic image displayed in a preset display space; and determining that the user's instructions are instructions corresponding to the target object.

[0014] In some embodiments, the holographic projection device includes a first holographic projection device and a second holographic projection device; wherein the first holographic projection device is arranged in a main cockpit; the second holographic projection device is arranged in a co-pilot cockpit; the preset display space includes a first preset display space located in the main cockpit and a second preset display space located in the co-pilot cockpit; the first holographic projection device is used to form a first holographic image in the first preset display space; the second holographic projection device is used to form a second holographic image in the second preset display space.

[0015] In some embodiments, the acquisition device includes a first acquisition device and a second acquisition device; the first acquisition device is arranged in the main cockpit; the second acquisition device is arranged in the co-pilot cockpit; the first acquisition device is used to collect the interaction information between the main driver user and the first holographic image; the second acquisition device is used to collect the interaction information between the co-pilot user and the second holographic image.

[0016] In some embodiments, the control device is further configured to: when it is detected that the vehicle is in an abnormal state, control the holographic projection device to project a holographic image including warning information in a preset display space.

[0017] In some embodiments, the acquisition device is also used to acquire the user's status image; the control device is also configured to: analyze the user's status based on the user's status image; and when the user is in a preset state, control the holographic projection device to project a preset holographic warning image in a preset display space.

[0018] In a second aspect, the present invention provides a driving operation method, comprising: receiving interaction information between a user and a holographic image; the holographic image is formed by a holographic projection device in a preset display space; based on the interaction information, the user's instructions are identified and the user's instructions are executed.

[0019] It is understandable that the driving operation method provided in the embodiment of the present application forms a holographic image in a preset display space, which can reduce the occlusion of the user's line of sight by the holographic image and improve the user's driving safety. In addition, the present application realizes intelligent interaction between the user and the holographic image by receiving the interaction information between the user and the holographic image, identifying the user's instructions based on the interaction information, and executing the user's instructions, thereby improving the user's experience.

[0020] In some embodiments, when the user's instruction is an operation instruction, executing the user's instruction includes: executing the operation corresponding to the operation instruction; or sending the operation instruction to a corresponding operation unit.

[0021] In some embodiments, the interaction information includes at least one of the following: sound information, action information, facial expression information, and eye contact information.

[0022] In some embodiments, the action information is a hand image including the user's hand actions; when the interaction information includes action information, identifying the user's instructions based on the interaction information includes: processing the hand image to extract the user's hand area image; extracting the user's hand contour and the position information of each fingertip of the hand from the user's hand area image; determining the user's hand actions based on the user's hand contour and the position information of each fingertip of the hand; determining the user's instructions based on the hand actions and a first preset instruction library; the first preset instruction library is used to reflect the instructions corresponding to multiple preset hand actions.

[0023] In some embodiments, the action information is a hand image including the user's hand actions; when the interaction information includes action information, identifying the user's instructions based on the interaction information includes: processing the hand image to extract the user's hand area image; extracting the user's hand contour and the position information of each fingertip of the hand from the user's hand area image; if the distance between one of the user's fingertips and the virtual key fingertip is less than or equal to a preset distance, determining that the user's instruction is an instruction corresponding to the virtual key; wherein the virtual key is a virtual key in the holographic image displayed in a preset display space.

[0024] In some embodiments, the facial expression information includes a facial image of a user; when the interaction information includes facial expression information, identifying the user's instructions based on the interaction information includes: identifying the user's facial expression based on the facial image; determining the user's instructions based on the user's facial expression and a second preset instruction library; the second preset instruction library is used to reflect instructions corresponding to multiple preset facial expressions.

[0025] In some embodiments, the eye information includes a head image of the user and an eye image of the user; when the interaction information includes eye information, identifying the user's instructions based on the interaction information includes: extracting the user's head coordinates from the head image; determining the user's line of sight direction based on the user's head coordinates and the user's eye image; determining a target object that the user is looking at based on the user's line of sight direction; the target object is an object in a holographic image displayed in a preset display space; and determining that the user's instructions are instructions corresponding to the target object.

[0026] In some embodiments, the holographic projection device includes a first holographic projection device and a second holographic projection device; wherein the first holographic projection device is arranged in a main cockpit; the second holographic projection device is arranged in a co-pilot cockpit; the preset display space includes a first preset display space located in the main cockpit and a second preset display space located in the co-pilot cockpit; the first holographic projection device is used to form a first holographic image in the first preset display space; the second holographic projection device is used to form a second holographic image in the second preset display space; the interaction information between the user and the holographic image includes: interaction information between the main driver user and the first holographic image, interaction information between the co-pilot user and the second holographic image.

[0027] In some embodiments, the acquisition device includes a first acquisition device and a second acquisition device; the first acquisition device is arranged in the main cockpit; the second acquisition device is arranged in the co-pilot cockpit; the interaction information between the main driver user and the first holographic image is collected by the first acquisition device; the interaction information between the co-pilot user and the second holographic image is collected by the second acquisition device.

[0028] In some embodiments, the method further includes: when it is detected that the vehicle is in an abnormal state, controlling the holographic projection device to project a holographic image including warning information in a preset display space.

[0029] In some embodiments, the method further includes: receiving a status image of a user; analyzing the status of the user based on the status image of the user; and when the user is in a preset status, controlling the holographic projection device to project a preset holographic warning image in a preset display space.

[0030] In a third aspect, the present application provides an electronic device, comprising: a processor and a memory; the memory stores instructions executable by the processor; when the processor is configured to execute the instructions, the electronic device implements the method of the second aspect above.

[0031] In a fourth aspect, the present application provides a vehicle, which includes the driving operating system of the first aspect, or the electronic device of the third aspect.

[0032] In a fifth aspect, the present application provides a computer-readable storage medium, which includes: computer software instructions; when the computer software instructions are executed in an electronic device, the electronic device implements the method of the second aspect above.

[0033] In a sixth aspect, the present application provides a computer program product, which includes a computer program; when the computer program runs in an electronic device, the electronic device implements the method of the second aspect above.

[0034] The beneficial effects of the third to sixth aspects mentioned above refer to the corresponding description of the first or second aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 A schematic diagram of the structure of a driving operating system provided in this application;

[0037] Figure 2 A schematic diagram of the structure of another driving operating system provided for this application;

[0038] Figure 3 A schematic diagram of the structure of another driving operating system provided by this application;

[0039] Figure 4 A schematic diagram of the operating principle of a driving operating system provided in this application;

[0040] Figure 5 A flowchart of a driving operation method provided in this application;

[0041] Figure 6 A schematic diagram of the components of a driving operation device provided in this application;

[0042] Figure 7 A schematic diagram of the structure of the equipment involved in the driving operation method provided in this application.

[0043] Reference numerals: acquisition device 100 , first acquisition device 110 , second acquisition device 120 , control device 200 , holographic projection device 300 , first holographic projection device 310 , second holographic projection device 320 . DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] In the description of the present invention, it should be understood that the terms "upper", "lower", "left", "right", "front", "back", "inside", "outside" and the like indicate directions or positional relationships based on the directions or relative positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention. Unless otherwise specified, the above-mentioned directional description can be flexibly set in the process of actual application under the condition that the relative positional relationship shown in the accompanying drawings is satisfied.

[0046] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0047] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection. It can be directly connected, or indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0048] In the embodiments of the present invention, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, article or device including the element.

[0049] In the embodiments of the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0050] In the description of this specification, specific features, structures, materials or characteristics may be combined in an appropriate manner in any one or more embodiments or examples.

[0051] The application of holographic projection technology in vehicles has made progress. At present, 3D holographic projection intelligent robots can be realized in the car through holographic projection technology to interact with users and enhance the user's driving experience. The intelligent robot system realizes the intelligent control, entertainment interaction and information exchange of the car through the intelligent robot. The intelligent robot has complex environmental perception, intelligent decision-making and collaborative control functions. The intelligent robot system includes: vehicle information system, network communication processing unit, image processing unit and holographic image display unit, so that the intelligent robot can provide emotional companionship and interactive games during driving, and finally make the intelligent robot become a virtual life in the car, improving the overall experience of intelligent driving.

[0052] The above method uses 3D holographic projection intelligent robots with limited presentation effects, and can only provide some relaxing and entertaining companionship, and the sense of interaction is poor or cannot be achieved. Therefore, there is still much room for improvement in the application of holographic projection technology in the automotive field.

[0053] In response to the above technical problems, the present application provides a driving operation system, including: a collection device, a control device and a holographic projection device; the holographic projection device is used to form a holographic image in a preset display space, which can reduce the occlusion of the user's line of sight by the holographic image and improve the user's driving safety. The collection device is used to collect the interaction information between the user and the holographic image to realize the interaction between the user and the holographic image. At the same time, the control device is configured to: identify the user's instructions based on the interaction information and execute the user's instructions, thereby realizing the intelligent interaction between the user and the holographic image and improving the technological sense of the driving operation and the user's experience.

[0054] The driving operating system provided in the embodiment of the present application is introduced below in conjunction with the drawings in the specification.

[0055] See also Figure 1 , is a schematic diagram of the structure of a driving operating system provided in an embodiment of the present application. Figure 1 As shown, the driving operating system includes: a collection device 100, a control device 200 and a holographic projection device 300.

[0056] The acquisition device 100 , the control device 200 and the holographic projection device 300 are described below respectively.

[0057] In some embodiments, the holographic projection device 300 is used to form a holographic image in a preset display space.

[0058] Exemplarily, the preset display space is a pre-set three-dimensional area for displaying holographic images. The size, shape and position of the preset display space can be set and adjusted according to actual needs and are not limited here.

[0059] For example, a holographic image can form a 45° angle between a thin film transistor (TFT) screen and one or more coated reflective surfaces of a display triangular pyramid, and be emitted into the human eye through the principle of optical reflection, making people feel that the holographic image is displayed on a vertical surface in the triangular pyramid.

[0060] Exemplarily, the holographic projection device 300 includes a TFT screen and a triangular pyramid reflective surface treated with a coating.

[0061] Exemplarily, the TFT screen serves as an image source and is responsible for displaying various images or video contents.

[0062] Exemplarily, one or more side surfaces of the triangular pyramid are coated to form a reflective surface with high reflectivity, and at least one reflective surface forms an angle of 45° with the TFT screen so that the image or video on the TFT screen can be emitted into the human eye at a specific angle through the principle of optical reflection to form a holographic image.

[0063] It should be noted that the holographic projection device 300 can also form a holographic image through digital holographic display or phantom imaging and other technologies, which can be determined according to actual conditions and are not limited here.

[0064] Exemplarily, a holographic image is formed in a preset display space to display various image states. For example, the holographic image may be a 3D model of a vehicle to display the operating status of the vehicle; or, the holographic image may be an operation page including virtual buttons, wherein the virtual buttons may be associated with physical buttons of the vehicle; or, the holographic image may be a virtual human image.

[0065] In some embodiments, the collection device 100 is used to collect interaction information between the user and the holographic image.

[0066] In some embodiments, the interaction information includes at least one of the following: sound information, action information, facial expression information, and eye contact information.

[0067] In some embodiments, the collection device 100 may include a sound collection device and / or an optical collection device.

[0068] Exemplarily, the collection device 100 may be located on the A-pillar next to the front windshield.

[0069] In some embodiments, the sound collection device is used to collect the user's sound information and send it to the control device 200.

[0070] Exemplarily, the sound collection device may include: an audio input device and a processor.

[0071] Exemplarily, the audio input device is used to collect the user's voice information in real time and send it to the processor. For example, the audio input device can be a microphone, a voice acquisition device, or other audio input devices.

[0072] Exemplarily, the processor is used to convert the user's voice information into a digital signal and send it to the control device 200. For example, the processor can be a digital signal processor, an audio processor, a digital signal controller, etc.

[0073] In some embodiments, the optical collection device is used to collect the user's motion information, facial expression information, and eye information.

[0074] Exemplarily, the optical acquisition device may include: at least one optical camera and a processor.

[0075] Exemplarily, at least one optical camera is used to capture hand images, facial images, and eye images of the user and send them to the processor.

[0076] Exemplarily, the processor may convert the image data into a digital signal and send it to the control device 200. For example, a hand image, a facial image, or an eye image may be converted into a digital signal and sent to the control device 200.

[0077] In some embodiments, the control device 200 is configured to: identify the user's instructions based on the interaction information, and execute the user's instructions.

[0078] Exemplarily, the correspondence between the interaction information and the user's instruction can be configured according to the user's needs.

[0079] It is understandable that the driving operation system provided by the embodiment of the present application forms a holographic image in a preset display space, which can reduce the occlusion of the user's line of sight by the holographic image and improve the user's driving safety. In addition, the present application realizes intelligent interaction between the user and the holographic image by collecting the interaction information between the user and the holographic image, identifying the user's instructions based on the interaction information, and executing the user's instructions, thereby improving the technological sense of driving operation and the user's experience.

[0080] In some embodiments, when the user's instruction is an imaging instruction, the control device 200 is configured to execute the user's instruction, including: controlling the holographic projection device 300 to project a holographic image corresponding to the imaging instruction in a preset display space.

[0081] Exemplarily, the holographic image is determined by a preset correspondence relationship, which is a correspondence relationship between imaging instructions and holographic images. Users can set and add it according to actual needs.

[0082] For example, it is assumed that the auxiliary functions of the vehicle include a scenery appreciation function, wherein the scenery appreciation function can provide the user with holographic image data and voice introduction of the scenic area. When the vehicle turns on the scenery appreciation function, if an imaging instruction issued by the user through language or action is received, which is used to instruct to project the holographic image data of the selected scenic area in the preset display space, then in response to the user's imaging instruction, the control device 200 will immediately control the holographic projection device 300, and the holographic projection device 300 will project the scenery holographic image data of the selected scenic area in the preset display space according to the user's imaging instruction, and provide a language introduction.

[0083] Exemplarily, the holographic image data may be image data pre-configured in the control device.

[0084] Exemplarily, the control device 200 is connected to the Internet, and the holographic image data can also be obtained from the Internet.

[0085] Optionally, the control device 200 is connected to the Internet, and can respond to the user's selection of a scenic spot and join a virtual interactive platform corresponding to the scenic spot to increase the flexibility of driving operations. The virtual interactive platform can be in the form of an online community or chat group, which gathers all users who choose the same scenic spot and can provide users with functions such as building a car friend circle mutual assistance tour and automatic team formation.

[0086] In some embodiments, in combination Figure 1 The driving operating system shown in the figure illustrates the imaging process of the full-information image. Figure 2 As shown, the acquisition device 100 acquires the interaction information between the user and the holographic image and sends it to the control device 200. Then, the control device 200 identifies the user's instruction based on the interaction information between the user and the holographic image. In the case where the user's instruction is an imaging instruction, the control device 200 controls the holographic projection device 300 to generate a corresponding holographic image in a preset display space based on the user's imaging instruction.

[0087] It can be understood that automatically projecting the corresponding holographic image based on the user's imaging instructions provides users with a more convenient and intuitive operating experience, enhancing the user's driving experience and the sense of technology in driving operations.

[0088] In some embodiments, when the user's instruction is an operation instruction, the control device 200 is configured to execute the user's instruction, including: executing the operation corresponding to the operation instruction; or sending the operation instruction to the corresponding operation unit.

[0089] Exemplarily, when the user's operation instruction is to increase the current navigation volume, in response to the user's operation instruction, the control device 200 controls the current navigation volume to increase to a corresponding volume.

[0090] Exemplarily, when the user's operation instruction is to lower the air-conditioning temperature, the control device 200 sends the user's operation instruction to the air-conditioning control unit. In response to the user's operation instruction, the air-conditioning control unit turns on the air-conditioning and adjusts it to the corresponding temperature.

[0091] It can be understood that executing operating instructions through the control device 200 simplifies the user's operating process and improves the user's operating experience. At the same time, the operations corresponding to the user's operating instructions can be set by the user, which improves the flexibility of driving operations.

[0092] Below, taking different types of interaction information as an example, a specific implementation manner of the control device 200 identifying the user's instruction based on the interaction information is described.

[0093] (i) Interaction information includes action information.

[0094] In some embodiments, the action information is a hand image including the user's hand actions; when the interaction information includes action information, the control device 200 is configured to identify the user's instructions based on the interaction information, including: processing the hand image to extract the user's hand area image; extracting the user's hand contour and the position information of each fingertip of the hand from the user's hand area image; determining the user's hand action based on the user's hand contour and the position information of each fingertip of the hand; determining the user's instructions based on the hand action and the first preset instruction library.

[0095] The first preset instruction library is used to reflect instructions corresponding to a variety of preset hand movements.

[0096] Exemplarily, the first preset instruction library includes instructions as shown in the following Table 1:

[0097] Table 1 The first preset instruction library

[0098] Preset hand movements instruction OK Play Song Palm up wave Volume Up Palm down Volume down Swipe left or right Switch songs make a fist Pause song ... ...

[0099] In some embodiments, the control device 200 is configured to: process the hand image to extract the user's hand area image, including: preprocessing the hand image to obtain a preprocessed image; extracting the user's hand area image based on the preprocessed hand image.

[0100] In some embodiments, preprocessing of the hand image includes: denoising, grayscale processing, binarization processing, etc., to improve the accuracy of user instructions.

[0101] Exemplarily, a Gaussian filter, a mean filter, a median filter or other algorithms may be used for denoising.

[0102] It should be noted that the denoising process can be implemented by configuring a corresponding filtering algorithm in the control device, which will not be elaborated here.

[0103] Exemplarily, grayscale processing is the process of converting a color hand image into a grayscale hand image, which can be achieved by averaging the grayscale values ​​of the three color channels of red, green, and blue of the hand image or performing weighted averaging according to a certain weight. In this way, each pixel has only one grayscale value, thereby realizing the grayscale of the hand image. The grayscaled hand image is clearer, which is conducive to the extraction of the hand area image.

[0104] Exemplarily, binarization is the process of converting a grayscale hand image into a binary image, that is, dividing the pixels in the hand image into two categories: foreground and background. The global threshold method, local threshold method and dynamic threshold method can be used. The global threshold method is to select a unified threshold for the entire hand image for binarization; the local threshold method is to divide the hand image into multiple small areas, and select a threshold for each area for binarization; the dynamic threshold method is to determine the threshold of each pixel based on the grayscale value of each pixel and its surrounding pixels.

[0105] It should be noted that the user can select a suitable binarization method according to the characteristics of the hand image and the background conditions, and no limitation is made here.

[0106] In some embodiments, a skin color detection or edge detection algorithm is used to extract the user's hand area image from the preprocessed hand image.

[0107] Exemplarily, skin color detection is a method for extracting the hand region based on color space. Since the color range of the user's skin color has certain stability and uniqueness in a specific color space, the hand region can be extracted by color space conversion and threshold segmentation.

[0108] Exemplarily, edge detection utilizes edge information in an image and extracts the contour of the hand region through an edge algorithm.

[0109] In some embodiments, the control device 200 is configured to: extract the user's hand contour and the position information of each fingertip of the hand from the user's hand area image, including: using a contour detection algorithm to extract the user's hand contour; determining the position information of each fingertip of the hand based on the distance from a point on the user's hand contour to the center of mass of the hand area.

[0110] For example, the contour detection algorithm refers to the process of extracting the contour of the target in an image containing the target and the background, ignoring the influence of the texture and noise interference inside the background and the target. The contour detection algorithm can detect edges based on the gradient changes in the image, thereby determining the position information of the hand contour.

[0111] Exemplarily, the centroid of the hand region is the average position of all points of the hand contour, which can be determined by calculating the weighted sum of the contour points.

[0112] Illustratively, each fingertip of the hand is one of the points on the hand contour that is farthest from the center of mass.

[0113] In some embodiments, the control device 200 is configured to: extract hand features for hand motion recognition based on the user's hand contour and position information of each fingertip of the hand; and determine the user's hand motion based on the user's hand features.

[0114] Among them, hand features include: relative position between fingers, degree of finger bending, palm orientation, etc.

[0115] Exemplarily, a trained model is placed in the control device 200 , and the model is trained to determine the user's hand movements based on the user's hand features.

[0116] It is understandable that users can interact with holographic images simply through hand movements, making the driving operation interface more natural and intuitive, while improving the user's driving operation experience. The first preset instruction library can be updated and expanded according to user needs, thereby improving the flexibility of driving operations.

[0117] (ii) Interaction information includes action information.

[0118] In some embodiments, the action information is a hand image including the user's hand movements; when the interaction information includes action information, the control device 200 is configured to identify the user's instructions based on the interaction information, including: processing the hand image to extract the user's hand area image; extracting the user's hand contour and the position information of each fingertip of the hand from the user's hand area image; if the distance between one of the user's fingertips and the virtual key is less than or equal to the preset distance, determining that the user's instruction is the instruction corresponding to the virtual key.

[0119] The virtual button is a virtual button in the holographic image displayed in the preset display space.

[0120] Exemplarily, a three-dimensional coordinate system is established based on the vehicle's cockpit, the origin of the three-dimensional coordinate system can be set by the user, the position information of the preset display space can be determined based on the coordinate information of the preset display space, and the position information of the holographic image can be determined based on the coordinate information of the holographic image.

[0121] Exemplarily, the instruction corresponding to the virtual key may be preset in advance.

[0122] Exemplarily, the relationship between the user's fingertip and the instruction corresponding to the virtual key may include the relationship shown in Table 2 below.

[0123] Table 2 Fingertips and virtual buttons

[0124] User's fingertips instruction A fingertip hovers over the holographic surface Execute the command preset for the corresponding virtual button Two fingertips close together on the holographic surface Control the reduction of holograms ... ...

[0125] Exemplarily, when the holographic image includes virtual buttons, the control device 200 compares the position information of each fingertip of the user's hand with the position information of the virtual buttons. If the distance between one of the user's fingertips and the virtual button is less than or equal to a preset distance, the instruction corresponding to the virtual button is executed.

[0126] It should be noted that the preset distance is set according to user needs and is not limited here.

[0127] Exemplarily, the virtual button can also be associated with the physical function button of the vehicle. Exemplarily, the virtual button is associated with the sunroof button of the vehicle, and can realize the function of the sunroof button. When the user needs to open the sunroof, he clicks the virtual button. When it is detected that the distance between the user's fingertip and the virtual button is less than the preset distance, the control device 200 controls the sunroof to open.

[0128] It is understandable that virtual buttons simplify the driving operation process, enhance the user experience and the sense of technology of driving operation, and provide a more convenient way for driving operation.

[0129] (3) Interaction information includes facial expression information.

[0130] In some embodiments, the facial expression information includes a facial image of the user; when the interaction information includes facial expression information, the control device 200 is configured to recognize the user's instructions based on the interaction information, including: recognizing the user's facial expression based on the facial image; determining the user's instructions based on the user's facial expression and a second preset instruction library.

[0131] The second preset instruction library is used to reflect instructions corresponding to a variety of preset facial expressions.

[0132] Exemplarily, the second preset instruction library includes the instructions shown in Table 3 below:

[0133] Table 3 The second preset instruction library

[0134] Preset facial expressions instruction frown Turn down the volume in the car or provide navigation assistance Smile Turn on the music Continuous blinking Adjust the temperature in your car ... ...

[0135] In some embodiments, the control device 200 is configured to: recognize the user's facial expression based on the facial image.

[0136] Exemplarily, recognizing a user's facial expression based on a facial image includes: extracting facial key points from the facial image using a facial detection algorithm; and determining the user's facial expression based on the facial key points using feature extraction, feature encoding, and expression classification.

[0137] Exemplarily, facial key points include eyes, mouth, eyebrows, nose and other parts.

[0138] Exemplarily, feature extraction is used to extract features related to expression from facial key points, such as the degree of opening or closing of eyes, the degree of curvature of mouth, the degree of raising of eyebrows, etc.

[0139] Exemplarily, feature coding is used to encode the extracted features related to expression, and the coding may adopt methods such as local binary pattern or directional gradient histogram.

[0140] Exemplarily, expression classification may use machine learning to divert the encoded features to determine the user's facial expression.

[0141] Exemplarily, in response to the user's facial expression, the control device 200 finds an instruction corresponding to the facial expression in the second preset instruction library.

[0142] It can be understood that determining the user's instructions based on the user's facial expressions simplifies the user's operating process, makes it easier for the user to issue instructions while driving, and can improve the safety of driving operations.

[0143] (iv) Interactive information includes eye contact information.

[0144] In some embodiments, the eye information includes a head image of the user and an eye image of the user; when the interaction information includes eye information, the control device 200 is configured to identify the user's instructions based on the interaction information, including: extracting the user's head coordinates from the head image; determining the user's line of sight direction based on the user's head coordinates and the user's eye image; determining the target object that the user is looking at based on the user's line of sight direction; the target object is an object in the holographic image displayed in a preset display space; and determining that the user's instructions are instructions corresponding to the target object.

[0145] In some embodiments, the control device 200 is configured to: extract the user's head coordinates from the head image, including: using a facial detection algorithm to extract the key points of the user's face from the user's head image; based on the detected facial key points, using a geometric method to calculate the rotation matrix of the head, and determining the user's head coordinates through the rotation matrix.

[0146] Among them, the key points of the face include: eyes (inner corners, outer corners, pupils), nose (nose tip, nose bridge), mouth (corners of mouth, upper lip, lower lip) and other facial contour points (jaw).

[0147] Among them, the rotation matrix describes the rotation state of the head relative to the three-dimensional coordinate system.

[0148] In some embodiments, the control device 200 is configured to: determine the user's line of sight direction based on the user's head coordinates and the user's eye image; and determine the target object that the user is looking at based on the user's line of sight direction.

[0149] Exemplarily, an eye model is generated based on eye key points among the extracted facial key points, and the user's sight direction is determined based on the geometric shape of the eye and the position of the pupil.

[0150] Exemplarily, the user's head coordinates, eye geometry, and pupil position are combined to generate a plane from the eyes to the target object the user is gazing at, determine the direction of the user's line of sight in three-dimensional space, and regard the user's line of sight as a ray that extends from the user's eyes until it intersects with the position of the holographic image, and the intersecting position is the target object.

[0151] It can be understood that corresponding instructions are executed according to the target object that the user is looking at, so that the user can interact with the holographic image without excessive movements while driving, thereby improving the safety of driving operations.

[0152] (V) Interactive information includes sound information.

[0153] In some embodiments, the control device 200 is configured to recognize the user's instruction based on the user's voice information.

[0154] Exemplarily, the control device 200 is configured to: extract text information based on the user's voice information; and determine the user's instruction based on a third preset instruction library.

[0155] The third preset instruction library is used to reflect the user's instructions corresponding to various text messages.

[0156] Exemplarily, the third preset instruction library includes the instructions shown in Table 4 below:

[0157] Table 4 The third preset instruction library

[0158] Text message instruction Window Open the car window Playing Songs Turn on the music Increase the temperature inside the car Adjust the air conditioning temperature ... ...

[0159] For example, in order to meet the needs of users in different regions, the control device 200 supports the recognition of multiple languages ​​and dialects by adding a multi-language model, wherein the multi-language model is trained with a large amount of multi-language and dialect data.

[0160] In some embodiments, it is assumed that the auxiliary functions of the vehicle include a fleet interconnection function, wherein the fleet interconnection function is used to ensure communication between multiple vehicles when multiple vehicles travel together. The control devices 200 of multiple vehicles can be connected wirelessly. When multiple vehicles are driving, holographic images such as the location information of multiple vehicles traveling together, navigation maps, and group chat information of the fleet can be seen in the preset display space of each vehicle. The control device 200 responds to receiving the sound information to identify the user's command. If the user's command is used to chat in the group chat, the sound information is converted into text and displayed in the holographic image, which facilitates communication between users of multiple vehicles and enhances the fun of driving operations.

[0161] Optionally, the fleet interconnection function has a certain position monitoring function, and one of the vehicles is used as the central vehicle. When the speed of a vehicle among multiple vehicles exceeds a preset speed threshold or the distance between a vehicle and the central vehicle exceeds a preset distance threshold, the position point corresponding to the vehicle in the holographic image flashes and voice broadcasts to remind the central vehicle, and the control device 200 of the central vehicle can send a prompt message to the corresponding vehicle, reminding the user of the corresponding vehicle to slow down and stop in a safe area.

[0162] In some embodiments, Figure 3 As shown, the holographic projection device 300 includes a first holographic projection device 310 and a second holographic projection device 320 .

[0163] The first holographic projection device 310 is arranged in the main cockpit; the second holographic projection device 320 is arranged in the co-pilot cabin.

[0164] Exemplarily, the instrument area divides the main cockpit and the co-pilot cabin into two independent spaces, reducing interference between the main driver and the co-pilot users.

[0165] In some embodiments, the preset display space includes a first preset display space located in the main cockpit and a second preset display space located in the co-cockpit.

[0166] Exemplarily, the first holographic projection device 310 is used to form a first holographic image in a first preset display space.

[0167] Exemplarily, the second holographic projection device 320 is used to form a second holographic image in a second preset display space.

[0168] Exemplarily, the first holographic image is a holographic image viewed by the main driver, and can be generated based on an imaging instruction of the main driver. The main driver can interact with the first holographic image to generate interactive information.

[0169] Exemplarily, the second holographic image is a holographic image viewed by the co-pilot user, and can be generated based on an imaging instruction of the co-pilot user. The co-pilot user can interact with the second holographic image to generate interactive information.

[0170] In some embodiments, Figure 3 As shown, the collection device 100 includes a first collection device 110 and a second collection device 120; the first collection device 110 is arranged in the main cockpit; the second collection device 120 is arranged in the co-pilot.

[0171] Exemplarily, the first acquisition device 110 is used to collect interaction information between the main driver and the first holographic image.

[0172] Exemplarily, the second acquisition device 120 is used to collect interaction information between the passenger seat user and the second holographic image.

[0173] For example, the first collection device may be located at the A-pillar of the front windshield; the second collection device may be located at the B-pillar of the front windshield.

[0174] In some embodiments, the first acquisition device 110 and the first holographic image of the main cockpit are mainly used for auxiliary driving functions when the vehicle is driven by the user, focusing on the safety of driving operations, collecting interactive information of the main cockpit user, and executing the instructions of the main driving user. For example, the auxiliary driving functions are navigation instructions, vehicle status monitoring, visual feedback of the driving assistance system, etc. The holographic image of the main cockpit should be direct and clear, and should not affect the user's realization and attention.

[0175] Exemplarily, when the auxiliary driving function includes a navigation prompt function, the control device 200 is connected to the Internet to establish a user sharing information platform. According to the information shared by the user, the first holographic projection device 310 generates an instant recommendation of highway conditions. For example, if there is congestion ahead, according to the user sharing information platform, the first holographic projection device 310 projects the accident point that causes the congestion ahead in the first preset display space, and a holographic image (holographic road condition 3D schematic diagram) within a circle with the accident point as the center and the distance from the accident point to the vehicle as the radius. Combined with the current navigation route, it displays the estimated time to pass the congested section, the scope of the accident, and the live video near the accident point, accompanied by the function of voice broadcasting. The main driving user can decide whether to play the live video according to the actual situation.

[0176] In some embodiments, the second acquisition device 120 and the second holographic image in the co-pilot cabin are mainly used to realize the interaction between the co-pilot user and the holographic image, and can be used for entertainment interaction, such as holographic movie playback, music visualization, game interaction, etc. The holographic image in the co-pilot cabin does not affect the main cockpit user.

[0177] Exemplarily, when the imaging instruction of the co-pilot user is to play a holographic movie, the control device 200 controls the second holographic projection device 320 to project a second holographic image in the second preset display space based on the imaging instruction of the co-pilot user. The second holographic image is the holographic movie that the co-pilot user wants to play.

[0178] It can be understood that separately collecting the interaction information between the main cockpit and the co-pilot cockpit can reduce the functional disorder caused by the interaction information being mixed together. Separating the execution instructions of the main driver and the co-pilot user can separate the assisted driving interaction and the entertainment interaction, thereby improving the safety of driving operations.

[0179] In some embodiments, the control device 200 is further configured to: when it is detected that the vehicle is in an abnormal state, control the holographic projection device to project a holographic image including warning information in a preset display space.

[0180] Exemplarily, assuming that the control device 200 detects that the tire pressure of the vehicle tire is greater than a preset operating threshold, it controls the preset display space to project a holographic image of the vehicle (a miniature transparent floating 3D model) and highlights and flashes the tires with dangerous tire pressure.

[0181] Furthermore, the holographic image can also prompt the user to slow down and drive to a safe area for inspection and repair (for example, temporarily repairing the tire with repair fluid). After the tire pressure is temporarily repaired and stabilized, the holographic image can also provide the user with a navigation route to the nearest repair shop or after-sales center to repair the vehicle.

[0182] It can be understood that the control device 200 detects whether the vehicle is in an abnormal state. For a vehicle in an abnormal state, the control device 200 can timely project a holographic image through the holographic projection device 300 to remind the user to promptly understand the abnormal situation of the vehicle and improve the safety of driving operations.

[0183] In some embodiments, the acquisition device 100 is also used to acquire a status image of the user.

[0184] In some embodiments, the control device 200 is further configured to: analyze the user's status based on the user's status image; and when the user is in a preset state, control the holographic projection device to project a preset holographic warning image in a preset display space.

[0185] Exemplarily, the user's status includes at least one of the following: facial expression, number of blinks.

[0186] Exemplarily, assuming that the preset state is fatigue driving, it includes at least one of the following: the number of blinks is less than or equal to the preset blink threshold, the eye closure time is greater than or equal to the preset closure threshold, smiling is reduced, and pupil dilation.

[0187] It should be noted that the preset blink threshold and the preset closing threshold can be set according to actual conditions and are not limited here.

[0188] For example, facial expressions and blink times are combined for analysis to determine that the user is in a fatigue driving state, and a preset holographic warning image is projected in a preset display space, accompanied by a voice reminder function. The preset holographic warning image is used to remind the user to pay attention to driving safety.

[0189] Exemplarily, if the vehicle is driving, the control device 200 is also used to generate a solution and display the solution in a preset display space through the holographic projection device 300. Exemplarily, if the vehicle is driving on a highway, the control device 200 can display the navigation route of the nearest service area through the holographic projection device 300; if the vehicle is driving on a non-highway, the control device 200 can display the navigation route of the nearest safe area through the holographic projection device 300. In this way, on the one hand, the user can be prompted to take a rest in time, and on the other hand, a solution can be given intuitively, which can improve the driving user experience and improve driving safety.

[0190] Optionally, when the vehicle has an automatic driving function, if it is detected that the user is in a fatigue driving state, the control device 200 can control the vehicle to turn on the automatic driving function to assist driving; or, the control device 200 can control the vehicle to slow down and drive to a safe area based on road conditions, providing the user with rest time and avoiding long-term fatigue driving.

[0191] It can be understood that the driving operating system provided by the present application is able to judge the user's status and provide timely warnings and assistance when it detects that the user is driving fatigued, thereby improving the safety of driving operations.

[0192] It is understandable that if Figure 4 As shown, in the driving operating system provided by the present application, the holographic projection device 300 can form a holographic image in a preset display space, the acquisition device 100 can acquire the interaction information between the user and the holographic image, and send it to the control device 200, and the control device 200 can recognize the user's instructions based on the interaction information between the user and the holographic image, and execute the user's instructions.

[0193] It should be noted that through the 3D images, sounds, intelligent control and other outputs of the holographic image, the user can realize intelligent driving operations, entertainment interaction, intelligent information exchange, sharing and other functions, thereby improving the user's driving operation experience and driving safety.

[0194] See also Figure 5 , is a flowchart of a driving operation method provided by this application. It can be applied to the above Figure 1 The control device shown in Figure 5As shown, the method can be implemented as the following steps S201-S202:

[0195] S201: Receive interaction information between a user and a holographic image.

[0196] The holographic image is a holographic image formed by a holographic projection device in a preset display space.

[0197] Exemplarily, the preset display space is a pre-set three-dimensional area for displaying holographic images. The size, shape and position of the preset display space can be set and adjusted according to actual needs and are not limited here.

[0198] Exemplarily, the interaction information includes at least one of the following: sound information, action information, facial expression information, and eye contact information.

[0199] Exemplarily, the interaction information is collected by a collection device and sent to the control device.

[0200] S202: Identify the user's instruction based on the interaction information, and execute the user's instruction.

[0201] Exemplarily, the user's instructions include imaging instructions and / or operation instructions.

[0202] It is understandable that the driving operation method provided in the embodiment of the present application forms a holographic image in a preset display space, which can reduce the occlusion of the user's line of sight by the holographic image and improve the user's driving safety. In addition, the present application realizes intelligent interaction between the user and the holographic image by receiving the interaction information between the user and the holographic image, identifying the user's instructions based on the interaction information, and executing the user's instructions.

[0203] In some embodiments, when the user's instruction is an imaging instruction, the "executing the user's instruction" in the above step S202 can be specifically implemented as: controlling the holographic projection device to project the holographic image corresponding to the imaging instruction in a preset display space.

[0204] Exemplarily, the holographic image is determined by a preset correspondence relationship, which is a correspondence relationship between imaging instructions and holographic images. Users can set and add it according to actual needs.

[0205] For example, it is assumed that the auxiliary functions of the vehicle include a gourmet function. Among them, the gourmet function can provide users with holographic image data and voice introductions of gourmets within a preset range. When the gourmet function is turned on in the vehicle, the control device is connected to the Internet. If an imaging instruction issued by the user is received to instruct to generate holographic image data of gourmet recommendations within a preset range of the vehicle in a preset display space, then in response to the user's imaging instruction, the holographic image projects the holographic image data of gourmet recommendations within the preset range accompanied by voice introductions.

[0206] Optionally, the control device combines with the navigation system to obtain the route and time for walking and driving to the food destination selected by the user, and controls the holographic projection device to project a holographic image to display the route and time for walking and driving to the food destination selected by the user for the user to choose.

[0207] In some embodiments, when the user's instruction is an operation instruction, "executing the user's instruction" in the above step S202 can be specifically implemented as: executing the operation corresponding to the operation instruction; or sending the operation instruction to the corresponding operation unit.

[0208] Exemplarily, when the user's operation instruction is to increase the current navigation volume, the control device controls the current navigation volume to increase to a corresponding volume.

[0209] Exemplarily, when the user's operation instruction is to lower the air-conditioning temperature, the control device sends the user's operation instruction to the air-conditioning control unit, and the air-conditioning control unit turns on the air-conditioning and adjusts it to a corresponding temperature.

[0210] In some embodiments, the motion information is a hand image including the user's hand motions.

[0211] Exemplarily, in the case where the interaction information includes action information, the "identifying the user's instruction based on the interaction information" in the above step S202 can be specifically implemented as the following steps Sa1 to Sa4:

[0212] Sa1. Process the hand image and extract the user's hand area image.

[0213] Exemplarily, the control device preprocesses the hand image, and uses a skin color detection or edge detection algorithm to extract the user's hand area image from the preprocessed hand image.

[0214] Among them, preprocessing includes: denoising processing, grayscale processing, binarization processing, etc., which are used to improve the accuracy of user instructions.

[0215] Sa2. Extract the user's hand contour and the position information of each fingertip of the hand from the user's hand area image.

[0216] Exemplarily, a contour detection algorithm is used to extract the contour of the user's hand; and the position information of each fingertip of the hand is determined based on the distance from a point on the contour of the user's hand to the center of mass of the hand area.

[0217] Sa3. Determine the user's hand movements based on the user's hand contour and the position information of each fingertip of the hand.

[0218] Exemplarily, hand features for hand motion recognition are extracted based on the user's hand contour and position information of each fingertip of the hand; and the user's hand motion is determined based on the user's hand features.

[0219] Among them, hand features include: relative position between fingers, degree of finger bending, palm orientation, etc.

[0220] Sa4. Determine the user's instruction based on the hand movement and the first preset instruction library.

[0221] The first preset instruction library is used to reflect instructions corresponding to a variety of preset hand movements.

[0222] Exemplarily, the first preset instruction library can refer to Table 1, which will not be described in detail here.

[0223] Exemplarily, in the case where the interaction information includes action information, the "identifying the user's instruction based on the interaction information" in the above step S202 can also be implemented as the following steps Sc1 to Sc3:

[0224] Sc1. Process the hand image and extract the user's hand area image.

[0225] Exemplarily, the extraction of the hand area image may refer to the above step Sa1, which will not be described in detail here.

[0226] Sc2. Extract the user's hand contour and the position information of each fingertip of the hand from the user's hand area image.

[0227] Exemplarily, the extraction of the user's hand contour and the position information of each fingertip of the hand refers to the above-mentioned step Sa2, which will not be described in detail here.

[0228] Sc3. If the distance between one of the user's fingertips and the virtual key fingertip is less than or equal to a preset distance, it is determined that the user instruction is an instruction corresponding to the virtual key.

[0229] Exemplarily, a three-dimensional coordinate system is established based on the vehicle's cockpit, the origin of the three-dimensional coordinate system can be set by the user, the position information of the preset display space can be determined based on the coordinate information of the preset display space, and the position information of the holographic image can be determined based on the coordinate information of the holographic image.

[0230] Exemplarily, the instruction corresponding to the virtual key may be preset in advance.

[0231] Exemplarily, the relationship between the user's fingertips and the instructions corresponding to the virtual keys can refer to the relationship in Table 2 above, which will not be elaborated here.

[0232] It should be noted that the preset distance is set according to user needs and is not limited here.

[0233] Exemplarily, the virtual button can also be associated with the vehicle's physical function buttons. Exemplarily, the virtual button is associated with the vehicle's sunroof button to realize the function of the sunroof button. When the user needs to open the sunroof, he clicks the virtual button. When it is detected that the distance between the user's fingertips and the virtual button is less than a preset distance, the control device controls the sunroof to open.

[0234] In some embodiments, the facial expression information includes an image of the user's face.

[0235] Exemplarily, in the case where the interaction information includes facial expression information, the "identifying the user's instruction based on the interaction information" in the above step S202 can be specifically implemented as steps Sd1 to Sd2:

[0236] Sd1. Recognize the user's facial expression based on the facial image.

[0237] Exemplarily, a facial detection algorithm is used to extract facial key points from a facial image; based on the facial key points, feature extraction, feature encoding, and expression classification are used to determine the user's facial expression.

[0238] Sd2. Determine the user's instruction based on the user's facial expression and the second preset instruction library.

[0239] The second preset instruction library is used to reflect instructions corresponding to a variety of preset facial expressions.

[0240] Exemplarily, the second preset instruction library refers to the above Table 3, which will not be described in detail here.

[0241] Exemplarily, based on identifying the user's facial expression, an instruction corresponding to the facial expression is found in a second preset instruction library.

[0242] In some embodiments, the eye information includes an image of the user's head and an image of the user's eyes.

[0243] Exemplarily, in the case where the interaction information includes eye contact information, the "instruction of identifying the user based on the interaction information" in the above step S202 can be specifically implemented as the following steps Se1 to Se4:

[0244] Se1. Extract the user's head coordinates from the head image.

[0245] Exemplarily, extracting the user's head coordinates from the head image includes: extracting the key points of the user's face from the user's head image using a facial detection algorithm; calculating the rotation matrix of the head using a geometric method based on the detected facial key points, and determining the user's head coordinates through the rotation matrix.

[0246] Among them, the key points of the face include: eyes (inner corners, outer corners, pupils), nose (nose tip, nose bridge), mouth (corners of mouth, upper lip, lower lip) and other facial contour points (jaw).

[0247] Among them, the rotation matrix describes the rotation state of the head relative to the three-dimensional coordinate system.

[0248] Se2. Determine the user's line of sight direction based on the user's head coordinates and the user's eye image.

[0249] Exemplarily, an eye model is generated based on eye key points among the extracted facial key points, and the user's sight direction is determined based on the geometric shape of the eye and the position of the pupil.

[0250] Se3. Determine the target object the user is looking at based on the user's line of sight.

[0251] Exemplarily, the user's head coordinates, eye geometry, and pupil position are combined to generate a plane from the eyes to the target object the user is gazing at, determine the direction of the user's line of sight in three-dimensional space, and regard the user's line of sight as a ray that extends from the user's eyes until it intersects with the position of the holographic image, and the intersecting position is the target object.

[0252] Se4. Determine that the user instruction is the instruction corresponding to the target object.

[0253] Exemplarily, the instructions corresponding to the target object are preset instructions and can be set by oneself.

[0254] In some embodiments, the holographic projection device includes a first holographic projection device and a second holographic projection device; wherein the first holographic projection device is arranged in the main cockpit; and the second holographic projection device is arranged in the co-pilot cabin.

[0255] In some embodiments, the preset display space includes a first preset display space located in the main cockpit and a second preset display space located in the co-cockpit.

[0256] Exemplarily, the first holographic projection device is used to form a first holographic image in a first preset display space; the second holographic projection device is used to form a second holographic image in a second preset display space;

[0257] In some embodiments, the interaction information between the user and the holographic image includes: interaction information between the main driver user and the first holographic image, and interaction information between the co-driver user and the second holographic image.

[0258] In some embodiments, the collection device includes a first collection device and a second collection device.

[0259] Exemplarily, the first acquisition device is disposed in the main cockpit; the second acquisition device is disposed in the co-pilot cockpit.

[0260] Exemplarily, the interaction information between the main driver and the first holographic image is collected by the first collection device.

[0261] Exemplarily, the interaction information between the front passenger user and the second holographic image is collected by the second collection device.

[0262] In some embodiments, the driving operation method provided by the embodiments of the present application also includes: when it is detected that the vehicle is in an abnormal state, controlling the holographic projection device to project a holographic image including warning information in a preset display space.

[0263] Assuming that the control device detects that the tire pressure of the vehicle is greater than the preset operating threshold, the control device projects a holographic image of the vehicle (a miniature transparent floating 3D model) into the preset display space, and highlights and flashes the tires with dangerous tire pressure.

[0264] In some embodiments, the driving operation method provided in the embodiment of the present application further includes the following steps S203 to S205:

[0265] S203: Receive a user status image.

[0266] S204: Analyze the user's status based on the user's status image.

[0267] Exemplarily, the user's status includes at least one of the following: facial expression, number of blinks.

[0268] S205: When the user is in a preset state, control the holographic projection device to project a preset holographic warning image in a preset display space.

[0269] Exemplarily, the preset holographic warning image can serve as a reminder, which can be set by oneself and is not limited here.

[0270] Exemplarily, assuming that the preset state is fatigue driving, it includes at least one of the following: the number of blinks is less than or equal to the preset blink threshold, the eye closure time is greater than or equal to the preset closure threshold, smiling is reduced, and pupil dilation.

[0271] It should be noted that the preset blink threshold and the preset closing threshold can be set according to actual conditions and are not limited here.

[0272] For example, facial expressions and the number of blinks are combined for analysis to determine that the user is in a state of fatigue driving, and a preset holographic warning image is projected in a preset display space, accompanied by a voice reminder function.

[0273] In some embodiments, the user can add special features according to actual needs, and establish a holographic image menu based on the special features. The menu may include but is not limited to at least one of the following: intersection guards, game cars, holographic interactive game platforms, camping headquarters, etc. The user's instructions are determined and executed based on the user's interactive information with the holographic image.

[0274] In some embodiments, when the vehicle is in a waiting state, the driving operation method provided in the present application can monitor the real-time environment of the road surface. The control device can not only control the radar to detect vehicles within a preset range and issue an alarm, but also combine the vehicle's navigation system to obtain the road conditions ahead in the waiting state, and control the holographic image display in the preset display space. The control device can also obtain the location information of nearby parking lots, and display shortcut icons of nearby parking lots in the holographic image.

[0275] For example, when the vehicle's assisted driving function includes the intersection guard function, the control device obtains the local dynamic information of the intersection that the vehicle is about to pass, and controls the holographic projection device to form a holographic image of the local dynamic information of the intersection that the vehicle is about to pass, thereby reducing the occurrence of accidents. The local dynamic information of the intersection includes: the speed and intended path of people crossing the road, and vehicles that are sideways or turning and cannot be directly seen.

[0276] For example, when the vehicle's entertainment function includes a game car function, the control device can simulate an existing game car module, allowing the user to operate the steering wheel and the accelerator and brake without starting the vehicle, and control the generation of a coordinated holographic image in a preset display space to play a racing game. The holographic image can also be controlled to display the dynamic changes of the track, vehicle speed, road conditions and various parameter panels, truly restoring the tension and excitement of racing.

[0277] For example, the control device can also be combined with a large group of car owners to develop game interactions to increase the fun of driving operations.

[0278] For example, when the vehicle's entertainment function includes an independent camping management function, the control device can dynamically monitor the external power supply device and obtain various external power supply device status information, and display it in the holographic image. In addition, the control device can also include the built-in power parameters of the external power grid device, the vehicle's oil and electricity reserves, and the round-trip mileage, and display them in the holographic image.

[0279] It should be noted that the vehicle cannot be in driving state under the game car, holographic interactive game platform and camping headquarters functions.

[0280] For example, Figure 6 This is a schematic diagram of the components of a driving operation device provided in an embodiment of the present application. Figure 6As shown, the driving operation device 800 includes: a communication module 801 and a processing module 802, the communication module 801 is used to receive interaction information between the user and the holographic image; the holographic image is a holographic image formed by a holographic projection device in a preset display space; the processing module 802 uses the interaction information to identify the user's instructions and execute the user's instructions.

[0281] In some embodiments, the processing module 802 is specifically used to control the holographic projection device to project a holographic image corresponding to the imaging instruction in a preset display space.

[0282] In some embodiments, when the user's instruction is an operation instruction, the processing module 802 is specifically used to execute the operation corresponding to the operation instruction; or, send the operation instruction to the corresponding operation unit.

[0283] In some embodiments, the interaction information includes at least one of the following: sound information, action information, facial expression information, and eye contact information.

[0284] In some embodiments, the action information is a hand image including the user's hand actions; when the interaction information includes action information, the processing module 802 is specifically used to process the hand image to extract the user's hand area image; extract the user's hand contour and the position information of each fingertip of the hand from the user's hand area image; determine the user's hand action based on the user's hand contour and the position information of each fingertip of the hand; determine the user's instruction based on the hand action and the first preset instruction library; the first preset instruction library is used to reflect the instructions corresponding to multiple preset hand actions.

[0285] In some embodiments, the action information is a hand image including the user's hand action; when the interaction information includes action information, the processing module 802 is specifically used to process the hand image and extract the user's hand area image; extract the user's hand contour and the position information of each fingertip of the hand from the user's hand area image; if the distance between one of the user's fingertips and the virtual key fingertip is less than or equal to the preset distance, the user command is determined to be the command corresponding to the virtual key; wherein the virtual key is a virtual key in the holographic image displayed in the preset display space.

[0286] In some embodiments, facial expression information includes a facial image of a user; when the interaction information includes facial expression information, the processing module 802 is specifically used to recognize the user's facial expression based on the facial image; based on the user's facial expression and a second preset instruction library, determine the user's instructions; the second preset instruction library is used to reflect instructions corresponding to multiple preset facial expressions.

[0287] In some embodiments, the eye information includes a head image of the user and an eye image of the user; when the interaction information includes eye information, the processing module 802 is specifically used to extract the head coordinates of the user from the head image; determine the direction of the user's line of sight based on the head coordinates of the user and the eye image of the user; determine the target object that the user is looking at based on the direction of the user's line of sight; the target object is an object in the holographic image displayed in a preset display space; and determine that the user's command is a command corresponding to the target object.

[0288] In some embodiments, the holographic projection device includes a first holographic projection device and a second holographic projection device; wherein the first holographic projection device is arranged in a main cockpit; the second holographic projection device is arranged in a co-pilot cockpit; the preset display space includes a first preset display space located in the main cockpit and a second preset display space located in the co-pilot cockpit; the first holographic projection device is used to form a first holographic image in the first preset display space; the second holographic projection device is used to form a second holographic image in the second preset display space; the interaction information between the user and the holographic image includes: interaction information between the main driver user and the first holographic image, interaction information between the co-pilot user and the second holographic image.

[0289] In some embodiments, the acquisition device includes a first acquisition device and a second acquisition device; the first acquisition device is arranged in the main cockpit; the second acquisition device is arranged in the co-pilot cockpit; the interaction information between the main driver user and the first holographic image is collected by the first acquisition device; the interaction information between the co-pilot user and the second holographic image is collected by the second acquisition device.

[0290] In some embodiments, the processing module 802 is further configured to control the holographic projection device to project a holographic image including warning information in a preset display space when it is detected that the vehicle is in an abnormal state.

[0291] In some embodiments, the communication module 801 is also used to receive a user's status image; the processing module 802 is also used to analyze the user's status based on the user's status image; when the user is in a preset state, the holographic projection device is controlled to project a preset holographic warning image in a preset display space.

[0292] The embodiment of the present invention provides a possible structural diagram of the device involved in the above embodiment. Figure 7 As shown, the electronic device 900 includes: a processor 902 , a communication interface 903 , and a bus 904 . Optionally, the electronic device 900 may further include a memory 901 .

[0293] The processor 902 may be a processor that implements or executes various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of the present application. The processor 902 may be a central processing unit, a general purpose processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may be a processor that implements or executes various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of the present application.

[0294] The processor 902 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0295] The communication interface 903 is used to connect with other devices via a communication network, such as Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0296] The memory 901 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0297] As a possible implementation, the memory 901 may exist independently of the processor 902, and the memory 901 may be connected to the processor 902 via a bus 904 to store instructions or program codes. When the processor 902 calls and executes the instructions or program codes stored in the memory 901, the identification resolution method provided in the embodiment of the present invention can be implemented.

[0298] In another possible implementation, the memory 901 may also be integrated with the processor 902 .

[0299] The bus 904 may be an extended industry standard architecture (EISA) bus, etc. The bus 904 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7It is represented by only one thick line, but it does not mean that there is only one bus or one type of bus.

[0300] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the service calling device can be divided into different functional modules to complete all or part of the functions described above.

[0301] The embodiment of the present application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be completed by computer instructions to instruct the relevant hardware, and the program can be stored in the above computer-readable storage medium. When the program is executed, it may include the processes of the above method embodiments. The computer-readable storage medium can be the memory or memory of any of the above embodiments. The above computer-readable storage medium can also be an external storage device of the above service calling device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the above service calling device. Further, the above computer-readable storage medium can also include both the internal storage unit of the above service calling device and an external storage device. The above computer-readable storage medium is used to store the above computer program and other programs and data required by the above service calling device. The above computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0302] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program product runs on a computer, the computer is enabled to execute any one of the identification resolution methods provided in the above embodiments.

[0303] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A driving operating system, characterized in that: include: Acquisition device, control device and holographic projection device; The holographic projection device is used to form a holographic image in a preset display space; The collection device is used to collect interaction information between the user and the holographic image; The control device is configured to: identify a user's instruction based on the interaction information, and execute the user's instruction.

2. The driving operating system according to claim 1, characterized in that: In the case where the user's instruction is an imaging instruction, the control device is configured to execute the user's instruction, including: controlling the holographic projection device to project a holographic image corresponding to the imaging instruction in the preset display space.

3. The driving operating system according to claim 2, characterized in that: In the case where the user's instruction is an operation instruction, the control device is configured to execute the user's instruction, including: Execute the operation corresponding to the operation instruction; Alternatively, the operation instruction is sent to a corresponding operation unit.

4. The driving operating system according to claim 1, characterized in that: The interaction information includes at least one of the following: Sound information, movement information, facial expression information, and eye information.

5. The driving operating system according to claim 4, characterized in that: The action information is a hand image including a hand action of the user; when the interaction information includes the action information, the control device is configured to recognize a user instruction based on the interaction information, including: Processing the hand image to extract a hand area image of the user; Extracting the user's hand contour and position information of each fingertip of the hand from the user's hand area image; Determining a hand motion of the user based on the hand contour of the user and the position information of each fingertip of the hand; Based on the hand motion and a first preset instruction library, the user's instruction is determined; the first preset instruction library is used to reflect instructions corresponding to a variety of preset hand motions.

6. The driving operation system according to claim 4, characterized in that: The action information is a hand image including a hand action of the user; when the interaction information includes the action information, the control device is configured to recognize a user instruction based on the interaction information, including: Processing the hand image to extract a hand area image of the user; Extracting the user's hand contour and position information of each fingertip of the hand from the user's hand area image; If the distance between one of the user's fingertips and the virtual key fingertip is less than or equal to a preset distance, the user instruction is determined to be an instruction corresponding to the virtual key; wherein the virtual key is a virtual key in the holographic image displayed in the preset display space.

7. The driving operating system according to claim 4, characterized in that: The facial expression information includes a facial image of a user; when the interaction information includes the facial expression information, the control device is configured to recognize the user's instruction based on the interaction information, including: recognizing a facial expression of the user based on the facial image; Based on the facial expression of the user and a second preset instruction library, the user's instruction is determined; the second preset instruction library is used to reflect instructions corresponding to a variety of preset facial expressions.

8. The driving operating system according to claim 4, characterized in that: The eye information includes a head image of the user and an eye image of the user; In the case where the interaction information includes the eye information, the control device is configured to recognize the user's instruction based on the interaction information, including: Extracting the head coordinates of the user from the head image; Determine the sight direction of the user based on the head coordinates of the user and the eye image of the user; Determining a target object that the user is gazing at based on the user's line of sight; the target object is an object in the holographic image displayed in the preset display space; Determine that the user instruction is an instruction corresponding to the target object.

9. The driving operating system according to claim 1, characterized in that: The holographic projection device comprises a first holographic projection device and a second holographic projection device; wherein the first holographic projection device is arranged in the main cockpit; the second holographic projection device is arranged in the co-pilot; the preset display space comprises a first preset display space in the main cockpit and a second preset display space in the co-pilot; The first holographic projection device is used to form a first holographic image in the first preset display space; The second holographic projection device is used to form a second holographic image in the second preset display space.

10. The driving operation system according to claim 9, characterized in that: The collection device includes a first collection device and a second collection device; the first collection device is arranged in the main cockpit; the second collection device is arranged in the co-pilot cabin; The first acquisition device is used to acquire interaction information between the main driver and the first holographic image; The second collecting device is used to collect interaction information between the front passenger user and the second holographic image.

11. The driving operating system according to claim 1, characterized in that: The control device is also configured to: When it is detected that the vehicle is in an abnormal state, the holographic projection device is controlled to project a holographic image including warning information in the preset display space.

12. The driving operating system according to claim 1, characterized in that: The acquisition device is also used to acquire the user's status image; The control device is further configured to: analyze the state of the user based on the state image of the user; and control the holographic projection device to project a preset holographic warning image in the preset display space when the user is in a preset state.

13. A driving operation method, characterized in that: The method comprises: Receiving interaction information between a user and a holographic image; the holographic image is a holographic image formed by a holographic projection device in a preset display space; The user's instruction is identified based on the interaction information, and the user's instruction is executed.

14. The method according to claim 13, characterized in that In the case where the user's instruction is an imaging instruction, executing the user's instruction includes: The holographic projection device is controlled to project a holographic image corresponding to the imaging instruction in the preset display space.

15. The method according to claim 13, characterized in that In the case where the user's instruction is an operation instruction, executing the user's instruction includes: Execute the operation corresponding to the operation instruction; Alternatively, the operation instruction is sent to a corresponding operation unit.

16. The method according to claim 13, characterized in that The interaction information includes at least one of the following: Sound information, movement information, facial expression information, and eye information.

17. The method according to claim 16, characterized in that The action information is a hand image including a hand action of the user; when the interaction information includes the action information, the instruction for identifying the user based on the interaction information includes: Processing the hand image to extract a hand area image of the user; Extracting the user's hand contour and position information of each fingertip of the hand from the user's hand area image; Determining a hand motion of the user based on the hand contour of the user and the position information of each fingertip of the hand; Based on the hand motion and a first preset instruction library, the user's instruction is determined; the first preset instruction library is used to reflect instructions corresponding to a variety of preset hand motions.

18. The method according to claim 16, characterized in that The action information is a hand image including a hand action of the user; when the interaction information includes the action information, the instruction for identifying the user based on the interaction information includes: Processing the hand image to extract a hand area image of the user; Extracting the user's hand contour and position information of each fingertip of the hand from the user's hand area image; If the distance between one of the user's fingertips and the virtual key fingertip is less than or equal to a preset distance, the user instruction is determined to be an instruction corresponding to the virtual key; wherein the virtual key is a virtual key in the holographic image displayed in the preset display space.

19. The method according to claim 16, characterized in that The facial expression information includes a facial image of a user; when the interaction information includes the facial expression information, the instruction for identifying the user based on the interaction information includes: recognizing a facial expression of the user based on the facial image; Based on the facial expression of the user and a second preset instruction library, the user's instruction is determined; the second preset instruction library is used to reflect instructions corresponding to a variety of preset facial expressions.

20. The method according to claim 16, characterized in that The eye information includes a head image of the user and an eye image of the user; when the interaction information includes the eye information, the instruction for identifying the user based on the interaction information includes: Extracting the head coordinates of the user from the head image; Determine the sight direction of the user based on the head coordinates of the user and the eye image of the user; Determining a target object that the user is gazing at based on the user's line of sight; the target object is an object in the holographic image displayed in the preset display space; Determine that the user instruction is an instruction corresponding to the target object.

21. The method according to claim 13, characterized in that The holographic projection device comprises a first holographic projection device and a second holographic projection device; wherein the first holographic projection device is arranged in the main cockpit; the second holographic projection device is arranged in the co-pilot; the preset display space comprises a first preset display space in the main cockpit and a second preset display space in the co-pilot; the first holographic projection device is used to form a first holographic image in the first preset display space; the second holographic projection device is used to form a second holographic image in the second preset display space; The interaction information between the user and the holographic image includes: interaction information between the main driver user and the first holographic image, and interaction information between the co-driver user and the second holographic image.

22. The method according to claim 21, characterized in that The collection device includes a first collection device and a second collection device; the first collection device is arranged in the main cockpit; the second collection device is arranged in the co-pilot cabin; The interaction information between the main driving user and the first holographic image is collected by the first collecting device; The interaction information between the front passenger user and the second holographic image is collected by the second collection device.

23. The method according to claim 13, characterized in that The method further comprises: When it is detected that the vehicle is in an abnormal state, the holographic projection device is controlled to project a holographic image including warning information in the preset display space.

24. The method according to claim 13, characterized in that The method further comprises: receiving a status image of the user; analyzing the state of the user based on the state image of the user; When the user is in a preset state, the holographic projection device is controlled to project a preset holographic warning image in the preset display space.

25. An electronic device, characterized in that: It comprises a processor and a memory, the processor is coupled to the memory; the memory is used to store computer instructions, the computer instructions are loaded and executed by the processor to enable the computer device to implement the driving operation method as described in any one of claims 13 to 24.

26. A vehicle, characterized in that: Comprising the driving operating system as described in any one of claims 1 to 12, or the electronic device as described in claim 25.

27. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes computer-executable instructions, and when the computer-executable instructions are executed on a computer, the computer is caused to execute the driving operation method according to any one of claims 13 to 24.

28. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is run on an electronic device, the electronic device executes the driving operation method according to any one of claims 13 to 24.

Images