Vehicle control method and device, electronic equipment and storage medium

Abstract

The invention discloses a vehicle control method and device, electronic equipment and a storage medium, and the method comprises the steps that mixed reality MR equipment displays a virtual space model corresponding to a vehicle, and the virtual space model is generated according to the internal space information of the vehicle; if the control information of the target equipment in the vehicle is obtained, displaying a virtual control panel of the target equipment pair in the virtual space model; if the control operation for the virtual control panel is obtained, control parameters are sent to the vehicle so that the vehicle can correspondingly control the target equipment based on the control parameters, and the control parameters are determined according to the control operation, so that the three-dimensional interaction space of the vehicle-mounted scene is expanded through the vehicle-mounted MR equipment; a new and more intuitive interaction mode is expanded for the user to control the vehicle, and the operation intuition and immersive interaction experience of the user are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mixed reality, and more particularly, to a vehicle control method and device, an electronic device, and a storage medium. BACKGROUND

[0002] With the development of Internet of Vehicles, artificial intelligence and mixed reality display technology, people increasingly expect to provide a full range of immersive experiences in vision, hearing and somatosensory for user control of vehicles in limited cabin space. SUMMARY

[0003] Therefore, embodiments of the present application provide a vehicle control method and device, an electronic device, and a storage medium to improve the above problems.

[0004] In a first aspect, embodiments of the present application provide a vehicle control method, which includes: displaying, by a mixed reality (MR) device, a virtual space model corresponding to a vehicle, the virtual space model being generated according to internal space information of the vehicle; if control information of a target device in the vehicle is acquired, displaying a virtual control panel of the target device pair in the virtual space model; if a control operation for the virtual control panel is acquired, sending a control parameter to the vehicle to enable the vehicle to perform corresponding control on the target device based on the control parameter, the control parameter being determined according to the control operation.

[0005] In a second aspect, embodiments of the present application provide a vehicle control device, which includes: a space model display module, a control panel display module, and a device control module. The space model display module is configured to display, by an MR device, a virtual space model corresponding to a vehicle, the virtual space model being generated according to internal space information of the vehicle. The control panel display module is configured to display a virtual control panel of a target device pair in the virtual space model if control information of the target device in the vehicle is acquired. The device control module is configured to send a control parameter to the vehicle to enable the vehicle to perform corresponding control on the target device based on the control parameter if a control operation for the virtual control panel is acquired, the control parameter being determined according to the control operation.

[0006] In a third aspect, embodiments of the present application provide an electronic device including a memory and a processor, the memory being coupled to the processor, and the memory storing instructions that, when executed by the processor, cause the processor to perform the vehicle control method provided in the first aspect.

[0007] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, which stores program codes. The program codes can be invoked by a processor to execute the vehicle control method provided in the first aspect.

[0008] In the scheme of the present application, the virtual space model generated according to the internal space information of the vehicle is displayed through the mixed reality (MR) device. If the control information of the target device in the vehicle is obtained, a virtual control panel of the target device pair is displayed in the virtual space model. If a control operation for the virtual control panel is obtained, a control parameter determined according to the control operation is sent to the vehicle, so that the vehicle performs corresponding control on the target device based on the control parameter. Thus, the three-dimensional interactive space of the vehicle-mounted scene is expanded through the vehicle-mounted MR device, a new and more intuitive interaction mode is provided for the user to control the vehicle, and the operation intuitiveness and immersive interaction experience of the user are improved. BRIEF DESCRIPTION OF DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0010] Figure 1 A flowchart of a vehicle control method provided by an embodiment of the present application is shown; Figure 2 A system architecture diagram of air conditioner control provided by an embodiment of the present application is shown; Figure 3 A flowchart of a vehicle control method provided by an embodiment of the present application is shown; Figure 4 A flowchart of controlling the air conditioner of the vehicle provided by an embodiment of the present application is shown; Figure 5 A module block diagram of a vehicle control device provided by an embodiment of the present application is shown; Figure 6 A block diagram of an electronic device for executing the vehicle control method according to the embodiments of the present application is shown. DETAILED DESCRIPTION

[0011] In order to enable those skilled in the art to better understand the schemes of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application.

[0012] In order to better understand the schemes of the embodiments of the present application, the technical terms used in the embodiments of the present application will be explained first.

[0013] Mixed Reality (MR) is a technology that deeply integrates virtual world and real environment.

[0014] Controller Area Network (CAN) is a serial communication protocol bus for real-time applications, which can use twisted pair to transmit signals, and is one of the most widely used field buses in the world.

[0015] HVAC is the abbreviation of Heating Ventilation and Air Conditioning, which means: heating, ventilation and air conditioning. It is a component of the vehicle interior environment control system, responsible for controlling the temperature, air flow and air quality inside the vehicle.

[0016] Inertial Measurement Unit (IMU) is a device that measures the three-axis attitude angle (or angular rate) and acceleration of an object, Internet Data Center (IDC) refers to a kind of perfect equipment (including high-speed Internet access bandwidth, high-performance local area network, safe and reliable computer room environment, etc.), professional management, perfect application service platform.

[0017] The implementation details of the technical solutions of the embodiments of the present application are described in detail as follows: In the related art, a scheme of vehicle control based on voice interaction is proposed; however, for most existing vehicle models, there are problems such as long time consumption in the interaction process and poor interaction accuracy in the process of vehicle control by the rear passenger through voice interaction (for example, vehicle air conditioner control by the rear passenger through voice interaction). Based on this, how to provide users with immersive experience in vision, hearing and somatosensory in the limited cabin space, so that they can accurately control the vehicle, is still a major challenge.

[0018] To solve the above problems, the inventors have found, after long-term research, that the vehicle control method, device, electronic equipment and storage medium provided in the embodiments of the present application are proposed, which expand the three-dimensional interaction space of the vehicle scene through the vehicle-mounted MR device, and provide a new more intuitive interaction way for users to control the vehicle, thereby improving the operation intuitiveness and immersive interaction experience of the users. The specific vehicle control method is described in detail in the subsequent embodiments.

[0019] The embodiments related to the present application will be described below with reference to the accompanying drawings.

[0020] Please refer to Figure 1 ,Figure 1 A flowchart of a vehicle control method provided by an embodiment of the present application is shown. In specific embodiments, the vehicle control method can be applied to the vehicle control device 200 shown in Figure 5 and the electronic device 100 configured with the vehicle control device 200 (as shown in Figure 6 ). In the following, the specific flow of the embodiment will be described by taking the electronic device as an example. It should be understood that the electronic device to which the embodiment is applied can include MR devices, vehicles, vehicle-mounted terminals, and other devices, which are not limited herein. In the following, the flow shown in Figure 1 will be described in detail. The vehicle control method can specifically include the following steps: Step S110: The MR device displays a virtual space model corresponding to the vehicle, which is generated according to internal space information of the vehicle.

[0021] In the embodiment, the electronic device can be understood as an MR device, which can be mounted in a vehicle and establish a communication connection with the vehicle, so as to realize spatial interaction control of a target device in the vehicle based on the spatial interaction capability of the vehicle-mounted MR device and the communication capability of the whole vehicle, to improve the operation intuitiveness and immersive interaction experience of user vehicle control.

[0022] In some embodiments, the MR device can be in communication connection with the vehicle and can display a virtual space model corresponding to the vehicle. The virtual space model is generated according to internal space information of the vehicle. Alternatively, the MR device can obtain the virtual space model corresponding to the vehicle generated by the vehicle according to the internal space information of the vehicle in the case of establishing a communication connection with the vehicle. Alternatively, the MR device can obtain the internal space information corresponding to the vehicle in the case of establishing a communication connection with the vehicle, and build a virtual space model corresponding to the vehicle based on the internal space information.

[0023] The internal space information corresponding to the vehicle can include three-dimensional calibration information of the internal space of the vehicle. After the MR device obtains the virtual space model corresponding to the vehicle, the MR device can display the virtual space model corresponding to the vehicle in the MR device, to expand the three-dimensional interaction space of the vehicle-mounted scene based on the MR device.

[0024] Step S120: If the control information of a target device in the vehicle is obtained, a virtual control panel of the target device pair is displayed in the virtual space model.

[0025] In some embodiments, the MR device can obtain the control information of the target device in the vehicle through the communication connection established with the vehicle in the process of displaying the virtual space model corresponding to the vehicle. If the MR device obtains the control information of the target device in the vehicle, a virtual control panel of the target device pair can be displayed in the virtual space model. The target device can include an air conditioner, an interior light, a window, an audio controller, etc. in the vehicle. The virtual control panel can correspond to a digital twin model of the control panel of the target device corresponding to the console of the vehicle. Optionally, the virtual control panel can include adjustment interfaces corresponding to various control parameters of the target device. The content included in the virtual control panel is not limited in this embodiment.

[0026] For example, the target device can include an air conditioner of the vehicle. If the MR device obtains the control information of the air conditioner in the vehicle through the communication connection with the vehicle, a virtual control panel corresponding to the air conditioner can be displayed in the virtual space model. The control information of the air conditioner includes but is not limited to the current temperature adjustment parameter, the blowing parameter, the air volume parameter, the air direction parameter, etc. of the air conditioner. The virtual control panel can correspond to a digital twin model of the control panel of the air conditioner on the console of the vehicle. The digital twin model can include adjustment interfaces of the control parameters of the air conditioner, which can include but are not limited to the temperature adjustment parameter, the blowing parameter, the air volume parameter, the air direction parameter, etc.

[0027] Step S130: If a control operation for the virtual control panel is obtained, a control parameter is sent to the vehicle to enable the vehicle to control the target device based on the control parameter, which is determined according to the control operation.

[0028] In some embodiments, after the MR device displays the virtual control panel of the target device pair in the virtual space model, if a control operation for the virtual control panel is obtained, a control parameter can be sent to the vehicle to enable the vehicle to control the target device based on the control parameter. The control parameter can be determined according to the control operation.

[0029] The MR device can be provided with a physical button, and the MR device can obtain the pressing information of the physical button and obtain the control operation for the virtual control panel according to the pressing information. The MR device can also be provided with a camera, and the MR device can obtain the gesture information of the user based on the camera and obtain the control operation for the virtual control panel according to the gesture information. Optionally, the MR device can also be associated with a joystick, and the MR device can obtain the operation information of the joystick and obtain the control operation for the virtual control panel according to the operation information.

[0030] For example, refer to Figure 2 Fig. 1 shows a system architecture diagram of air conditioning control provided by an embodiment of the present application. Taking control of air conditioning in a vehicle as an example, the air conditioning control system can include an MR device and a vehicle. The MR device is in communication connection with the vehicle. The MR device can be configured with an MR vehicle control application APP (vehicle-mounted MR application). The vehicle can include an IDC module and an HVAC module. The IDC module and the HVAC module can transmit data through a CAN bus network in the vehicle. The vehicle-mounted MR application in the MR device can be used to provide a vehicle control space graphical interface for a user, to process positioning of a vehicle cabin space, and to identify gesture information in the vehicle cabin. Thus, the user can have an immersive experience in visual, auditory and somatosensory aspects of vehicle control. The IDC module of the vehicle can also be configured with a vehicle-mounted MR data service, which can be used to collect and process data of the HVAC module in the vehicle, to collect and process data of an IMU module in the vehicle, and to communicate with the vehicle-mounted MR device.

[0031] The CAN bus network in the vehicle can be used for communication and signal transmission between a TBOX module of the IDC module and a vehicle ECU module. The HVAC module in the vehicle can be understood as an air conditioning system controller, which can control and logically process the air conditioning system of the vehicle and communicate with the vehicle CAN bus network. The IMU module in the vehicle can integrate an accelerometer, a gyroscope and a magnetometer in the vehicle, which can be used to collect vehicle motion information and direction information and communicate with the vehicle CAN bus network.

[0032] The IMU module and the HVAC module can communicate with the IDC module through a vehicle CAN bus network. The IDC module can run a vehicle MR control service software, and can process the data sent by the IMU module and the data sent by the HVAC module for subsequent transmission to the vehicle MR device. The MR control service running on the vehicle MR device can correspond to the MR control service running on the IDC module (IDC end). The vehicle MR device and the vehicle can establish a communication connection through wireless or wired means (such as a WiFi or USB harness connection), thereby establishing a communication channel between the MR control services of the two parties. The vehicle MR device can obtain three-dimensional calibration information of the interior space of the vehicle connected through the communication connection, and can establish a virtual space model corresponding to the vehicle and display it in the vehicle MR device when the MR control application is running. The virtual information of the virtual space model can be positioned to a specific point in the actual vehicle interior space according to the data of the IMU module of the vehicle. For example, the virtual information of the virtual space model is positioned to a specific point in the actual vehicle interior space by rotating, translating, and other processing, so as to match the virtual space model with the actual vehicle interior space, thereby expanding the three-dimensional interactive space of the vehicle scene and improving the naturalness of the expanded vehicle scene.

[0033] When the vehicle MR device runs the MR control application, if the air conditioning control information of the vehicle is obtained from the IDC module of the vehicle, a virtual control panel corresponding to the air conditioner can be displayed in the virtual space model. At the same time, the vehicle MR device can run a gesture recognition module of the MR device when running the MR control application, perceive the position of the user's hand in the vehicle cabin through the camera of the MR device, and recognize the gesture information. Accordingly, the user can control the "knob" or "key" on the virtual control panel of the air conditioner through gestures to adjust the parameters of the air conditioner. If the vehicle MR device obtains a control operation for the virtual control panel of the air conditioner, it can send a control parameter to the IDC module of the vehicle, so that the IDC module instructs the HVAC module of the vehicle to adjust the air conditioning parameters based on the control parameter, and returns the adjustment result of the air conditioning parameters to the MR device through the IDC module, and displays the adjustment result on the virtual control panel of the air conditioner through the MR device, thereby expanding the three-dimensional interactive space of the vehicle scene based on the vehicle MR device, providing a new and more intuitive interaction way for the user to control the air conditioner of the vehicle, and improving the user's operation intuitiveness and immersive interaction experience.

[0034] The vehicle control method provided by an embodiment of the present application displays a virtual space model generated according to internal space information of a vehicle through a mixed reality (MR) device, displays a virtual control panel of a target device pair in the virtual space model if control information of the target device pair in the vehicle is acquired, and sends a control parameter determined according to a control operation on the virtual control panel to the vehicle to enable the vehicle to perform corresponding control on the target device based on the control parameter, thereby expanding a three-dimensional interaction space of a vehicle-mounted scene through the vehicle-mounted MR device, providing a new and more intuitive interaction mode for a user to control the vehicle, and improving operation intuitiveness and immersive interaction experience of the user.

[0035] Please refer to Figure 3 , Figure 3 A flowchart of a vehicle control method provided by an embodiment of the present application is shown. The method is applied to the electronic device described above, and will be described in detail below with reference to the flowchart shown in Figure 3 The vehicle control method can specifically include the following steps: Step S201: In a case where the MR device and the vehicle establish a communication connection, internal space information of the vehicle is acquired.

[0036] In some embodiments, the MR device can establish a communication connection with the vehicle in a case where the vehicle is powered on, or in a case where the vehicle runs an MR vehicle control application. The MR device and the vehicle can establish the communication connection in a wireless or wired manner. For example, the MR device and the vehicle can establish the communication connection through WiFi, zigbee, Bluetooth, USB harness connection, etc. The MR device can acquire the internal space information of the vehicle in a case where the MR device and the vehicle establish the communication connection. Alternatively, the MR device can acquire the internal space information of the vehicle from the vehicle, or from an associated cloud or electronic device. In this embodiment, the manner in which the MR device acquires the internal space information of the vehicle is not limited.

[0037] As an implementable manner, the MR device can establish a communication connection with the vehicle in a case where the vehicle is powered on, and can acquire the vehicle model of the vehicle based on the communication connection, and acquire three-dimensional calibration information of the internal space corresponding to the vehicle model as the internal space information of the vehicle according to the vehicle model.

[0038] The MR device can obtain the internal space information corresponding to the vehicle model from an associated cloud or electronic device. Alternatively, the MR device can also pre-store internal space information corresponding to a plurality of vehicle models, and obtain the internal space information corresponding to the vehicle after obtaining the vehicle model of the vehicle connected via communication.

[0039] Step S202: generating the virtual space model based on the internal space information.

[0040] In some embodiments, after the MR device obtains the internal space information corresponding to the vehicle, the MR device can generate a virtual space model corresponding to the vehicle based on the internal space information.

[0041] Step S203: obtaining motion data corresponding to the vehicle.

[0042] In some embodiments, after the MR device obtains the virtual space model corresponding to the vehicle, the MR device can obtain motion data corresponding to the vehicle. The motion data includes but is not limited to vehicle motion state information collected by an inertial measurement unit in the vehicle and direction information of vehicle motion, etc. The inertial measurement unit of the vehicle can integrate an accelerometer, a gyroscope and a magnetometer, and can provide motion state information and direction information of the vehicle. The inertial measurement unit of the vehicle can communicate with an infotainment system of the vehicle through a CAN bus network of the vehicle, and can send the motion state information and the direction information of the vehicle to the MR device as the motion data of the vehicle through the infotainment system.

[0043] The motion data corresponding to the vehicle can include acceleration, speed, heading angle, etc.

[0044] Step S204: updating the virtual space model according to the motion data.

[0045] In some embodiments, after the MR device obtains the motion data corresponding to the vehicle, the MR device can update the virtual space model corresponding to the vehicle according to the motion data. For example, the MR device can rotate a virtual point position corresponding to a specific point in the virtual space model and the internal space of the vehicle according to the heading angle of the vehicle, so as to position the virtual point position to the specific point in the actual internal space of the vehicle. The specific point includes but is not limited to a center point of a steering wheel of the vehicle, a center point of a wireless charging plate of the vehicle, a center point of an air outlet of an air conditioner of the vehicle, and a center point of a microphone array of the vehicle, etc.

[0046] The MR device can rotate, translate, or the like the virtual point position corresponding to the specific point of the vehicle interior space in the virtual space model based on the motion data corresponding to the vehicle, update the virtual space model, and display the updated virtual space model, so as to realize matching of the virtual point position and the position of the specific point of the actual vehicle interior space, improve the effectiveness, authenticity, and naturalness of the three-dimensional interactive space of the vehicle MR device in expanding the vehicle scene, and improve the intuitive operation of the user in controlling the vehicle and the immersive interactive experience.

[0047] Step S205: The mixed reality (MR) device displays a virtual space model corresponding to the vehicle, which is generated according to interior space information of the vehicle.

[0048] For specific descriptions of step S205, refer to the description of step S110 above, which will not be elaborated here.

[0049] Step S206: Generating the virtual control panel based on the control information.

[0050] In some embodiments, after the MR device obtains the control information of the target device in the vehicle, the MR device can generate a virtual control panel corresponding to the target device based on the control information. The graphical interface corresponding to the virtual control panel can be the same as the control interface of the target device displayed on the display screen of the vehicle. The virtual control panel can also correspond to a digital twin model of the console of the vehicle. The virtual control panel can also be a graphical interface for adjusting various parameters of the target device.

[0051] Step S207: Displaying the virtual control panel on the seat backrest in the virtual space model.

[0052] In some embodiments, after the MR device generates the virtual control panel corresponding to the target device, the MR device can display the virtual control panel in the virtual space model. For example, the MR device can display the virtual control panel on the seat backrest corresponding to the vehicle in the virtual space model, or display the virtual control panel on the window in the virtual space model, or display the virtual control panel on the roof corresponding to the vehicle in the virtual space model. In this embodiment, the position of the virtual control panel displayed by the MR device in the virtual space model is not limited.

[0053] As an implementable manner, considering that the intuitiveness and feedback of the voice vehicle control interaction for the rear passengers are insufficient, in the embodiment, the MR device can display a virtual control panel on the seat backrest of the vehicle in the virtual space model, thereby improving the intuitive feeling and immersive interaction experience of the rear passengers for the vehicle control process. Meanwhile, through the MR display and operation capability, the three-dimensional interaction space of the vehicle scene is expanded, a new vehicle control interaction manner is provided, and the user experience is improved. In addition, by displaying the control interface of the vehicle and other vehicle content in the virtual space model through the MR device, the vehicle control display screen is avoided from being installed at different positions of the vehicle for the convenience of the user controlling the vehicle. In this way, the user can accurately control the vehicle, and the cost of vehicle control is reduced.

[0054] Step S208: Based on the image data collected by the visual sensor on the MR device, gesture information in the real scene is recognized.

[0055] In some embodiments, a visual sensor is arranged on the MR device; wherein the MR device can perform image collection through the visual sensor in the process of displaying the virtual control panel corresponding to the target device in the virtual space model. The MR device can recognize gesture information in the real scene based on the image data collected by the visual sensor.

[0056] The MR device can perceive the user's hand through the visual sensor, and can obtain gesture information corresponding to the perceived user's hand information. The user perceived by the sensor includes but is not limited to the rear passengers, the co-pilot passengers, the driver, etc. of the vehicle. The MR device can determine the gesture type of the user based on the gesture information, such as twisting a knob, touching a button, etc.

[0057] Step S209: If the gesture information meets the preset condition, it is determined that the control operation for the virtual control panel is obtained.

[0058] In some embodiments, after the MR device recognizes the gesture information of the user in the real scene, the MR device can compare the gesture information with the preset condition, and can determine whether the control operation for the virtual panel is received according to the comparison result. For example, if it is determined that the gesture information represents twisting a virtual button on the virtual control panel, it is determined that the gesture information meets the preset condition; if it is determined that the gesture information represents touching a button on the virtual control panel, it is determined that the gesture information meets the preset condition.

[0059] If the MR device determines that the gesture information meets the preset condition, it can be determined that the control operation for the virtual control panel is obtained. The MR device can further determine the control parameter of the target device input by the user through the virtual control panel according to the control operation. Optionally, the MR device can determine the type of the control parameter input by the user from the virtual control panel according to the gesture type corresponding to the control operation. For example, if the gesture type of the control operation is twisting a knob, the MR device can determine that the user inputs the control parameter through the knob on the control panel, and can determine the type of the control parameter according to the knob, and can determine the size of the control parameter according to the degree of the twisted knob corresponding to the control operation. For example, if the gesture type of the control operation is touching a button, the MR device can determine that the user inputs the control parameter through the button on the control panel, and can determine the type of the control parameter according to the button, and can determine the size of the control parameter according to the number of times of touching the button corresponding to the control operation. The types of the control parameters corresponding to different target devices are different. For example, the target device is an air conditioner, and the types of the control parameters corresponding to the air conditioner include, but are not limited to, a control parameter of air temperature adjustment, a control parameter of air blowing mode, a control parameter of air volume, and a control parameter of air direction, etc. For example, the target device is in-vehicle lighting, and the types of the control parameters corresponding to the in-vehicle lighting include, but are not limited to, a control parameter of light intensity, a control parameter of light color, and a control parameter of light bulb quantity, etc. Step S210: If the control operation for the virtual control panel is obtained, the control parameter is sent to the vehicle, so that the vehicle controls the target device based on the control parameter.

[0060] For specific description of step S210, please refer to the description of step S130 above, which will not be described in detail here.

[0061] Step S211: The control result of the target device returned by the vehicle is obtained, and the control result is displayed on the virtual control panel.

[0062] In some embodiments, after the MR device obtains the control parameter corresponding to the target device, the MR device can send the control parameter to the vehicle, so that the vehicle controls the target device based on the control parameter. After the vehicle controls the target device based on the control parameter, the vehicle can return the control result of the target device to the MR device. Accordingly, the MR device can obtain the control result of the target device returned by the vehicle, and can display the control result on the virtual control panel.

[0063] For example, please refer to Figure 4Fig. 1 shows a flowchart of a method for controlling an air conditioner of a vehicle according to an embodiment of the present application. In the example of controlling an air conditioner of a vehicle, the MR device can create a communication service module for communicating with an IDC module of the vehicle when the MR vehicle control APP is started and the vehicle is powered on, and can display a virtual space model corresponding to the vehicle according to the internal space information of the vehicle corresponding to the vehicle model connected by communication, and can display a virtual control panel corresponding to the air conditioner in the virtual space model when the control information of the air conditioner in the vehicle is obtained, and can activate a gesture recognition module in the MR device. The MR device can obtain a control operation of adjusting the air volume of the air conditioner by the user based on the MR vehicle control APP based on the gesture recognition module, and can determine the control parameters of the air conditioner according to the control operation. The MR device can send the control parameters to the IDC module of the vehicle through the communication service module between the MR device and the vehicle; correspondingly, the IDC module of the vehicle can send the control parameters to the HVAC module of the vehicle, so that the HVAC module adjusts the air volume of the air conditioner based on the control parameters, and the IDC module of the vehicle feeds back the result of the successful adjustment of the air volume of the air conditioner to the MR device through the communication service module between the MR device and the vehicle. The MR device can display the change of the air volume value of the air conditioner on the virtual control panel of the air conditioner based on the result of the successful adjustment when the MR vehicle control APP is running. Thus, the three-dimensional interactive space of the vehicle scene is expanded by the vehicle-mounted MR device, a new and more intuitive interaction method is provided for the user to control the vehicle, the operation intuitiveness and immersive interaction experience of the user are improved, the accuracy of the user in controlling the air conditioner of the vehicle is improved, and compared with the voice interaction method, the time for controlling the air conditioner of the vehicle is shortened and the user experience is improved.

[0064] The vehicle control method provided by the embodiment of the present application can expand the three-dimensional interactive space of the vehicle scene by the vehicle-mounted MR device, provide a new and more intuitive interaction method for the user to control the vehicle, improve the operation intuitiveness and immersive interaction experience of the user, improve the accuracy of the user in controlling the air conditioner of the vehicle, shorten the time for controlling the air conditioner of the vehicle compared with the voice interaction method, and improve the user experience. Figure 1 As shown in the vehicle control method, the embodiment of the present application can also obtain the internal space information of the vehicle before the mixed reality MR device displays the virtual space model corresponding to the vehicle, and generate the virtual space model based on the internal space information, so that the vehicle-mounted MR virtual space model corresponding to the vehicle is established by the MR device based on the pre-recorded internal space information of the vehicle, thereby providing an intuitive and natural vehicle scene for the user.

[0065] Meanwhile, the embodiment can also obtain the motion data corresponding to the vehicle before the mixed reality MR device displays the virtual space model corresponding to the vehicle, and update the virtual space model according to the motion data, so that the virtual space model is positioned to a position matching the actual vehicle space for display according to the motion data of the vehicle, thereby improving the authenticity and naturalness of the three-dimensional interactive space of the vehicle scene expanded by the vehicle-mounted MR device, and improving the immersive interaction experience of the user.

[0066] In addition, this embodiment can also generate a virtual control panel based on control information; the virtual control panel is displayed on the seat back in the virtual space model, thereby generating the virtual control panel corresponding to the target device based on the current control information of the target device, which improves the accuracy of the user's vehicle control, and the display of the virtual control panel on the seat back in the virtual space model enhances the intuitiveness of the rear passengers' vehicle control operation and the immersive interactive experience.

[0067] In addition, this embodiment can also send control parameters to the vehicle if a control operation for the virtual control panel is obtained, so that the vehicle can control the target device accordingly based on the control parameters. Before the vehicle controls the target device, it can identify gesture information in the real scene based on the image data collected by the visual sensor on the MR device. If the gesture information meets the preset conditions, it is determined that a control operation for the virtual control panel has been obtained, so that the user can adjust the control parameters of the target device by gesture. Compared with the voice vehicle control interaction method, the vehicle control time is shortened and the user experience is improved.

[0068] In addition, this embodiment can also send control parameters to the vehicle if a control operation for the virtual control panel is obtained, so that the vehicle can control the target device accordingly based on the control parameters. After that, the control result of the target device returned by the vehicle is obtained and displayed on the virtual control panel. Thus, the vehicle control result is displayed on the virtual control panel, which improves the intuitiveness of vehicle control compared to voice vehicle control interaction and makes up for the lack of vehicle control feedback.

[0069] Please see Figure 5 , Figure 5 A block diagram of a vehicle control device according to an embodiment of this application is shown. This vehicle control device 200 is applied to the aforementioned electronic device, and will be discussed below. Figure 5 The process shown is described in detail. The vehicle control device 200 includes: a space model display module 210, a control panel display module 220, and an equipment control module 230, wherein: The spatial model display module 210 is used to display a virtual spatial model corresponding to the vehicle in a mixed reality (MR) device. The virtual spatial model is generated based on the vehicle's internal spatial information.

[0070] The control panel display module 220 is used to display the virtual control panel of the target device pair in the virtual space model if control information of the target device in the vehicle is obtained.

[0071] The device control module 230 is configured to, if a control operation for the virtual control panel is acquired, send a control parameter to the vehicle to enable the vehicle to perform corresponding control on the target device based on the control parameter, where the control parameter is determined according to the control operation.

[0072] Further, before the mixed reality (MR) device displays the virtual space model corresponding to the vehicle, the vehicle control apparatus 200 can further include an internal space information acquisition unit and a space model generation unit, where: The internal space information acquisition unit is configured to, if the MR device establishes a communication connection with the vehicle, acquire internal space information of the vehicle.

[0073] The space model generation unit is configured to generate the virtual space model based on the internal space information.

[0074] Further, the internal space information acquisition unit can include a communication connection establishment unit, a vehicle model acquisition unit, and an internal space information acquisition subunit, where: The communication connection establishment unit is configured to, if the vehicle is powered on, establish a communication connection with the vehicle.

[0075] The vehicle model acquisition unit is configured to acquire a vehicle model of the vehicle based on the communication connection.

[0076] The internal space information acquisition subunit is configured to acquire internal space information corresponding to the vehicle according to the vehicle model.

[0077] Further, before the mixed reality (MR) device displays the virtual space model corresponding to the vehicle, the vehicle control apparatus 200 can further include a motion data acquisition unit and a space model update unit, where: The motion data acquisition unit is configured to acquire motion data corresponding to the vehicle.

[0078] The space model update unit is configured to update the virtual space model according to the motion data.

[0079] Further, the control panel display module 220 can include a control panel generation unit and a control panel display subunit, where: The control panel generation unit is configured to generate the virtual control panel based on the control information.

[0080] The control panel display subunit is configured to display the virtual control panel on a seat back in the virtual space model.

[0081] Further, before the vehicle control apparatus 200 sends the control parameter to the vehicle to make the vehicle perform corresponding control on the target device based on the control parameter if the control operation for the virtual control panel is acquired, the vehicle control apparatus 200 can further include a gesture information identifying unit and a control operation determining unit, wherein: The gesture information identifying unit is configured to identify gesture information in a real scene based on image data collected by a visual sensor on the MR device.

[0082] The control operation determining unit is configured to determine that the control operation for the virtual control panel is acquired if the gesture information meets a preset condition.

[0083] Further, after the vehicle control apparatus 200 sends the control parameter to the vehicle to make the vehicle perform corresponding control on the target device based on the control parameter if the control operation for the virtual control panel is acquired, the vehicle control apparatus 200 can further include a control result displaying unit, wherein: The control result displaying unit is configured to acquire a control result of the target device returned by the vehicle, and display the control result on the virtual control panel.

[0084] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and module can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0085] In several embodiments provided in the present application, the coupling between the modules can be electrical, mechanical or other forms of coupling.

[0086] In addition, each functional module in each embodiment of the present application can be integrated in one processing module, or each module can exist physically independently, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module.

[0087] Please refer to Figure 6 which shows a structural block diagram of an electronic device provided in an embodiment of the present application. The electronic device 100 can be an MR device, a vehicle, a vehicle-mounted terminal, a server, a computer or the like device with processing capability. The electronic device 100 in the present application can include one or more of the following components: a processor 110, a memory 120 and one or more application programs, wherein the one or more application programs can be stored in the memory 120 and configured to be executed by the one or more processors 110, and the one or more programs are configured to perform the method described in the foregoing method embodiments.

[0088] The processor 110 can include one or more processing cores. The processor 110 connects various parts within the vehicle 100 by various interfaces and lines, performs various functions of the vehicle 100 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Alternatively, the processor 110 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 110 can be integrated with a combination of one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes operating systems, user interfaces, and application programs; the GPU is responsible for rendering and drawing the content to be displayed; and the modem is used for processing wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 110, but can be implemented separately by a communication chip.

[0089] The memory 120 can include a random access memory (RAM) and can also include a read-only memory (ROM). The memory 120 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing each method embodiment described below, etc. The data storage area can also store data created by the electronic device 100 in use (such as a phone book, audio and video data, chat record data, etc.).

[0090] In this embodiment, the computer-readable medium stores program codes, which can be called and executed by the processor to perform the methods described in the above method embodiments.

[0091] The computer-readable storage medium can be an electronic, magnetic, optical, or other physical storage device that stores executable computer program code. The computer-readable storage medium can alternatively or additionally include a non-transitory computer-readable storage medium. The computer-readable storage medium has a storage space for storing program code that executes any of the method steps described above. The program code can be read from or written to one or more computer program products. The program code can be compressed, for example, in a suitable form.

[0092] In the present application, multiple refers to two or more.

[0093] In the present application, unless specifically limited otherwise, the terms "mounting", "connected", "connection" should be understood broadly, for example, can be fixedly connected, can be detachably connected, or integrally connected; can be mechanically connected, can be electrically connected; can be directly connected, can be indirectly connected through an intermediate medium, can be internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0094] The terms "first", "second", "third", "fourth" and the like (if any) in the present application are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0095] In the present application, the term "and / or" is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in the present application generally represents an "or" relationship between the front and rear associated objects.

[0096] If not specifically stated, all steps of the present application can be performed in sequence or randomly. For example, the method comprises steps A and B, which means that the method can comprise steps A and B performed in sequence, or steps B and A performed in sequence. For example, the method can further comprise step C, which means that step C can be added to the method in any order, for example, the method can comprise steps A, B and C, or steps A, C and B, or steps C, A and B, etc.

[0097] The above only describes the preferred embodiments of the present application and does not limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A vehicle control method characterized by, The method comprises: The mixed reality MR device displays a virtual space model corresponding to the vehicle, the virtual space model being generated according to internal space information of the vehicle; If control information of a target device in the vehicle is acquired, a virtual control panel of the target device pair is displayed in the virtual space model; If a control operation for the virtual control panel is acquired, a control parameter is sent to the vehicle, so that the vehicle controls the target device based on the control parameter, and the control parameter is determined according to the control operation.

2. The method of claim 1, wherein, Before the mixed reality MR device displays the virtual space model corresponding to the vehicle, the method further comprises: In the case that the MR device establishes a communication connection with the vehicle, the internal space information of the vehicle is acquired; The virtual space model is generated based on the internal space information.

3. The method of claim 2, wherein, The internal space information of the vehicle is acquired in the case that the MR device establishes a communication connection with the vehicle, comprising: In the case that the vehicle is powered on, a communication connection is established with the vehicle; The vehicle model of the vehicle is acquired based on the communication connection; The internal space information corresponding to the vehicle is acquired according to the vehicle model.

4. The method of claim 1, wherein, Before the mixed reality MR device displays the virtual space model corresponding to the vehicle, the method further comprises: Motion data corresponding to the vehicle is acquired; The virtual space model is updated according to the motion data.

5. The method according to any one of claims 1 to 4, characterized in that, The virtual control panel of the target device pair is displayed in the virtual space model, comprising: The virtual control panel is generated based on the control information; The virtual control panel is displayed on the seat back in the virtual space model.

6. The method according to any one of claims 1 to 4, characterized in that, Before the control parameter is sent to the vehicle if the control operation for the virtual control panel is acquired, so that the vehicle controls the target device based on the control parameter, the method further comprises: Gesture information in a real scene is identified based on image data collected by a visual sensor on the MR device; If the gesture information meets a preset condition, it is determined that the control operation for the virtual control panel is acquired.

7. The method according to any one of claims 1 to 4, characterized in that, After the control parameter is sent to the vehicle if the control operation for the virtual control panel is acquired, so that the vehicle controls the target device based on the control parameter, the method further comprises: The control result of the target device returned by the vehicle is acquired, and the control result is displayed on the virtual control panel.

8. A vehicle control device characterized by comprising: The device comprises: A space model display module for a mixed reality MR device to display a virtual space model corresponding to a vehicle, the virtual space model being generated according to internal space information of the vehicle; A control panel display module for displaying a virtual control panel of a target device pair in the virtual space model if control information of a target device in the vehicle is acquired; The device control module is configured to, if a control operation for the virtual control panel is acquired, send a control parameter to the vehicle to enable the vehicle to perform corresponding control on the target device based on the control parameter, the control parameter being determined according to the control operation.

9. An electronic device, comprising: Comprise: one or more processors; a memory; one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to perform the method of any one of claims 1-7.

10. A computer readable storage medium, characterized in that, The computer readable storage medium stores program code, and the program code can be called and executed by the processor to perform the method of any one of claims 1-7.

Images