Display method, system, device and equipment based on head-mounted display equipment

Abstract

The invention discloses a display method, system, device and equipment based on head-mounted display equipment, and belongs to the technical field of computers. The method is applied to a vehicle-mounted host in a vehicle, the vehicle-mounted host is connected with at least one camera on the vehicle, and the method comprises the following steps: establishing communication connection with at least one head-mounted display device; receiving a display request sent by the head-mounted display device; according to the content type corresponding to the display request, display data corresponding to the content type is generated based on a camera picture collected by at least one camera; and sending display data corresponding to the content type to the head-mounted display device, wherein the display data is used for displaying the to-be-displayed content corresponding to the content type. According to the method and the device, the interaction process between the vehicle and the passenger can be realized based on the camera picture acquired by the camera on the vehicle, and a novel interaction mode is provided for the passenger during travel.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a display method, system, device and equipment based on a head-mounted display device. Background Art

[0002] Providing display devices for passengers in transportation and providing display content to the passengers through the display devices helps to eliminate boredom and dullness of passengers during travel.

[0003] In the related art, the display devices provided in the vehicles provide the passengers with display contents which are usually pre-stored in the display devices, for example, pre-stored movies, music clips, variety shows, etc. Passengers can order the pre-stored contents in the display devices according to their own needs.

[0004] By providing display content to passengers of a vehicle in the above manner to achieve interaction between the vehicle and the passengers, there is a problem that the interaction method between the vehicle and the passengers is relatively single. Summary of the invention

[0005] The present application provides a display method, system, device and equipment based on a head-mounted display device, which can enrich the interaction between transportation tools and passengers. The technical solution is as follows:

[0006] According to one aspect of the present application, a display method based on a head mounted display device is provided, the method being applied to a vehicle-mounted host in a vehicle, the vehicle-mounted host being connected to at least one camera on the vehicle, the method comprising:

[0007] establishing a communication connection with at least one head mounted display device;

[0008] receiving a display request sent by the head-mounted display device, where different display requests are used to request different contents to be displayed for the head-mounted display device, and the different display requests are triggered by different human-computer interaction operations in the head-mounted display device;

[0009] According to the content type corresponding to the display request, based on the camera image captured by the at least one camera, generating display data corresponding to the content type;

[0010] Sending display data corresponding to the content type to the head mounted display device, wherein the display data is used to display the to-be-displayed content corresponding to the content type.

[0011] According to another aspect of the present application, a display method based on a head-mounted display device is provided, the method being applied to the head-mounted display device, the method comprising:

[0012] Establishing a communication connection with a vehicle-mounted host in a vehicle, wherein the vehicle-mounted host is connected to at least one camera on the vehicle;

[0013] Sending a display request to the vehicle-mounted host, where different display requests are used to request different contents to be displayed for the head-mounted display device, and the different display requests are triggered by different human-computer interaction operations in the head-mounted display device;

[0014] receiving display data sent by the vehicle-mounted host, wherein the display data is generated by the vehicle-mounted host according to the content type corresponding to the display request and based on the camera image captured by the at least one camera;

[0015] The content to be displayed corresponding to the content type is displayed according to the display data.

[0016] According to another aspect of the present application, a display system is provided, the display system comprising a vehicle-mounted host in a vehicle and at least one head-mounted display device, the vehicle-mounted host being connected to at least one camera on the vehicle;

[0017] The vehicle-mounted host is used to establish a communication connection with the at least one head-mounted display device;

[0018] The head-mounted display device is used to send a display request to the vehicle-mounted host, where different display requests are used to request different contents to be displayed for the head-mounted display device, and the different display requests are triggered by different human-computer interaction operations in the head-mounted display device;

[0019] The vehicle-mounted host is further configured to generate display data corresponding to the content type based on the camera image captured by the at least one camera according to the content type corresponding to the display request;

[0020] The vehicle-mounted host is further used to send the display data to the head-mounted display device;

[0021] The head mounted display device is further configured to display the to-be-displayed content corresponding to the content type according to the display data.

[0022] According to another aspect of the present application, a display device is provided, the device being applied to a vehicle-mounted host in a vehicle, the vehicle-mounted host being connected to at least one camera on the vehicle, the device comprising:

[0023] A communication module, configured to establish a communication connection with at least one head mounted display device;

[0024] The communication module is further used to receive a display request sent by the head-mounted display device, where different display requests are used to request different contents to be displayed for the head-mounted display device, and the different display requests are triggered by different human-computer interaction operations in the head-mounted display device;

[0025] A generating module, configured to generate display data corresponding to the content type according to the content type corresponding to the display request and based on the camera image captured by the at least one camera;

[0026] The communication module is further used to send display data corresponding to the content type to the head mounted display device, where the display data is used to display the to-be-displayed content corresponding to the content type.

[0027] In an optional design, the different display requests are triggered by triggering operations on different function controls, and the function controls are displayed in the user interface of the head-mounted display device; the generation module is used to determine the content type corresponding to the display request according to the request parameters of the display request, and the request parameters correspond to the function controls that trigger the display request; according to the content type, the display data is generated based on the camera image captured by the at least one camera.

[0028] In an optional design, the at least one camera is used to capture images of the surrounding environment of the vehicle; the generation module is used to generate first display data based on the camera image captured by the first camera when the content type is a first content type, and the display request corresponding to the first content type is triggered by a first function control in the function controls; wherein the first camera includes the camera among the at least one camera that corresponds to the perspective of the driver of the vehicle.

[0029] In an optional design, the at least one camera is used to capture images of the surrounding environment of the vehicle; the generation module is used to generate second display data based on the camera image captured in real time by the second camera when the content type is a second content type, and the display request corresponding to the second content type is triggered by a second function control in the function control; wherein the second camera is determined in the at least one camera based on the sitting position, facing direction, and at least one of the first selection operations triggered in the head-mounted display device of the wearer of the head-mounted display device.

[0030] In an optional design, the at least one camera is used to capture images of the surrounding environment of the vehicle; the generation module is used to generate third display data based on the camera images captured by the third camera within a historical period when the content type is a third content type, and the display request corresponding to the third content type is triggered by a third function control in the function control; wherein the third camera is determined in the at least one camera based on a seating position, a facing direction, and at least one of a second selection operation triggered in the head-mounted display device of a wearer of the head-mounted display device; the historical period is determined based on the third selection operation triggered in the head-mounted display device.

[0031] In an optional design, the at least one camera is used to capture images of the surrounding environment of the vehicle; the generation module is used to generate a game scene screen based on the camera screen captured by a fourth camera when the content type is a fourth content type, and the display request corresponding to the fourth content type is triggered by a fourth function control in the function control; a game screen is generated based on the game scene screen; and fourth display data is generated based on the game screen; wherein the fourth camera includes a camera among the at least one camera used to capture images of the environment in front of the vehicle.

[0032] In an optional design, there are one or more game characters corresponding to the head-mounted display device in the game screen; the device also includes: the communication module, which is used to receive the control operation of the head-mounted display device on the game character during the process of the head-mounted display device displaying the game screen; the update module, which is used to control the behavior of the game character in the game screen according to the control operation to update the fourth display data; the communication module, which is used to send the updated fourth display data to the head-mounted display device.

[0033] In an optional design, the communication module is used to send fifth display data to the head-mounted display device when the content type is a fifth content type, and the display request corresponding to the fifth content type is triggered by a fifth function control in the function control; wherein the fifth display data is used to display an environmental picture of a target driving route, the target driving route is different from the current driving route of the vehicle, and the fifth display data includes display data pre-stored in the vehicle-mounted host and at least one of the display data saved by the vehicle-mounted host during the vehicle's driving on a historical driving route.

[0034] In an optional design, the vehicle-mounted host establishes a communication connection with at least two mobile devices, each mobile device includes the head-mounted display device or the display device, and the display device includes a device other than the head-mounted display device; the communication module is used to send the sixth display data of the current display content of the second mobile device to the first mobile device when the content type corresponding to the display request of the first mobile device is the sixth content type, and the display request corresponding to the sixth content type is triggered by the sixth function control in the function control; wherein the first mobile device and the second mobile device are different devices among the at least two mobile devices, the first mobile device and the second mobile device belong to the same network room, and the network room is established according to the room creation operation triggered by the mobile device.

[0035] In an optional design, the communication module is used to send display data for displaying the user interface to the head-mounted display device.

[0036] In an optional design, the generation module is used to generate display data of a panoramic picture corresponding to the content type for the VR device when the head-mounted display device is a virtual reality (VR) device and the content type is related to the camera images captured by multiple cameras of the at least one camera; wherein the panoramic picture is obtained by splicing the camera images captured by the multiple cameras.

[0037] In an optional design, the head-mounted display device includes an augmented reality (AR) device, in which an integrated camera is provided; the device also includes: the communication module, which is used to receive an object recognition request sent by the AR device, and the object recognition request carries a captured image of the integrated camera; the recognition module, which is used to obtain a recognition result of the captured image by identifying an object in the captured image; and the communication module, which is used to send the recognition result to the AR device so that the recognition result can be displayed through the AR device.

[0038] According to another aspect of the present application, a head-mounted display device is provided, the head-mounted display device comprising:

[0039] A communication module, used to establish a communication connection with a vehicle-mounted host in a vehicle, wherein the vehicle-mounted host is connected to at least one camera on the vehicle;

[0040] The communication module is further used to send a display request to the vehicle-mounted host, where different display requests are used to request different contents to be displayed for the head-mounted display device, and the different display requests are triggered by different human-computer interaction operations in the head-mounted display device;

[0041] The communication module is further used to receive display data sent by the vehicle-mounted host, where the display data is generated by the vehicle-mounted host according to the content type corresponding to the display request and based on the camera image captured by the at least one camera;

[0042] A display module is used to display the to-be-displayed content corresponding to the content type according to the display data.

[0043] According to another aspect of the present application, a vehicle-mounted host is provided, which includes a processor and a memory, wherein at least one program is stored in the memory, and the at least one program is loaded and executed by the processor to implement the display method based on the head-mounted display device as described above.

[0044] According to another aspect of the present application, a head-mounted display device is provided, comprising a processor and a memory, wherein at least one program is stored in the memory, and the at least one program is loaded and executed by the processor to implement the display method based on the head-mounted display device as described above.

[0045] According to another aspect of the present application, a computer-readable storage medium is provided, in which at least one program is stored. The at least one program is loaded and executed by a processor to implement the display method based on a head-mounted display device as described above.

[0046] According to another aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the display method based on the head-mounted display device provided in the above aspect.

[0047] The beneficial effects of the technical solution provided by this application include at least:

[0048] The on-board host in the vehicle generates display data corresponding to the content type requested by the head-mounted display device based on the camera image captured by at least one camera on the vehicle, so that the head-mounted display device can display the corresponding display content according to the display data. Since the display content of the head-mounted display device is generated based on the camera image captured by the camera on the vehicle, the interaction process between the vehicle and the passengers can be realized based on the camera image captured by the camera on the vehicle, and different display content can be provided to the head-mounted display device according to the display request of the head-mounted display device, which not only enriches the display content provided to the passengers, but also provides a novel interaction method for passengers during their travels, enriching the interaction method between the vehicle and the passengers. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 is a schematic diagram of a display system provided by an exemplary embodiment of the present application;

[0051] Figure 2 is a flowchart of a display method based on a head-mounted display device provided by an exemplary embodiment of the present application;

[0052] Figure 3 is a flowchart of a display method based on a head-mounted display device provided by an exemplary embodiment of the present application;

[0053] Figure 4 is a flowchart of a display method based on a head-mounted display device provided by an exemplary embodiment of the present application;

[0054] Figure 5 is a schematic diagram of the setting position of a camera provided by an exemplary embodiment of the present application;

[0055] Figure 6 is a schematic diagram of a process of sending display data provided by an exemplary embodiment of the present application;

[0056] Figure 7 is a schematic diagram of a display process provided by an exemplary embodiment of the present application;

[0057] Figure 8 is a schematic diagram of a display system provided by an exemplary embodiment of the present application;

[0058] Fig. 9 is a schematic structural diagram of a display device provided by an exemplary embodiment of the present application;

[0059] Fig.10 is a schematic structural diagram of a display device provided by an exemplary embodiment of the present application;

[0060] Fig.11 is a schematic structural diagram of a display device provided by an exemplary embodiment of the present application;

[0061] Fig.12 is a schematic structural diagram of a head mounted display device provided by an exemplary embodiment of the present application;

[0062] Fig.13 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application.

[0063] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application. DETAILED DESCRIPTION

[0064] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0065] First, the nouns involved in this application are introduced:

[0066] Virtual Reality (VR): VR technology is a computer technology that can create and experience a virtual world. VR technology generates realistic three-dimensional virtual scenes, which can be used to create an immersive experience. In this application, VR technology can be applied to passenger entertainment, navigation, simulated driving and other aspects. Passengers can immerse themselves in virtual entertainment scenes through VR devices and enjoy a richer and more diverse entertainment experience. At the same time, VR technology can also provide passengers with intuitive navigation information, so that they can quickly find their location in an unfamiliar environment, and understand their current location in the entire driving route in real time.

[0067] Augmented Reality (AR): AR technology is a technology that seamlessly integrates virtual information with the real world. AR technology can provide users with a richer visual experience by superimposing virtual elements on the real world. In this application, AR technology can be applied to driving assistance, passenger information display and other aspects. By utilizing AR technology, both the driver and passengers of a vehicle can obtain virtual information around the vehicle, such as road conditions, pedestrians, etc., in real time, avoiding leaving the issue of safe driving entirely to the driver, thereby improving driving safety and accuracy. At the same time, passengers can also obtain more real-time information through AR devices, such as vehicle status, weather conditions, etc., to enhance the riding experience.

[0068] Mixed Reality (MR): MR technology can be understood as a further development of virtual reality technology. MR technology introduces real scene information into the virtual scene to build an interactive feedback information loop between the virtual world, the real world and the user to enhance the user experience. The implementation of MR requires an environment that can interact with various things in the real world. If everything is virtual, it can be understood as VR. If the displayed virtual information can only be simply superimposed on real things, it can be understood as AR. The combination of VR technology and AR technology involved in this application can be understood as MR technology.

[0069] Current transportation tools (such as buses) carry the travel needs of a large number of passengers. Their safety, comfort and information level are crucial to improving the passengers' riding experience. With the continuous development of information technology, the needs of passengers on the road are becoming increasingly diversified, and transportation tools are gradually becoming intelligent and information-based. However, traditional transportation tools mainly focus on basic ride services, such as seat comfort, air conditioning temperature adjustment, etc. The display devices inside the transportation tools, such as car-mounted TVs or fixed-position display screens, can often only provide single, fixed content, which cannot meet the passengers' needs for personalized, real-time interaction and immersive experience. The exploration of improving the riding experience through information technology is still insufficient. Passengers often feel bored and dull during long journeys, lacking diverse entertainment content. At the same time, for passengers who are not familiar with the route, the lack of intuitive navigation information will also increase the inconvenience of their travel. In addition, the driving of current transportation tools depends entirely on the driver. Considering that the driver's personal state may be unstable, there are also certain problems in the safe driving of transportation tools. Therefore, the existing transportation tools still need to be improved in terms of riding experience and safe driving.

[0070] In recent years, the rapid development of VR and AR technologies has brought innovative possibilities to the internal display systems of vehicles. VR technology can provide an immersive virtual environment experience, while AR technology can overlay virtual elements in the real environment to enhance the sense of reality. However, the application of VR and AR technologies in vehicles faces many challenges.

[0071] First, the space inside the vehicle is limited, and the space needs to be used efficiently to arrange the display screen. Through multi-screen display technology (such as the presentation mechanism of the Android system), it is possible to provide multiple display areas in a limited space, and each display area can display different content, thereby meeting the diverse needs of passengers. Secondly, due to the large number of passengers, each passenger has different needs and interests. In order to provide a personalized experience, it is necessary to be able to capture passenger information in real time and adjust the display content based on this information. Multi-camera technology can capture environmental information outside the vehicle and information about passengers inside in all directions, providing a rich data source for the system. Then, real-time and interactivity are also important considerations for VR and AR applications in vehicles. Passengers hope to obtain the required information in a short time and interact with the vehicle system in real time. Therefore, the system needs to have efficient data processing capabilities and fast response speed. However, it is not easy to integrate multi-camera, multi-screen display, VR and AR technology into a vehicle system. It is necessary to solve problems such as the collaborative work between multiple cameras, the efficient processing performance of the processor, the display control of multi-screen display, and the realization of real-time interaction.

[0072] In order to solve the above problems, this application proposes a display method and a corresponding display system for the vehicle-mounted host and head-mounted display device in a vehicle, which captures the external environmental information of the vehicle and the internal passenger information through multiple cameras (cameras), and after image processing, sends the processing results to the respective display screens of multiple devices for display. Each display screen can display different content to meet the personalized needs of passengers. This solves the problems existing in the integration of virtual reality and augmented reality technologies in existing vehicles, provides passengers with a more comfortable and safe riding experience, and provides strong auxiliary support for the driving of vehicles.

[0073] Figure 1 FIG. 1 is a schematic diagram of a display system 100 provided by an exemplary embodiment of the present application. Figure 1 As shown, the display system 100 includes a vehicle (e.g. Figure 1In some embodiments, the vehicle-mounted host 101 and at least one mobile device in the bus (as shown), the mobile device includes at least one of a head-mounted display device and a display device 102. In some embodiments, the head-mounted display device includes at least one of a VR device 103 and an AR device 104, and the display device 102 can be referred to as a non-head-mounted display device, such as a handheld display device. The display device 102 includes devices other than head-mounted display devices, such as smart phones, tablet computers, smart wearable devices, etc., which are not limited in this application. The vehicle-mounted host 101 is connected to at least one camera on the vehicle by wired or wireless means. Optionally, at least one camera is used to capture images of the surrounding environment of the vehicle. When the vehicle-mounted host 101 is turned on, the vehicle-mounted host 101 will establish a communication connection with the above-mentioned mobile device by wired or wireless means, thereby providing the mobile device with at least one of a simulated driving function, a panoramic view of scenery and road conditions, a movie viewing function, a game function, an intelligent recognition function, and a shared screen function.

[0074] For VR device 103: the vehicle host 101 receives the display request sent by VR device 103, thereby determining the content type of the content to be displayed by VR device 103 according to the display request, and sends the corresponding display data to VR device 103. In the case where the display request triggers the simulated driving function, the vehicle host 101 sends the first display data to VR device 103, and the first display data is generated based on the camera picture collected by the first camera of at least one camera. In the case where the display request triggers the scenery and road condition overview function, the vehicle host 101 sends the second display data to VR device 103, and the second display data is generated based on the camera picture collected by the second camera of at least one camera in real time. In the case where the display request triggers the movie viewing function, the vehicle host 101 sends the third display data to VR device 103, and the third display data is generated based on the camera picture collected by the third camera of at least one camera in the historical period; or sends the display data of the environment picture of the target driving route to VR device 103, and the target driving route is different from the current driving route of the vehicle. In the case where the display request triggers the game function, the vehicle host 101 sends fourth display data to the VR device 103, and the fourth display data is generated based on the game screen, which is generated based on the game scene screen, and the game scene screen is generated based on the camera screen captured by the fourth camera in the at least one camera. The first camera, the second camera, and the third camera in the at least one camera are the same as the fourth camera, or partially the same, or completely different. Optionally, in the case where the content type is related to the camera screens captured by multiple cameras, the vehicle host 101 will splice the camera screens captured by multiple cameras into a panoramic screen, thereby sending the display data of the panoramic screen corresponding to the content type to the VR device 103.

[0075] For the AR device 104: the vehicle host 101 will receive the display request sent by the AR device 104, thereby determining the content type of the content to be displayed by the AR device 104 according to the display request, and sending the corresponding display data to the AR device 104. In the case where the display request triggers the simulated driving function, the scenery and road condition overview function, the movie viewing function, and the game function, the process of the vehicle host 101 sending display data to the AR device 104 can refer to the above-mentioned process of sending display data to the VR device 103, and this application will not repeat it. For the triggering of the intelligent recognition function, the vehicle host 101 will receive the object recognition request sent by the AR device 104, and the object recognition request carries the captured image of the integrated camera set in the AR device 104. By identifying the object in the captured image, the vehicle host 101 will obtain the recognition result of the captured image, and then the vehicle host 101 will send the recognition result to the AR device 104 to display the recognition result through the AR device 104. It should be noted that the above-mentioned intelligent recognition function can also be implemented independently by the AR device 104.

[0076] For the process of the display device 102 triggering the function provided by the vehicle host 101, please refer to the above introduction of the VR device 103 and the AR device 104, and this application will not go into details. For the shared screen function: In this case, the vehicle host 101 establishes a communication connection with at least two mobile devices, and the vehicle host 101 will receive a display request sent by the first mobile device. When the content type of the content to be displayed of the first mobile device determined according to the display request is the current display content of the second mobile device, the vehicle host 101 will send the display data of the current display content of the second mobile device to the first mobile device, so that the first mobile device displays the current display content of the second mobile device. Among them, the first mobile device and the second mobile device are different devices among the at least two mobile devices, and the first mobile device and the second mobile device belong to the same network room, and the network room is established according to the room creation operation triggered by the mobile device.

[0077] In some embodiments, different display requests of the mobile device are triggered by different human-computer interaction operations in the mobile device, for example, by triggering operations on different functional controls. The functional controls used to trigger the display request are displayed in the user interface of the mobile device. Optionally, the display data used to display the user interface is provided by the vehicle-mounted host to the mobile device.

[0078] The vehicle-mounted host in the vehicle generates display data corresponding to the content type requested by the mobile device based on the camera image captured by at least one camera on the vehicle, so that the mobile device can display the corresponding display content according to the display data. Since the display content of the mobile device is generated based on the camera image captured by the camera on the vehicle, the interaction process between the vehicle and the passengers can be realized based on the camera image captured by the camera on the vehicle, and different display content can be provided to the mobile device according to the display request of the mobile device, which not only enriches the display content provided to the passengers, but also provides a novel interaction method for passengers during their travels, enriching the interaction method between the vehicle and the passengers.

[0079] Figure 2 FIG. 1 is a flow chart of a display method based on a head mounted display device provided by an exemplary embodiment of the present application. The method can be used for a vehicle-mounted host in a vehicle. Figure 2 As shown, the method includes:

[0080] Step 202: Establish a communication connection with at least one head mounted display device.

[0081] The means of transport includes any means / vehicle for transporting passengers, such as at least one of a coach, a bus, a car, a ship, an airplane, a train, a high-speed rail, and a subway. It should be noted that the embodiments of the present application are mainly introduced by taking the coach as an example. It is understandable that the method provided in the embodiments of the present application is also applicable to means of transport other than coaches.

[0082] Optionally, the vehicle-mounted host being located in the vehicle includes at least one of the following situations: the vehicle-mounted host is deployed in the vehicle, and the vehicle-mounted host is placed in the vehicle. In some embodiments, the vehicle-mounted host is a computer device.

[0083] A head-mounted display device is a display device that is worn on the head of a wearer for use, and the head-mounted display device includes a display screen. In some embodiments, the head-mounted display device includes at least one of a VR device and an AR device, and the embodiments of the present application do not limit the type of head-mounted display device. VR devices may also be referred to as VR glasses, and AR devices may also be referred to as AR glasses. In some embodiments, the head-mounted display device is located in a vehicle, for example, fixed in a vehicle or placed in a vehicle. In some embodiments, the head-mounted display device is located outside the vehicle.

[0084] In some embodiments, the vehicle-mounted host establishes a communication connection with at least one head-mounted display device in a wired or wireless manner. In the case of establishing a communication connection in a wireless manner, the vehicle-mounted host establishes a communication connection with at least one head-mounted display device through at least one of a wireless communication network, a mobile communication network, and other communication networks, and other communication networks include communication networks other than wireless communication networks and mobile communication networks. For example, the vehicle-mounted host establishes a communication connection with at least one head-mounted display device through at least one of a wireless local area network (WLAN), wireless fidelity (WIFI), a fourth generation mobile communication technology (4G), a fifth generation mobile communication technology (5G), and 5G subsequent evolution technologies. The way in which the vehicle-mounted host connects to different head-mounted display devices can be the same or different. In some implementations, the vehicle-mounted host establishes a communication connection with a head-mounted display device installed with an application / client corresponding to the vehicle-mounted host; or, the vehicle-mounted host establishes a communication connection with the head-mounted display device according to a connection relationship pre-set in the vehicle-mounted host.

[0085] In some embodiments, in addition to the head-mounted display device, the vehicle host also supports establishing a communication connection with the display device. The display device includes devices other than the head-mounted display device, which can be called non-head-mounted display devices, such as smart phones, tablet computers, laptops, smart wearable devices, etc. This application does not limit this. The display device includes a display screen. The head-mounted display device and the display device can be collectively referred to as a mobile device.

[0086] Step 204: Receive a display request sent by the head mounted display device.

[0087] Different display requests are used to request different to-be-displayed content for the head-mounted display device, where the to-be-displayed content is the content that the head-mounted display device is waiting to display using display data. Different display requests are triggered by different human-computer interaction operations in the head-mounted display device. Different human-computer interaction operations include at least one of different interaction objects and different interaction methods, for example, different human-computer interaction operations are operations of clicking different icons. In some embodiments, for the head-mounted display device, the human-computer interaction operation is triggered by at least one of an external device (such as a handle) corresponding to the head-mounted display device, and the head-mounted display device tracks the eye sight of the wearing object. For the display device, the human-computer interaction operation is triggered by at least one of touching the display screen of the display device and by an external device corresponding to the display device.

[0088] In some embodiments, different human-computer interaction operations include triggering operations on different functional controls displayed by the head-mounted display device, that is, different display requests are triggered by triggering operations on different functional controls displayed by the head-mounted display device. The triggering operation may be a touch operation, a selection operation, or the like on the functional control. Different functional controls have different display styles, such as different icons or buttons. The functional control used to trigger the display request is displayed in the user interface of the head-mounted display device. Optionally, the display data for displaying the user interface is provided by the vehicle-mounted host to the head-mounted display device. For example, when the vehicle-mounted host establishes a communication connection with the head-mounted display device, the vehicle-mounted host provides the head-mounted display device with display data for displaying the user interface, so that the head-mounted display device displays the user interface, which may be referred to as the main interface. Optionally, different functional controls are all displayed in the user interface.

[0089] Step 206: According to the content type corresponding to the display request, based on the camera images captured by at least one camera, display data corresponding to the content type is generated.

[0090] The vehicle-mounted host is connected to at least one camera on the vehicle. Optionally, at least one camera is disposed on the surface of the vehicle to collect images of the surrounding environment of the vehicle. In some embodiments, each surface around the vehicle is provided with one or more cameras to achieve comprehensive collection of images of the surrounding environment of the vehicle.

[0091] Optionally, the vehicle-mounted host directly generates the camera image captured by at least one camera as display data of the head-mounted display device; or, the vehicle-mounted host processes the camera image captured by at least one camera and generates display data of the head-mounted display device based on the processing result.

[0092] Optionally, the content type is related to the camera picture used in the camera picture captured by at least one camera when the vehicle-mounted host generates display data for the head-mounted display device; and / or, the content type is related to the way in which the camera picture captured by at least one camera is processed when the vehicle-mounted host generates display data for the head-mounted display device. Different display data can be generated for the head-mounted display device through different display requests. For example, when the content type of the content to be displayed is the first content type, the vehicle-mounted host generates display data based on the camera picture captured by the first camera of at least one camera, and the first camera includes at least one camera corresponding to the perspective of the driver of the vehicle. For another example, when the content type of the content to be displayed is the second content type, the vehicle-mounted host generates display data based on the camera picture captured in real time by the second camera of at least one camera, and the second camera belongs to at least one camera, for example, the second camera is selected according to the selection operation in the head-mounted display device.

[0093] Step 208: Send display data corresponding to the content type to the head mounted display device.

[0094] The display data is used by the head mounted display device to display the to-be-displayed content corresponding to the content type. The head mounted display device can display the to-be-displayed content corresponding to the content type by rendering the display data.

[0095] Optionally, in addition to the display data generated based on the camera image, the vehicle-mounted host can also send other display data to the head-mounted display device with which a communication connection is established. For example, the display data of the environmental image of the target driving route is sent to the head-mounted display device, and the target driving route is different from the current driving route of the vehicle. For another example, when the vehicle-mounted host has established a communication connection with at least two head-mounted display devices, the vehicle-mounted host can send the display data of the current display content of the second head-mounted display device to the first head-mounted display device, so as to realize the sharing of the current display content of the second head-mounted display device with the first head-mounted display device. The current display content is the content currently displayed by the second head-mounted display device. The first head-mounted display device and the second head-mounted display device are different head-mounted display devices among the at least two head-mounted display devices. It should be noted that the display data sent to the head-mounted display device in the embodiment of the present application is also applicable to display devices other than the head-mounted display device that have established a communication connection with the vehicle-mounted host.

[0096] In summary, the method provided in this embodiment generates display data corresponding to the content type requested by the head-mounted display device according to the camera image captured by at least one camera on the vehicle by the on-board host in the vehicle, so that the head-mounted display device can display the corresponding display content according to the display data. Since the display content of the head-mounted display device is generated according to the camera image captured by the camera on the vehicle, the interaction process between the vehicle and the passenger can be realized based on the camera image captured by the camera on the vehicle, and different display content is provided to the head-mounted display device according to the display request of the head-mounted display device, which not only enriches the display content provided to the passengers, but also provides a novel interaction method for passengers during their travels, enriching the interaction method between the vehicle and the passengers.

[0097] Figure 3 FIG. 1 is a flow chart of a display method based on a head-mounted display device provided by an exemplary embodiment of the present application. The method can be used for a head-mounted display device. Figure 3 As shown, the method includes:

[0098] Step 302: Establish a communication connection with a vehicle-mounted host in a vehicle.

[0099] The means of transport includes any means / vehicle for transporting passengers, such as at least one of a coach, a bus, a car, a ship, an airplane, a train, a high-speed rail, and a subway. Optionally, the vehicle-mounted host is located in the means of transport, including at least one of the situations that the vehicle-mounted host is deployed in the means of transport and the vehicle-mounted host is placed in the means of transport.

[0100] The vehicle-mounted host is connected to at least one camera on the vehicle. Optionally, at least one camera is arranged on the surface of the vehicle to collect images of the surrounding environment of the vehicle.

[0101] A head-mounted display device is a display device that is worn on the head of a wearer for use, and the head-mounted display device includes a display screen. In some embodiments, the head-mounted display device includes at least one of a VR device and an AR device, and the embodiments of the present application do not limit the type of the head-mounted display device. In some embodiments, the head-mounted display device is located in a vehicle, for example, fixed in the vehicle or placed in the vehicle. In some embodiments, the head-mounted display device is located outside the vehicle.

[0102] In some embodiments, the head mounted display device establishes a communication connection with the vehicle-mounted host in a wired or wireless manner. In the case of establishing a communication connection in a wireless manner, the head mounted display device establishes a communication connection with the vehicle-mounted host in at least one of a wireless communication network, a mobile communication network, and other communication networks, and the other communication networks include communication networks other than the wireless communication network and the mobile communication network.

[0103] Step 304: Send a display request to the vehicle-mounted host.

[0104] Different display requests are used to request different to-be-displayed contents for the head-mounted display device, where the to-be-displayed contents are contents that the head-mounted display device is waiting to display using display data. Different display requests are triggered by different human-computer interaction operations in the head-mounted display device. In some embodiments, the human-computer interaction operation is triggered by at least one of an external device (e.g., a handle) corresponding to the head-mounted display device and the head-mounted display device tracking the eye gaze of the wearer.

[0105] In some embodiments, different human-computer interaction operations include triggering operations on different functional controls displayed by the head-mounted display device. The triggering operation may be a touch operation, a selection operation, etc. on the functional control. Different functional controls have different display styles. The functional control used to trigger the display request is displayed in the user interface of the head-mounted display device. Optionally, the display data for displaying the user interface is provided by the vehicle-mounted host to the head-mounted display device. Optionally, the different functional controls are all displayed in the user interface.

[0106] Step 306: Receive display data sent by the vehicle-mounted host.

[0107] The display data is generated by the vehicle host according to the content type corresponding to the display request and based on the camera image captured by at least one camera. Optionally, the vehicle host directly generates the camera image captured by at least one camera as display data of the head-mounted display device; or, the vehicle host processes the camera image captured by at least one camera and generates display data of the head-mounted display device based on the processing result.

[0108] Optionally, the content type is related to the camera image used in the camera image captured by at least one camera when the vehicle-mounted host generates display data for the head-mounted display device; and / or, the content type is related to the way in which the camera image captured by at least one camera is processed when the vehicle-mounted host generates display data for the head-mounted display device.

[0109] Step 308: Display the content to be displayed corresponding to the content type according to the display data.

[0110] The display data is used by the head-mounted display device to display the content to be displayed corresponding to the content type. The head-mounted display device can display the content to be displayed corresponding to the content type by rendering the display data. In some embodiments, when the head-mounted display device is a VR device, the display content of the head-mounted display device is VR content; when the head-mounted display device is an AR device, the display content of the head-mounted display device is AR content. Among them, VR content can be display content with a display range of more than 120 degrees, such as 360 degrees, and AR content can be display content superimposed on the real world.

[0111] Optionally, in addition to the display data generated based on the camera image, the head-mounted display device can also receive other display data sent by the vehicle-mounted host. For example, the display data of the environmental image of the target driving route sent by the vehicle-mounted host is received, and the target driving route is different from the current driving route of the vehicle. For another example, when the vehicle-mounted host has established a communication connection with at least two head-mounted display devices, the vehicle-mounted host can send the display data of the current display content of the second head-mounted display device to the first head-mounted display device, so as to realize the sharing of the current display content of the second head-mounted display device with the first head-mounted display device. The first head-mounted display device and the second head-mounted display device are different head-mounted display devices among the at least two head-mounted display devices.

[0112] In summary, the method provided in this embodiment generates display data corresponding to the content type requested by the head-mounted display device according to the camera image captured by at least one camera on the vehicle by the on-board host in the vehicle, so that the head-mounted display device can display the corresponding display content according to the display data. Since the display content of the head-mounted display device is generated according to the camera image captured by the camera on the vehicle, the interaction process between the vehicle and the passenger can be realized based on the camera image captured by the camera on the vehicle, and different display content is provided to the head-mounted display device according to the display request of the head-mounted display device, which not only enriches the display content provided to the passengers, but also provides a novel interaction method for passengers during their travels, enriching the interaction method between the vehicle and the passengers.

[0113] Figure 4 FIG. 1 is a flow chart of a display method based on a head mounted display device provided by an exemplary embodiment of the present application. The method can be used for Figure 1 The system shown. Figure 4 As shown, the method includes:

[0114] Step 402: The vehicle-mounted host establishes a communication connection with at least one mobile device.

[0115] The vehicle-mounted host is located in a vehicle. The vehicle includes any vehicle / vehicle for transporting passengers, such as at least one of a coach, a bus, a car, a ship, an airplane, a train, a high-speed rail, and a subway. Optionally, the vehicle-mounted host is located in a vehicle, including at least one of the vehicle-mounted host being deployed in the vehicle and the vehicle-mounted host being placed in the vehicle. In some embodiments, the vehicle-mounted host is a computer device.

[0116] The mobile device includes at least one of a head-mounted display device and a display device. The head-mounted display device is a display device worn on the head of the wearer for use, and the head-mounted display device includes a display screen. In some embodiments, the head-mounted display device includes at least one of a VR device and an AR device, which is not limited in the embodiments of the present application. The display device includes devices other than the head-mounted display device, such as smart phones, tablet computers, laptops, smart wearable devices, etc., which is not limited in the present application, and the display device includes a display screen. In some embodiments, the mobile device is located in a vehicle, such as fixed in the vehicle or placed in the vehicle. In some embodiments, the mobile device is located outside the vehicle.

[0117] In some embodiments, the vehicle-mounted host establishes a communication connection with at least one mobile device via a wired or wireless manner. In the case of establishing a communication connection via a wireless manner, the vehicle-mounted host establishes a communication connection with at least one mobile device via at least one of a wireless communication network, a mobile communication network, and other communication networks, and other communication networks include communication networks other than wireless communication networks and mobile communication networks. The manner in which the vehicle-mounted host connects to different mobile devices may be the same or different. In some implementations, the vehicle-mounted host establishes a communication connection with a mobile device that has an application / client corresponding to the vehicle-mounted host installed; or, the vehicle-mounted host establishes a communication connection with the mobile device based on a connection relationship pre-set in the vehicle-mounted host.

[0118] The vehicle-mounted host is connected to at least one camera on the vehicle. In some embodiments, the vehicle-mounted host is connected to at least one camera by wire or wirelessly. Optionally, at least one camera is set on the surface of the vehicle to collect images of the surrounding environment of the vehicle. In some embodiments, each surface around the vehicle is provided with one or more cameras to achieve comprehensive collection of images of the surrounding environment of the vehicle.

[0119] For example, Figure 5 FIG. 1 is a schematic diagram of the camera installation position provided by an exemplary embodiment of the present application. Figure 5 As shown in Figure , the means of transportation is a bus. Figure 5 As shown in (a), a camera 501 is provided on the front of the bus. Figure 5As shown in (b), cameras 502 and 503 are provided on the right side of the bus. Figure 5 As shown in (c), a camera 504 is provided on the back of the bus. Figure 5 As shown in (d), cameras 505 and 506 are provided on the left side of the bus. It should be noted that for vehicles other than buses, reference can also be made to Figure 5 The camera is set at the position shown, and this embodiment of the present application will not be described in detail.

[0120] In some embodiments, when the vehicle is a subway or a train, the at least one camera on the vehicle includes a camera arranged in front of the head car, a camera arranged behind the tail car, and cameras on both sides of each car. In the case where the vehicle is an airplane, the at least one camera on the vehicle includes a camera on the top of the airplane, a camera at the bottom of the airplane, a camera at the head of the airplane, a camera on the side of the airplane, and a camera on the wing. In some embodiments, when the vehicle is connected to a drone, the at least one camera on the vehicle may also include a camera on a drone connected to the vehicle. In some embodiments, the camera on the vehicle may also include a camera for photographing the blind spot of the vehicle.

[0121] Step 404: The vehicle-mounted host receives a display request sent by the mobile device.

[0122] The display request is used to request the vehicle host to provide display data to the mobile device. The display data is used for the mobile device to display the content to be displayed corresponding to the display data. The content to be displayed is the content that the mobile device is waiting to display using the display data. Different display requests are used to request different content to be displayed for the mobile device.

[0123] The display request is triggered by a human-computer interaction operation in the mobile device, and different display requests are triggered by different human-computer interaction operations in the mobile device. For example, it is triggered by a human-computer interaction operation in a user interface displayed in the mobile device. The display request can also be automatically triggered when the mobile device establishes a communication connection with the vehicle-mounted host. Optionally, the data used to display the user interface in the mobile device is provided by the vehicle-mounted host to the mobile device, for example, when the mobile device establishes a communication connection with the vehicle-mounted host, the vehicle-mounted host automatically provides the data to the mobile device; or, the data used to display the user interface in the mobile device is stored in the mobile device. Different human-computer interaction operations include at least one of different interaction objects and different interaction methods. In some embodiments, for a head-mounted display device, the human-computer interaction operation is triggered by at least one of the two methods of an external device corresponding to the head-mounted display device and the head-mounted display device tracking the eye sight of the wearing object. For a display device, the human-computer interaction operation is triggered by at least one of the two methods of touching the display screen of the display device and by an external device corresponding to the display device.

[0124] In some embodiments, different human-computer interaction operations include triggering operations on different functional controls displayed by the head-mounted display device, that is, different display requests are triggered by triggering operations on different functional controls displayed by the head-mounted display device. The triggering operation may be a touch operation, a selection operation, etc. on the functional control. Different functional controls have different display styles. The functional control for triggering the display request is displayed in a user interface of the head-mounted display device, which may be referred to as a main interface. Optionally, different functional controls are all displayed in the user interface.

[0125] Step 406: The vehicle-mounted host determines the content type of the content to be displayed on the mobile device according to the display request.

[0126] The content type is the type of content to be displayed, and different content types correspond to different content to be displayed. Different mobile devices support the same or different content types for requests. In some embodiments, the display request of the mobile device carries the content type. In some embodiments, the display request of the mobile device carries request parameters, and the on-board host determines the content type corresponding to the display request based on the request parameters of the display request. The request parameters correspond to the function controls that trigger the display request, and different function controls are used to trigger display requests corresponding to different content types. For example, icon 1, icon 2, and icon 3 are displayed in the user interface. By triggering icon 1, the display request corresponding to content type 1 can be triggered; by triggering icon 2, the display request corresponding to content type 2 can be triggered; by triggering icon 3, the display request corresponding to content type 3 can be triggered.

[0127] Step 408: The vehicle-mounted host obtains display data corresponding to the content type of the content to be displayed.

[0128] The display data is used by the mobile device to display the to-be-displayed content corresponding to the display data. The mobile device can display the to-be-displayed content corresponding to the display data by rendering the display data.

[0129] In some embodiments, the vehicle-mounted host generates display data corresponding to the content type based on the camera images captured by all or part of the at least one camera, thereby obtaining display data corresponding to the content type of the content to be displayed. In some embodiments, the vehicle-mounted host obtains display data corresponding to the content type of the content to be displayed from locally stored data. In some embodiments, the vehicle-mounted host obtains display data for sending to a first mobile device as display data for sending to a second mobile device, and the first mobile device and the second mobile device are different mobile devices that have established a communication connection with the vehicle-mounted host.

[0130] Optionally, when the display data is generated by the vehicle-mounted host based on the camera image captured by at least one camera, the content type is related to the camera image used in the camera image captured by at least one camera when the vehicle-mounted host generates display data for the mobile device; and / or, the content type is related to the way in which the camera image captured by at least one camera is processed when the vehicle-mounted host generates display data for the head-mounted display device.

[0131] For the first type of display data:

[0132] When the content type of the content to be displayed is the first content type, the vehicle-mounted host generates first display data based on the camera image captured by the first camera. The display request corresponding to the first content type is triggered by the first function control in the function control displayed by the mobile device. The first camera includes at least one camera connected to the vehicle-mounted host, a camera corresponding to the perspective of the driver of the vehicle, and the camera corresponding to the perspective of the driver includes a camera whose shooting area is within the driver's line of sight and / or intersects with the driver's line of sight. For example, in Figure 5 Among the multiple cameras shown, the first camera includes camera 501, camera 503 and camera 505. It should be noted that when the mobile device is located outside the vehicle, that is, the vehicle-mounted host is remotely connected to the mobile device, the vehicle-mounted host sends the first display data to the mobile device, so as to achieve the effect of remote simulated driving.

[0133] In some embodiments, when the vehicle-mounted host receives a first display request sent by a mobile device, the above-mentioned first display data will be generated, and the first display request is used to request the content to be displayed of the first content type. Optionally, the first display request is triggered by a first function control in the function control displayed by the mobile device in the user interface. In some embodiments, the first function control is a control for triggering a simulated driving function, and the display style of the first function control is related to the simulated driving function, for example, the name of the first function control is a simulated driving function and / or the first function control is an icon related to driving behavior. In some embodiments, in the process of displaying a user interface including different function controls, the head-mounted display device detects the gaze position of the wearer in the user interface, and when the gaze position is located in the display area of ​​the first function control, the head-mounted display device highlights the first function control. In the process of highlighting the first function control, when a confirmation operation is detected by an external handle, the head-mounted display device triggers the first display request. Highlighting the first function control includes at least one of highlighting the first function control, bolding the first function control, and enlarging the first function control.

[0134] For the second type of display data:

[0135] When the content type of the content to be displayed is the second content type, the vehicle host generates second display data based on the camera image captured in real time by the second camera, wherein the display request corresponding to the second content type is triggered by the second function control in the function control displayed by the mobile device, and the second camera belongs to at least one camera.

[0136] Optionally, in the case where the mobile device is a head mounted display device, the second camera is determined by the vehicle-mounted host in at least one camera according to the sitting position, the facing direction of the wearer of the head mounted display device, and at least one of the first selection operations triggered in the head mounted display device. Optionally, the second camera corresponding to the sitting position includes a camera whose shooting area is within the visible range of the sitting position and / or intersects with the visible range of the sitting position. For example, continue to refer to Figure 5 , when the seat is located on the right side of the bus, the second camera corresponding to the seat includes camera 502 and camera 503. Optionally, the facing direction is sent by the head mounted display device to the vehicle host, and the second camera corresponding to the facing direction includes a camera whose shooting area is within the visible range of the facing direction and / or intersects with the visible range of the facing direction. For example, continue to refer to Figure 5, when the facing direction is the left side of the bus, the second camera corresponding to the facing direction includes camera 505 and camera 506. The first selection operation is triggered by a human-computer interaction operation in the head-mounted display device, and the first selection operation has a corresponding relationship with the display request of the head-mounted display device. Optionally, the selection result of the first selection operation is carried in the display request of the head-mounted display device. In the case where the mobile device is a display device, the process of determining the second camera by the vehicle-mounted host can also be performed with reference to the above content, and the embodiments of the present application are not repeated here.

[0137] In some embodiments, when the vehicle-mounted host receives a second display request sent by the mobile device, the above-mentioned second display data will be generated, and the second display request is used to request the content to be displayed of the second content type. Optionally, the second display request is triggered by the second function control in the function control displayed by the mobile device in the user interface. In some embodiments, the second function control is a control for triggering the real-time viewing function, and the display style of the second function control is related to the real-time viewing function, for example, the name of the second function control is the real-time viewing function and / or the second function control is an icon related to the viewing behavior. In some embodiments, in the process of displaying a user interface including different function controls, the head-mounted display device detects the gaze position of the wearer in the user interface, and when the gaze position is located in the display area of ​​the second function control, the head-mounted display device highlights the second function control. In the process of highlighting the second function control, when a confirmation operation is detected by the external handle, the head-mounted display device displays the first selection control, and when a selection operation for selecting the second camera triggered in the first selection control is received, the head-mounted display device triggers the second display request.

[0138] For the third type of display data:

[0139] When the content type of the content to be displayed is the third content type, the vehicle-mounted host generates third display data based on the camera images collected by the third camera in the historical period. The display request corresponding to the third content type is triggered by the third function control in the function control displayed by the mobile device, and the third camera belongs to at least one camera. Optionally, the vehicle-mounted host stores the camera image data collected by at least one camera connected thereto for use in the process of generating the third display data.

[0140] Optionally, in the case where the mobile device is a head mounted display device, the third camera is determined by the vehicle-mounted host in at least one camera according to the sitting position, the facing direction of the wearer of the head mounted display device, and at least one of the second selection operations triggered in the head mounted display device. Optionally, the third camera corresponding to the sitting position includes a camera whose shooting area is within the visible range of the sitting position and / or intersects with the visible range of the sitting position. For example, continue to refer to Figure 5 , when the seat is located on the right side of the bus, the third camera corresponding to the seat includes camera 502 and camera 503. Optionally, the facing direction is sent by the head mounted display device to the vehicle host, and the third camera corresponding to the facing direction includes a camera whose shooting area is within the visible range of the facing direction and / or intersects with the visible range of the facing direction. For example, continue to refer to Figure 5 , when the facing direction is the outside of the left side of the bus, the third camera corresponding to the facing direction includes camera 505 and camera 506. The second selection operation is triggered by the human-computer interaction operation in the head-mounted display device, and the second selection operation has a corresponding relationship with the display request of the head-mounted display device. Optionally, the selection result of the second selection operation is carried in the display request of the head-mounted display device. In the case where the mobile device is a display device, the process of determining the third camera by the on-board host can also be performed with reference to the above content, and the embodiments of the present application are not repeated here. In the process of generating the third display data, the on-board host will also determine the historical time period corresponding to the third display data according to the third selection operation triggered in the mobile device. The third selection operation has a corresponding relationship with the display request of the mobile device. Optionally, the selection result of the third selection operation is carried in the display request of the mobile device.

[0141] In some embodiments, when the vehicle-mounted host receives a third display request sent by the mobile device, the third display data is generated, and the third display request is used to request the content to be displayed of the third content type. Optionally, the third display request is triggered by a third function control in the function control displayed by the mobile device in the user interface. In some embodiments, the third function control is a control for triggering the historical viewing function, and the display style of the third function control is related to the historical viewing function, for example, the name of the third function control is the historical viewing function and / or the third function control is an icon related to the viewing behavior. In some embodiments, in the process of displaying a user interface including different function controls, the head-mounted display device detects the gaze position of the wearer in the user interface, and when the gaze position is located in the display area of ​​the third function control, the head-mounted display device highlights the third function control. In the process of highlighting the third function control, when a confirmation operation is detected by the external handle, the head-mounted display device displays a second selection control, and when a selection operation for selecting a third camera and a selection operation for selecting a historical period are received in the second selection control, the head-mounted display device triggers the third display request.

[0142] For the fourth type of display data:

[0143] When the content type of the content to be displayed is the fourth content type, the vehicle-mounted host generates a game scene screen based on the camera screen captured by the fourth camera, and generates a game screen based on the game scene screen, thereby generating fourth display data based on the game screen, and the game scene screen is a screen of the scene involved in the game process. Among them, the display request corresponding to the fourth content type is triggered by the fourth function control in the function control displayed by the mobile device. Optionally, the game screen includes at least one of the screen of a shooting game and the screen of a parkour game. In some embodiments, when the mobile device is a VR device, the game screen refers to the screen of a VR game; when the mobile device is an AR device, the game screen refers to the screen of an AR game; when the mobile device is a display device, the game screen refers to the screen of other games, and other games include games in forms other than VR games and AR games. The game screen is generated based on the game scene screen, and the game screen is composed of the game scene screen. The game scene screen is generated based on the camera screen captured by the fourth camera. Optionally, the vehicle-mounted host can obtain a game scene picture generated by the large language model based on the camera picture captured by the fourth camera by calling a large language model that supports image generation and inputting the camera picture captured by the fourth camera into the large language model, and then obtain the game picture by adding preset elements corresponding to the game type of the game picture to the game scene picture. The game scene can be a simulation of the real world, a semi-simulation and semi-fictional scene, or a purely fictional scene. The game scene can be any one of a two-dimensional scene, a 2.5-dimensional scene, and a three-dimensional scene, which is not limited in the embodiments of the present application. The fourth camera includes at least one camera for capturing pictures of the environment in front of the vehicle. For example, in Figure 5 Among the multiple cameras shown, the fourth camera includes camera 501. Optionally, when the game screen is a shooting game screen, the vehicle host will also generate a virtual shooting object in the game screen; when the game screen is a parkour game screen, the vehicle host will also generate a virtual obstacle in the game screen.

[0144] In some embodiments, when the vehicle-mounted host receives a fourth display request sent by the mobile device, the fourth display data is generated, and the fourth display request is used to request the content to be displayed of the fourth content type. Optionally, the fourth display request is triggered by a fourth function control in the function control displayed by the mobile device in the user interface. In some embodiments, the fourth function control is a control for triggering a game function, and the display style of the fourth function control is related to the game function, for example, the name of the fourth function control is the game function and / or the fourth function control is a game-related icon. In some embodiments, in the process of displaying a user interface including different function controls, the head-mounted display device detects the gaze position of the wearer in the user interface, and when the gaze position is within the display area of ​​the fourth function control, the head-mounted display device highlights the fourth function control. In the process of highlighting the fourth function control, when a confirmation operation is detected through an external handle, the head-mounted display device triggers the fourth display request.

[0145] Optionally, there are one or more game characters corresponding to mobile devices in the game screen, and the game character corresponding to the mobile device refers to the game character controlled by the mobile device. In the case where there are multiple game characters corresponding to each mobile device in the game screen, the game screen is a multiplayer game screen. Multiple mobile devices belong to the same network room, and the network room is established in the vehicle host according to the room creation operation triggered by the mobile device. Optionally, in the case where the mobile device is a head-mounted display device, the room creation operation is triggered by at least one of the two methods of an external device (such as a handle) corresponding to the head-mounted display device and the head-mounted display device tracking the eye sight of the wearing object. In the case where the mobile device is a display device, the room creation operation is triggered by touching the display screen of the display device and by at least one of the two methods of an external device corresponding to the display device. The game character refers to an active object in the game scene, and the active object can be a virtual character, a virtual animal, an animated character, etc., such as a character displayed in a three-dimensional game scene. Optionally, the game character is a three-dimensional stereo model created based on animation skeleton technology. Each game character has its own shape and volume in the three-dimensional game scene and occupies a part of the space in the three-dimensional game scene. In the process of displaying the game screen on the mobile device, the vehicle-mounted host will receive the control operation of the mobile device on the corresponding game character. Optionally, in the case where the mobile device is a head-mounted display device, the control operation is triggered by at least one of the two methods: an external device (such as a handle) corresponding to the head-mounted display device, and the head-mounted display device tracks the eye sight of the wearing object. In the case where the mobile device is a display device, the control operation is triggered by touching the display screen of the display device, and by at least one of the two methods of the external device corresponding to the display device. After receiving the control operation, the vehicle-mounted host will control the behavior of the game character in the game screen according to the control operation, such as controlling the game character to run, jump, shoot, etc. in the game screen, so as to update the fourth display data, and then the vehicle-mounted host will send the updated fourth display data to the mobile device. The mobile device displays the game screen according to the updated fourth display data, so as to realize the control of the game character in the game screen according to the control operation.

[0146] It should be noted that in the process of the vehicle host generating display data according to the camera screen collected by at least one camera, when the mobile device is a VR device in the head-mounted display device, and the content type of the content to be displayed by the VR device is related to the camera screen collected by multiple cameras in at least one camera, that is, when the vehicle host needs to use the camera screen collected by multiple cameras to generate display data, the vehicle host will generate display data of a panoramic screen corresponding to the content type for the VR device. In this case, the screen displayed by the four types of display data in the above text is a panoramic screen. Among them, the panoramic screen is obtained by the vehicle host by splicing the camera screens collected by multiple cameras. Optionally, the vehicle host splices the camera screens collected by multiple cameras according to the positional relationship between the multiple cameras to obtain a panoramic screen. In some embodiments, the vehicle host will also perform anti-distortion processing on the panoramic screen, and the anti-distortion processing is used to eliminate the distortion of the panoramic screen. Optionally, the vehicle host performs anti-distortion processing on the panoramic screen according to the anti-distortion coefficient, and the anti-distortion coefficient is related to the field of view (FOV) information of the lens of the VR device. In some embodiments, the vehicle host maps the stitched panoramic image as a texture onto a preset model to obtain a panoramic VR image, thereby sending display data of the panoramic VR image corresponding to the content type to the VR device. The preset model includes at least one of a preset sky box and a panoramic sphere.

[0147] In some embodiments, when the VR device is a high-definition VR device, the vehicle-mounted device will also obtain the gaze point information corresponding to the VR device during the process of stitching the panoramic picture. Among them, the high-definition VR device refers to a VR device that supports dynamic adjustment of the clarity (resolution) of the display screen according to the gaze point of the wearer, and the gaze point information is used to indicate the gazed area in the display screen of the VR device. After obtaining the gaze point information, the vehicle-mounted device will stitch the camera images captured by multiple cameras according to the gaze point information to obtain a panoramic picture. Among them, the clarity of the target area in the panoramic picture is higher than that of the area outside the target area, and the target area includes the area in the panoramic picture that is used to be displayed in the gazed area indicated by the gaze point information.

[0148] For the fifth type of display data:

[0149] In the case where the content type of the content to be displayed is the fifth content type, the vehicle-mounted host will obtain the fifth display data. Among them, the display request corresponding to the fifth content type is triggered by the fifth function control in the function control displayed by the mobile device. The target driving route is different from the current driving route of the vehicle. In some embodiments, the fifth display data is used to display the environment picture of the target driving route, and the target driving route can be determined by the vehicle-mounted host in the preset driving route according to the fourth selection operation. The fourth selection operation is triggered by the human-computer interaction operation in the mobile device, and the fourth selection operation has a corresponding relationship with the display request of the mobile device. Optionally, the selection result of the fourth selection operation is carried in the display request of the head-mounted display device. The fifth display data includes display data pre-stored in the vehicle-mounted host, in which case the vehicle-mounted host pre-stores display data of the environment picture of the target driving route; and at least one of the display data stored by the vehicle-mounted host during the vehicle driving in the historical driving route, and the display data stored by the vehicle-mounted host includes the display data of the camera picture collected by at least one camera connected to the vehicle-mounted host during the vehicle driving in the historical driving route.

[0150] In some embodiments, when the vehicle-mounted host receives the fifth display request sent by the mobile device, the fifth display data is generated, and the fifth display request is used to request the content to be displayed of the fifth content type. Optionally, the fifth display request is triggered by the fifth function control in the function control displayed by the mobile device in the user interface. In some embodiments, the fifth function control is a control for triggering the movie viewing function, and the display style of the fifth function control is related to the movie viewing function, for example, the name of the fifth function control is the movie viewing function and / or the fifth function control is an icon related to the movie viewing behavior. In some embodiments, in the process of displaying a user interface including different function controls, the head-mounted display device detects the gaze position of the wearer in the user interface, and when the gaze position is located in the display area of ​​the fifth function control, the head-mounted display device highlights the fifth function control. In the process of highlighting the fifth function control, when a confirmation operation is detected by the external handle, the head-mounted display device displays a third selection control, and multiple preset driving routes are displayed in the third selection control. When a selection operation for selecting a target driving route is received in the third selection control, the head-mounted display device triggers the fifth display request.

[0151] For the sixth type of display data:

[0152] In some embodiments, the vehicle host establishes a communication connection with at least two mobile devices, each of which includes a head-mounted display device or a display device. When the content type corresponding to the display request of the first mobile device is the sixth content type, the vehicle host will obtain the sixth display data of the current display content of the second mobile device, thereby sharing the current display content of the second mobile device with the first mobile device. Among them, the display request corresponding to the sixth content type is triggered by the sixth function control in the function control displayed by the mobile device. The first mobile device and the second mobile device are different devices among at least two mobile devices, and the first mobile device and the second mobile device belong to the same network room, which is established in the vehicle host according to the room creation operation triggered by the mobile device. The current display content is the content currently displayed by the second mobile device. Optionally, in the case where the mobile device is a head-mounted display device, the room creation operation is triggered by at least one of the two methods of an external device (such as a handle) corresponding to the head-mounted display device and the head-mounted display device tracking the eye sight of the wearing object. In the case where the mobile device is a display device, the room creation operation is triggered by at least one of the two methods of touching the display screen of the display device and by the external device corresponding to the display device.

[0153] Optionally, the display data of the current display content of the second mobile device is provided by the vehicle-mounted host to the second mobile device. In the case where the display data of the current display content of the second mobile device is not provided by the vehicle-mounted host to the second mobile device, the vehicle-mounted host will also obtain the display data of its current display content through the second mobile device, thereby sending the display data of the current display content of the second mobile device to the first mobile device, so as to share the current display content of the second mobile device with the first mobile device.

[0154] In some embodiments, when the vehicle-mounted host receives the sixth display request sent by the mobile device, the sixth display data is generated, and the sixth display request is used to request the content to be displayed of the sixth content type. Optionally, the sixth display request is triggered by the sixth function control in the function control displayed by the mobile device in the user interface. In some embodiments, the sixth function control is a control for triggering the screen sharing function, and the display style of the sixth function control is related to the screen sharing function, for example, the name of the sixth function control is the screen sharing function and / or the sixth function control is an icon related to the sharing behavior. In some embodiments, in the process of displaying a user interface including different function controls, the head-mounted display device detects the gaze position of the wearer in the user interface, and when the gaze position is located in the display area of ​​the sixth function control, the head-mounted display device highlights the sixth function control. In the process of highlighting the sixth function control, when a confirmation operation is detected by the external handle, the head-mounted display device displays a fourth selection control, and one or more candidate mobile devices belonging to the same network room as the first mobile device are displayed in the fourth selection control. When a selection operation for selecting a second mobile device is received in the fourth selection control, the head-mounted display device triggers the sixth display request.

[0155] It should be noted that the first function control, the second function control, the third function control, the fourth function control, the fifth function control and the sixth function control are different function controls in the user interface of the mobile device.

[0156] In some embodiments, the head-mounted display device includes an AR device, and an integrated camera is provided in the AR device. The vehicle-mounted host receives an object recognition request sent by the AR device, and the object recognition request carries a captured image of the integrated camera. In some embodiments, the object recognition request is triggered by at least one of an external device (such as a handle) corresponding to the AR device and the AR device tracking the eye sight of the wearing object. By identifying the object in the captured image, the vehicle-mounted host can obtain the recognition result of the captured image. Optionally, the object in the captured image identified by the vehicle-mounted host includes at least one of a person, an animal, a plant, a building, and a landmark in the real environment. The recognition result includes information about the object in the captured image, for example, including at least one of the name, type, introduction information of the object in the captured image, and the geographical location corresponding to the object. After the recognition result of the captured image is obtained, the vehicle-mounted host sends the recognition result to the AR device to display the recognition result through the AR device. It should be noted that the above-mentioned recognition process of the object in the captured image can also be independently implemented by the AR device, and this application does not limit this.

[0157] For example, Figure 6FIG. 1 is a schematic diagram of a process of sending display data provided by an exemplary embodiment of the present application. Figure 6 As shown, the vehicle-mounted host can obtain the display data of the camera screen 601 collected by each camera by calling multiple cameras (cameras). In the case where the content to be displayed of the display device (mobile device) 602 and the AR device 603 involves the shooting screen 601, the vehicle-mounted host can send the display data of the spliced ​​camera screen 601 or the unspliced ​​camera screen 601 to the display device 602 and the AR device 603 through the multi-screen MR system. For VR devices, the vehicle-mounted host will splice the camera screen 601 to obtain a panoramic screen 604. For ordinary VR devices 605, the vehicle-mounted host will send the display data of the panoramic screen 604 to it, and ordinary VR devices include VR devices other than high-definition VR devices 606. For high-definition VR devices 606, the vehicle-mounted host will splice the panoramic screen 604 according to the gaze point information, thereby sending the display data of the partial high-definition panoramic screen 604 to it. In addition, the vehicle-mounted host can also share the display data of the panoramic screen 604 of the VR device with it according to the display request of the display device 602.

[0158] Step 410: The vehicle-mounted host sends display data corresponding to the content type to the mobile device.

[0159] The display data is used for the head mounted display device to display the content to be displayed corresponding to the content type. Exemplarily, when the content type of the content to be displayed is the first content type, the vehicle-mounted host sends the first display data to the mobile device; when the content type of the content to be displayed is the second content type, the vehicle-mounted host sends the second display data to the mobile device; when the content type of the content to be displayed is the third content type, the vehicle-mounted host sends the third display data to the mobile device; when the content type of the content to be displayed is the fourth content type, the vehicle-mounted host sends the fourth display data to the mobile device; when the content type of the content to be displayed is the fifth content type, the vehicle-mounted host sends the fifth display data to the mobile device; when the content type of the content to be displayed is the sixth content type, the vehicle-mounted host sends the sixth display data to the mobile device.

[0160] Step 412: The mobile device displays the to-be-displayed content corresponding to the content type according to the display data.

[0161] The display data is used by the head mounted display device to display the to-be-displayed content corresponding to the content type. The head mounted display device can display the to-be-displayed content that conforms to the content type by rendering the display data.

[0162] For example, when the content type of the content to be displayed is the first content type, the mobile device displays the first screen according to the first display data; when the content type of the content to be displayed is the second content type, the mobile device displays the second screen according to the second display data; when the content type of the content to be displayed is the third content type, the mobile device displays the third screen according to the third display data; when the content type of the content to be displayed is the fourth content type, the mobile device displays the fourth screen according to the fourth display data; when the content type of the content to be displayed is the fifth content type, the mobile device displays the environmental screen of the target driving route according to the fifth display data; when the content type of the content to be displayed on the first mobile device is the sixth content type, the mobile device displays the current display content of the second mobile device according to the sixth display data.

[0163] In summary, the method provided in this embodiment generates display data corresponding to the content type requested by the head-mounted display device according to the camera image captured by at least one camera on the vehicle by the on-board host in the vehicle, so that the head-mounted display device can display the corresponding display content according to the display data. Since the display content of the head-mounted display device is generated according to the camera image captured by the camera on the vehicle, the interaction process between the vehicle and the passenger can be realized based on the camera image captured by the camera on the vehicle, and different display content is provided to the head-mounted display device according to the display request of the head-mounted display device, which not only enriches the display content provided to the passengers, but also provides a novel interaction method for passengers during their travels, enriching the interaction method between the vehicle and the passengers.

[0164] The method provided in this embodiment further enriches the display content provided to the passengers of the means of transport by determining the content type of the content to be displayed according to the display request of the head-mounted display device, so as to provide the head-mounted display device with display data corresponding to the content type. By providing the first display data to the head-mounted display device, an immersive driving experience is provided to the passengers or remote drivers, which helps to improve driving safety. By providing the second display data to the head-mounted display device, it is convenient for passengers to obtain external environmental information in real time, which helps passengers to quickly find their location in an unfamiliar environment and understand their current location in real time in the entire driving route. By providing the third display data to the head-mounted display device, it is convenient for passengers to replay the pictures in the trip on demand, so as to flexibly meet the different needs of passengers. By providing the fourth display data to the head-mounted display device, it is possible to provide a gaming experience for passengers and improve the fun of the travel process. By providing the display data of the environmental picture of the target driving route to the head-mounted display device, it is possible to flexibly meet the viewing needs of passengers. By providing the display data of the current display content of other mobile devices to the mobile device, it is possible to share the current display content of different passengers. By stitching panoramic images for VR devices, an immersive viewing experience can be provided to the wearer of the VR device. By providing recognition functions for AR devices, the wearer of the AR device can meet the needs of the wearer of the AR device to obtain information about surrounding objects, thereby improving the user experience of the AR device.

[0165] The present application provides a display method for a multi-screen MR (combined with VR and AR) system for transportation vehicles based on multiple cameras. Through multi-angle shooting with multiple cameras, the data of the collected camera images is obtained, and the different images required by different mobile devices are transmitted to the corresponding mobile devices after processing and analysis to meet different types of display requirements. The multi-screen MR system based on multiple cameras is a mixed reality system that combines multi-camera technology and multi-screen display technology. By integrating multiple cameras, image processing, multi-screen display and interactive technology, a multi-perspective, multi-screen display and highly interactive experience in a mixed reality environment is achieved. The system is highly scalable and flexible, and can be adjusted and optimized according to different needs, providing strong support for the development and application of mixed reality technology, while also creating a richer and more immersive mixed reality experience.

[0166] Taking the vehicle in the embodiment of the present application as a bus as an example, the method provided in the embodiment of the present application is introduced as follows: The architecture of the multi-screen heterogeneous display MR system based on multiple cameras can be referred to Figure 1The host device (vehicle host) renders and processes the content to be displayed on the mobile device according to the user identification (Identity Document, ID) logged in by different mobile devices and the display request sent by the mobile device at the moment, and feeds back the corresponding display data to the corresponding mobile device for display.

[0167] After logging into the user ID on the mobile device, the corresponding functions can be viewed. Ordinary VR devices and high-definition VR devices can select simulated driving functions, scenery and road conditions overview functions, movie viewing functions, and game functions. Among them, high-definition VR devices refer to VR devices that support dynamic adjustment of the clarity of the display screen according to the gaze point of the wearer, and ordinary VR devices include VR devices other than high-definition VR devices. The display content of the movie viewing function is generated based on the camera image collected by the camera connected to the host device. The camera records the environmental images of the entire journey of the bus during driving. In the case that the passenger misses the scenery or other information at certain moments, the corresponding display request can be triggered when necessary, so as to re-simulate the scene of the road section at the specified time. This function requires the full recording of the data of each frame of the panoramic picture spliced ​​by the pictures collected by the camera. Each frame of data corresponds to the real scene at a certain moment in the past. By sending these data to the mobile device, the mobile device can achieve 3D video playback effect through continuous playback of the player, thereby achieving the time and space restoration effect. By selecting the historical period of playback, the mobile device can switch to the panoramic picture of a certain moment in the past, so that passengers can observe things in any direction in the panoramic picture at this time. For example, when a collision occurs, passengers looking at the front of the bus can return to the panoramic picture of a period of time before the collision. By detecting the head movement of the passengers through the sensors of the mobile device, the passengers can observe what happened at the collision direction at that time. At the same time, by setting multiple image-taking positions in the panoramic picture, multiple perspectives (such as four standard positions of front, back, left and right) can be customized, so that passengers can simultaneously observe all the information happening around the current bus at the same time, so as to achieve the purpose of not missing all the scene information even if the bus moves forward. In addition, if the passenger is not satisfied with the scenery and road conditions under the current driving route, he can also switch to the environmental picture of the target driving route. For the game function, the real road conditions are combined with shooting and parkour games to generate the game screen. Under the same game screen, passengers in different seats of the bus can control their own game characters to shoot the virtual shooting objects in the game screen or avoid the virtual obstacles in the game scene in combination with the real road conditions. Passengers in the same network room can also eliminate virtual obstacles through cooperation. During the game, passengers can interact through the handle or the eye tracking of the head-mounted display device. For AR devices, it supports the selection of intelligent recognition functions (such as intelligent recognition of people and objects), movie viewing functions, landscape and road condition overview functions, and game functions, etc. In addition, the mobile device can also select a single or multiple screens accessible by its own user ID (such as the screen captured by the camera or the display screen in other mobile devices), such as the current display screen of the VR device of the friend ID or the current display screen of the AR device (such as game or chess and card applications, realizing a new form of game participation) for shared display.After receiving display requests from different mobile devices, the host device will retrieve and process the images captured by multiple cameras, and then return the processing results to each mobile device for display. For VR devices, different shooting images will be retrieved for different functions. For example, in single-player mode, the camera will be selected according to the riding role selected by the passenger. For example, when the passenger selects the driver mode, all camera contents on the bus will be retrieved in real time and then image stitching will be performed to form a complete real-time panoramic picture of the real road conditions, and then the panoramic picture will be corrected to show a real sense of space and proportion. After that, the panoramic picture is mapped as a texture to a pre-defined model (such as a skybox, a panoramic ball), and the display content to be displayed is determined based on the passenger's posture information and lens FOV information, so as to obtain a distortion-free full-field picture based on the picture captured by the camera. In addition, when requesting to display the parkour game screen, it is only necessary to retrieve the picture captured by the camera in front of the bus for stitching and mapping, which can save the device's processor usage and reduce power consumption. In addition, after the host device detects the user ID of different mobile devices, it will process the corresponding content to be displayed differently: for ordinary VR devices, the picture will be anti-distorted according to the anti-distortion coefficient of the device lens before being sent to the device; for high-definition VR devices, the image rules defined by the screen integrated circuit (IC) will be spliced ​​based on the gaze point information of the wearer and then sent to the device. After IC analysis, the high-definition VR device will output a local high-definition picture. This method improves the experience of high-resolution requirements while considering bandwidth costs. For AR devices, there is no need to use the real-time modeling function. The picture collected by the corresponding camera is called according to the function selected by the passenger. At this time, the pictures of multiple cameras can be spliced ​​into the effect required by the passenger for display. In addition, the AR software development kit (SDK) can also be used to add animation elements and markers to the transmitted picture to obtain the final picture and transmit it to the AR device and display device. The embodiment of the present application realizes a highly playable, diversified, dynamic and real bus riding experience.

[0168] For example, Figure 7 FIG. 1 is a schematic diagram of a display process provided by an exemplary embodiment of the present application. Figure 7As shown, in steps S1 to S13, the host device establishes a communication connection with at least one mobile device, and sends data for rendering the system main interface to the mobile device with which the communication connection is established, so that the mobile device displays the system main interface, thereby using different functions provided by the host device. In this process, after the host device is successfully connected to the mobile device, the host device will perform different processing on the display data after detecting different mobile device IDs. The processing process can refer to the relevant content in the foregoing text, and the embodiments of the present application will not be described in detail here. Afterwards, the system main interface of the corresponding device will be rendered on the corresponding device through the display data, and then subsequent steps will be executed. If the connection is not successful, the reason why the mobile device ID failed to log in successfully will be prompted on the host device, and a system instruction to re-login will be sent to it.

[0169] In step S14, the host device determines whether the mobile device selects team mode or single mode in the main interface. In steps S15 to S18, if the mobile device is in team mode, it will be required to fill in the corresponding team identification number to enter the corresponding network room. All mobile devices joining the network room can share their current display content in real time and can access the current display content of other mobile devices at any time. During this process, the host device will confirm whether the mobile device requests to view the current display content of the display device in the network room. If so, it will provide it with the current display content of the corresponding mobile device. The same mobile device can select the current display content of multiple mobile devices at the same time, and it also supports the mobile device to customize the position and size of each current display content when it is displayed.

[0170] In steps S19 to S34, the host device determines the type of mobile device, and different mobile devices correspond to different functions. For example, if the mobile device is a VR device, steps S20 to S22 are executed, if the mobile device is an AR device, steps S24 to S30 are executed, and if the mobile device is a display device, steps S31 to S33 are executed. In the case where the mobile device is determined to be a VR device, it will be made to enter the VR scene selection interface, and the application scenarios provided by the interface may include simulated driving, VR real-time traffic conditions, VR games, and VR viewing, etc. Among them, VR real-time traffic conditions can be selected to watch any combination of the 6 cameras equipped on the vehicle body. In the case of determining that the mobile device is an AR device, the host device will also determine whether the AR device is a multi-functional AR device. An integrated camera is provided in the multi-functional AR device, while an ordinary AR device is lighter than a multi-functional AR device and does not need to be equipped with a camera. After the judgment, the host device will make the AR device enter the corresponding AR scene selection interface. The application scenarios provided by the interface of the ordinary AR device may include AR viewing, AR real-time traffic conditions, AR games, etc., which can call any picture of the 6 cameras equipped on the vehicle body. The multifunctional AR device itself has a camera. Although it may be slightly inferior to ordinary AR devices in terms of weight and thinness, its available functions are richer and more intelligent. On the basis of the functions supported by ordinary AR devices, the multifunctional AR device also has the function of intelligent recognition, which can perform real-time recognition and processing of the pictures taken by its own camera. For example, the tour guide can use this device to familiarize the team members, facilitate management and interaction; passengers can recognize the scenic landmarks on the road and obtain introduction information. In the case where the mobile device is a display device, the host device needs to detect its screen information, obtain the matching image resolution, and make it enter the scene selection interface. The application scene provided by the interface of the display device can include all the contents of VR and AR scenes, and can be connected to the display device in the network room for screen viewing. For example, passengers who are more prone to dizziness when using VR devices can use their own or rented mobile devices to share the screen with family and friends by joining the network room. After that, the host device will determine whether the mobile device has exited the current application, and if so, it will exit to the corresponding scene selection interface. After that, the host device will determine whether the mobile device has exited the system, and if so, the process will end, otherwise it will switch to the system main interface display.

[0171] The embodiments of the present application include at least the following beneficial effects:

[0172] 1. Multiple display screens of different mobile devices can be controlled independently to display different contents. This makes the display of content more flexible and diverse, and can meet the personalized needs of passengers.

[0173] 2. According to the passenger's interaction and preferences, the display content and display effect are adjusted in real time to provide a better immersive experience and improve the satisfaction and comfort of each passenger.

[0174] 3. Through the coordinated work of multiple cameras, it is possible to capture all-round images of the real world and avoid blind spots. In addition, during driving, all passengers can obtain road information in real time, so that multiple people can grasp driving and road condition information at the same time, avoiding the situation where only the driver is responsible for safe driving.

[0175] 4. Different mobile devices can share images with each other and call the images captured by the camera. Through processing and 3D modeling, the effect of space redefinition can be achieved. For example, passengers in a vehicle can realize the virtual environment experience effect of simulated driving by calling the images collected by the camera corresponding to the driver. Passengers who are not sitting by the window can also call cameras in different positions to watch multiple images in real time.

[0176] 5. Control and coordinate the entire system through the host device to ensure the stability and reliability of the system.

[0177] It should be noted that before collecting relevant data of the user (such as the audio to be separated and the sample audio in this application) and during the process of collecting relevant data of the user, this application can display a prompt interface, a pop-up window or output a voice prompt information, and the prompt interface, pop-up window or voice prompt information is used to prompt the user that the relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining the relevant data of the user after obtaining the confirmation operation issued by the user to the prompt interface or pop-up window, otherwise (that is, when the confirmation operation issued by the user to the prompt interface or pop-up window is not obtained), the relevant steps of obtaining the relevant data of the user are terminated, that is, the relevant data of the user is not obtained. In other words, all user data collected by this application are collected with the consent and authorization of the user, and the collection, use and processing of relevant user data need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0178] It should be noted that the sequence of the method steps provided in the embodiments of the present application can be appropriately adjusted, and the steps can be increased or decreased accordingly according to the circumstances. Any technician familiar with the technical field can easily think of the changed methods within the technical scope disclosed in the present application, and they should be included in the protection scope of the present application, so they will not be repeated here.

[0179] Figure 8 FIG. 1 is a schematic diagram of a display system provided by an exemplary embodiment of the present application. Figure 8As shown, the display system 800 includes a vehicle-mounted host (host device 801) in a vehicle and at least one head-mounted display device. In some embodiments, the head-mounted display device includes at least one of a VR device and an AR device, which is not limited in the embodiments of the present application. Optionally, the VR device includes at least one of a high-definition VR device 802 and a general VR device 803. The AR device includes at least one of a multi-functional AR device 804 and a thin and light AR device (other AR device) 805. Among them, the high-definition VR device 802 refers to a VR device that supports dynamic adjustment of the clarity of the display screen according to the gaze point of the wearing object, and the general VR device 803 includes VR devices other than the high-definition VR device 802. The multi-functional AR device 804 includes an AR device with an integrated camera, and the thin and light AR device 805 includes an AR device with fewer cameras than the multi-functional AR device 804, such as an AR device without a camera. In some embodiments, the display system 800 may also include a display device 806, and the display device 806 includes a display device other than a head-mounted display device. The head-mounted display device and the display device 806 can be collectively referred to as a mobile device.

[0180] In some embodiments, the host device 801 includes a high-speed processor and a display. The host device 801 is connected to at least one camera on the vehicle, such as multiple omnidirectional cameras around the vehicle. The camera settings can refer to Figure 5 As shown. In some embodiments, multiple cameras are used to capture images of the surrounding environment of the vehicle from different angles. The high-definition VR device 802 is composed of a high-definition screen IC display and a microprocessor, which can meet the high-demand visual experience. The high-definition screen IC outputs the high-definition display effect of the corresponding high-definition area according to the splicing rules of the high-definition area and the panoramic image. At the same time, the microprocessor does not involve large-scale calculations and image processing, but only involves a small amount of interaction and transmission functions. It is small in size and light in weight, and will not significantly increase the volume and weight of the VR device, which can reduce the wearing burden and load-bearing of the wearer. The ordinary VR device 803 is composed of an ordinary display and a microprocessor, which can meet the low-demand VR scene experience. The multifunctional AR device 804 is composed of a microprocessor, a display and a camera, and supports the use of identification functions by the wearer. For example, the tour guide can identify team members through the AR device, and the road signs and regional identification buildings can be identified through the AR device in the off-site travel scene. Different from the multifunctional AR device 804, the thin and light AR device 805 reduces the camera and processor structure accordingly, and will be thinner and more suitable for long-term wear. The display device 806 can be a smart phone, a tablet computer, a laptop computer, etc., and can be a device carried by the passenger himself. All of the above mobile devices can be connected to the host device 801 via a local area network for communication and data transmission.

[0181] The vehicle-mounted host is used to establish a communication connection with at least one head-mounted display device; the head-mounted display device is used to send a display request to the vehicle-mounted host, and different display requests are used to request different contents to be displayed for the head-mounted display device, and different display requests are triggered by different human-computer interaction operations in the head-mounted display device; the vehicle-mounted host is also used to generate display data corresponding to the content type based on the camera image captured by at least one camera according to the content type corresponding to the display request; the vehicle-mounted host is also used to send display data to the head-mounted display device; the head-mounted display device is also used to display the contents to be displayed corresponding to the content type according to the display data. For the interaction process between the vehicle-mounted host and the mobile device, reference may be made to the relevant contents in other embodiments of the present application, and the embodiments of the present application will not be repeated here.

[0182] Fig. 9 1 is a schematic diagram of the structure of a display device provided by an exemplary embodiment of the present application. The device can be used for a vehicle-mounted host in a vehicle, and the vehicle-mounted host is connected to at least one camera on the vehicle. Fig. 9 As shown, the device comprises:

[0183] A communication module 901 is used to establish a communication connection with at least one head mounted display device;

[0184] The communication module 901 is further used to receive a display request sent by the head mounted display device, where different display requests are used to request different contents to be displayed for the head mounted display device, and the different display requests are triggered by different human-computer interaction operations in the head mounted display device;

[0185] A generating module 902, configured to generate display data corresponding to the content type according to the content type corresponding to the display request and based on the camera image captured by the at least one camera;

[0186] The communication module 901 is further configured to send display data corresponding to the content type to the head mounted display device, wherein the display data is used to display the to-be-displayed content corresponding to the content type.

[0187] In an optional design, the different display requests are triggered by triggering operations on different functional controls, and the functional controls are displayed in the user interface of the head-mounted display device; the generation module 902 is used to determine the content type corresponding to the display request according to the request parameters of the display request, and the request parameters correspond to the functional controls that trigger the display request; according to the content type, the display data is generated based on the camera image captured by the at least one camera.

[0188] In an optional design, the at least one camera is used to capture images of the surrounding environment of the vehicle; the generation module 902 is used to generate first display data based on the camera image captured by the first camera when the content type is a first content type, and the display request corresponding to the first content type is triggered by a first function control in the function controls; wherein the first camera includes the camera among the at least one camera that corresponds to the perspective of the driver of the vehicle.

[0189] In an optional design, the at least one camera is used to capture images of the surrounding environment of the vehicle; the generation module 902 is used to generate second display data based on the camera image captured in real time by the second camera when the content type is a second content type, and the display request corresponding to the second content type is triggered by a second function control in the function control; wherein the second camera is determined in the at least one camera based on the sitting position, facing direction, and at least one of the first selection operations triggered in the head-mounted display device of the wearer of the head-mounted display device.

[0190] In an optional design, the at least one camera is used to capture images of the surrounding environment of the vehicle; the generation module 902 is used to generate third display data based on the camera images captured by the third camera within a historical period when the content type is a third content type, and the display request corresponding to the third content type is triggered by a third function control in the function control; wherein the third camera is determined in the at least one camera based on the sitting position, facing direction, and at least one of the second selection operations triggered in the head-mounted display device of the wearer of the head-mounted display device; the historical period is determined based on the third selection operation triggered in the head-mounted display device.

[0191] In an optional design, the at least one camera is used to capture images of the surrounding environment of the vehicle; the generation module 902 is used to generate a game scene screen based on the camera screen captured by a fourth camera when the content type is a fourth content type, and the display request corresponding to the fourth content type is triggered by a fourth function control in the function control; a game screen is generated based on the game scene screen; and fourth display data is generated based on the game screen; wherein the fourth camera includes a camera among the at least one camera used to capture images of the front environment of the vehicle.

[0192] In an optional design, there are one or more game characters corresponding to the head mounted display device in the game screen; Fig.10As shown, the device also includes: the communication module 901, which is used to receive the control operation of the head-mounted display device on the game character during the process of the head-mounted display device displaying the game screen; the update module 903, which is used to control the behavior of the game character in the game screen according to the control operation to update the fourth display data; the communication module 901, which is used to send the updated fourth display data to the head-mounted display device.

[0193] In an optional design, the communication module 901 is used to send fifth display data to the head-mounted display device when the content type is a fifth content type, and the display request corresponding to the fifth content type is triggered by a fifth function control in the function control; wherein the fifth display data is used to display an environmental picture of a target driving route, the target driving route is different from the current driving route of the vehicle, and the fifth display data includes display data pre-stored in the vehicle-mounted host and at least one of the display data saved by the vehicle-mounted host during the vehicle's driving on a historical driving route.

[0194] In an optional design, the vehicle-mounted host establishes a communication connection with at least two mobile devices, each mobile device includes the head-mounted display device or the display device, and the display device includes a device other than the head-mounted display device; the communication module 901 is used to send the sixth display data of the current display content of the second mobile device to the first mobile device when the content type corresponding to the display request of the first mobile device is the sixth content type, and the display request corresponding to the sixth content type is triggered by the sixth function control in the function control; wherein the first mobile device and the second mobile device are different devices among the at least two mobile devices, the first mobile device and the second mobile device belong to the same network room, and the network room is established according to the room creation operation triggered by the mobile device.

[0195] In an optional design, the communication module 901 is used to send display data for displaying the user interface to the head-mounted display device.

[0196] In an optional design, the generation module 902 is used to generate display data of a panoramic picture corresponding to the content type for the VR device when the head-mounted display device is a VR device and the content type is related to the camera images captured by multiple cameras of the at least one camera; wherein the panoramic picture is obtained by splicing the camera images captured by the multiple cameras.

[0197] In an optional design, the head mounted display device includes an AR device, and the AR device is provided with an integrated camera; Fig.11 As shown, the apparatus further includes: the communication module 901, which is used to receive an object recognition request sent by the AR device, wherein the object recognition request carries the captured image of the integrated camera; the recognition module 904, which is used to obtain a recognition result of the captured image by recognizing the object in the captured image; and the communication module 901, which is used to send the recognition result to the AR device so as to display the recognition result through the AR device.

[0198] Fig.12 FIG. 1 is a schematic diagram of a head mounted display device provided by an exemplary embodiment of the present application. Fig.12 As shown, the device comprises:

[0199] The communication module 1201 is used to establish a communication connection with a vehicle-mounted host in a vehicle, and the vehicle-mounted host is connected to at least one camera on the vehicle;

[0200] The communication module 1201 is further used to send a display request to the vehicle-mounted host, where different display requests are used to request different contents to be displayed for the head-mounted display device, and the different display requests are triggered by different human-computer interaction operations in the head-mounted display device;

[0201] The communication module 1201 is further used to receive display data sent by the vehicle-mounted host, where the display data is generated by the vehicle-mounted host according to the content type corresponding to the display request and based on the camera image captured by the at least one camera;

[0202] The display module 1202 is configured to display the to-be-displayed content corresponding to the content type according to the display data.

[0203] It should be noted that the display device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the display device provided in the above embodiment and the display method embodiment based on the head-mounted display device belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0204] An embodiment of the present application also provides a vehicle-mounted host, which includes: a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the display method based on the head-mounted display device provided by the above-mentioned method embodiments.

[0205] An embodiment of the present application also provides a head-mounted display device, which includes: a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the display method based on the head-mounted display device provided by the above-mentioned method embodiments.

[0206] For example, Fig.13 1 is a schematic diagram of the structure of a computer device provided by an exemplary embodiment of the present application. The computer device can be implemented as a vehicle-mounted host and / or a head-mounted display device.

[0207] The computer device 1300 includes a central processing unit (CPU) 1301, a system memory 1304 including a random access memory (RAM) 1302 and a read-only memory (ROM) 1303, and a system bus 1305 connecting the system memory 1304 and the central processing unit 1301. The computer device 1300 also includes a basic input / output system (I / O system) 1306 for facilitating information transmission between various components in the computer device, and a large-capacity storage device 1307 for storing an operating system 1313, application programs 1314, and other program modules 1315.

[0208] The basic input / output system 1306 includes a display 1308 for displaying information and an input device 1309 such as a mouse and a keyboard for user inputting information. The display 1308 and the input device 1309 are connected to the central processing unit 1301 through an input / output controller 1310 connected to the system bus 1305. The basic input / output system 1306 may also include an input / output controller 1310 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 1310 also provides output to a display screen, a printer, or other types of output devices.

[0209] The mass storage device 1307 is connected to the central processing unit 1301 through a mass storage controller (not shown) connected to the system bus 1305. The mass storage device 1307 and its associated computer readable storage medium provide non-volatile storage for the computer device 1300. That is, the mass storage device 1307 may include a computer readable storage medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0210] Without loss of generality, the computer-readable storage medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable storage instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electronically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other solid-state storage devices, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer storage medium is not limited to the above. The above-mentioned system memory 1304 and mass storage device 1307 can be collectively referred to as memory.

[0211] The memory stores one or more programs, and the one or more programs are configured to be executed by one or more central processing units 1301. The one or more programs contain instructions for implementing the above-mentioned method embodiments. The central processing unit 1301 executes the one or more programs to implement the methods provided by the above-mentioned method embodiments.

[0212] According to various embodiments of the present application, the computer device 1300 can also be connected to a remote computer device on a network through a network such as the Internet. That is, the computer device 1300 can be connected to a network 1312 through a network interface unit 1311 connected to the system bus 1305, or the network interface unit 1311 can be used to connect to other types of networks or remote computer device systems (not shown).

[0213] The memory also includes one or more programs, which are stored in the memory and include steps executed by a computer device in the method provided in the embodiment of the present application.

[0214] A computer-readable storage medium is also provided in an embodiment of the present application, in which at least one instruction, at least one program, code set or instruction set is stored. When the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor of a computer device, the display method based on a head-mounted display device provided in the above-mentioned method embodiments is implemented.

[0215] The present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the display method based on the head-mounted display device provided by the above-mentioned method embodiments.

[0216] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned computer-readable storage medium may be a read-only memory, a disk or an optical disk, etc.

[0217] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent switching, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A display method based on a head mounted display device, characterized in that: The method is applied to a vehicle-mounted host in a vehicle, the vehicle-mounted host is connected to at least one camera on the vehicle, and the method comprises: establishing a communication connection with at least one head mounted display device; receiving a display request sent by the head-mounted display device, where different display requests are used to request different contents to be displayed for the head-mounted display device, and the different display requests are triggered by different human-computer interaction operations in the head-mounted display device; According to the content type corresponding to the display request, based on the camera image captured by the at least one camera, generating display data corresponding to the content type; Sending display data corresponding to the content type to the head mounted display device, wherein the display data is used to display the to-be-displayed content corresponding to the content type.

2. The method according to claim 1, characterized in that The different display requests are triggered by triggering operations on different function controls, and the function controls are displayed in the user interface of the head mounted display device; The step of generating display data corresponding to the content type according to the content type corresponding to the display request and based on the camera image captured by the at least one camera includes: Determining, according to a request parameter of the display request, a content type corresponding to the display request, wherein the request parameter corresponds to a function control that triggers the display request; The display data is generated according to the content type and based on a camera image captured by the at least one camera.

3. The method according to claim 2, characterized in that The at least one camera is used to collect images of the surrounding environment of the vehicle; and the generating of the display data based on the camera images collected by the at least one camera according to the content type includes: In a case where the content type is a first content type, first display data is generated based on a camera image captured by a first camera, and a display request corresponding to the first content type is triggered by a first function control in the function controls; The first camera includes a camera among the at least one camera that corresponds to the viewing angle of the driver of the vehicle.

4. The method according to claim 2, characterized in that: The at least one camera is used to collect images of the surrounding environment of the vehicle; and the generating of the display data based on the camera images collected by the at least one camera according to the content type includes: In the case where the content type is a second content type, second display data is generated based on a camera image captured in real time by a second camera, and a display request corresponding to the second content type is triggered by a second function control in the function controls; The second camera is determined in the at least one camera according to a sitting position, a facing direction, and at least one of a first selection operation triggered in the head-mounted display device of a wearing object of the head-mounted display device.

5. The method according to claim 2, characterized in that: The at least one camera is used to collect images of the surrounding environment of the vehicle; and the generating of the display data based on the camera images collected by the at least one camera according to the content type includes: In the case where the content type is a third content type, generating third display data based on camera images collected by a third camera within a historical period, and a display request corresponding to the third content type is triggered by a third function control in the function controls; Among them, the third camera is determined in the at least one camera based on the sitting position, facing direction, and at least one of the second selection operation triggered in the head-mounted display device of the wearing object of the head-mounted display device; the historical period is determined based on the third selection operation triggered in the head-mounted display device.

6. The method according to claim 2, characterized in that The at least one camera is used to collect images of the surrounding environment of the vehicle; and the generating of the display data based on the camera images collected by the at least one camera according to the content type includes: In the case where the content type is a fourth content type, a game scene screen is generated based on a camera screen captured by a fourth camera, and a display request corresponding to the fourth content type is triggered by a fourth function control in the function controls; Generate a game screen based on the game scene screen; generating fourth display data based on the game screen; The fourth camera includes a camera among the at least one camera that is used to capture images of the environment in front of the vehicle.

7. The method according to claim 6, characterized in that There are one or more game characters corresponding to the head mounted display device in the game screen; the method further includes: In the process of the head mounted display device displaying the game screen, receiving a control operation of the head mounted display device on the game character; controlling the behavior of the game character in the game screen according to the control operation to update the fourth display data; The updated fourth display data is sent to the head mounted display device.

8. The method according to any one of claims 2 to 7, characterized in that: The method further comprises: In a case where the content type is a fifth content type, sending fifth display data to the head mounted display device, wherein a display request corresponding to the fifth content type is triggered by a fifth function control in the function controls; Among them, the fifth display data is used to display the environmental picture of the target driving route, and the target driving route is different from the current driving route of the vehicle. The fifth display data includes display data pre-stored in the vehicle-mounted host and at least one of the display data saved by the vehicle-mounted host during the vehicle's driving in the historical driving route.

9. The method according to any one of claims 2 to 7, characterized in that: The vehicle-mounted host establishes a communication connection with at least two mobile devices, each of which includes the head-mounted display device or the display device, and the display device includes a device other than the head-mounted display device; the method further includes: When the content type corresponding to the display request of the first mobile device is a sixth content type, sending sixth display data of the currently displayed content of the second mobile device to the first mobile device, and the display request corresponding to the sixth content type is triggered by a sixth function control in the function controls; The first mobile device and the second mobile device are different devices among the at least two mobile devices, the first mobile device and the second mobile device belong to the same network room, and the network room is established according to a room creation operation triggered by the mobile device.

10. The method according to any one of claims 2 to 7, characterized in that: The method further comprises: Display data for displaying the user interface is sent to the head mounted display device.

11. The method according to any one of claims 1 to 7, characterized in that: The step of generating display data corresponding to the content type according to the content type corresponding to the display request and based on the camera image captured by the at least one camera includes: When the head mounted display device is a virtual reality (VR) device and the content type is related to camera images captured by multiple cameras of the at least one camera, generating display data of a panoramic image corresponding to the content type for the VR device; The panoramic image is obtained by splicing the camera images captured by the multiple cameras.

12. The method according to any one of claims 1 to 7, characterized in that: The head mounted display device includes an augmented reality (AR) device, wherein the AR device is provided with an integrated camera; the method further includes: Receiving an object recognition request sent by the AR device, wherein the object recognition request carries an image captured by the integrated camera; Obtaining a recognition result of the captured image by recognizing an object in the captured image; The recognition result is sent to the AR device so that the recognition result is displayed through the AR device.

13. A display method based on a head mounted display device, characterized in that: The method is applied to a head mounted display device, and the method comprises: Establishing a communication connection with a vehicle-mounted host in a vehicle, wherein the vehicle-mounted host is connected to at least one camera on the vehicle; Sending a display request to the vehicle-mounted host, where different display requests are used to request different contents to be displayed for the head-mounted display device, and the different display requests are triggered by different human-computer interaction operations in the head-mounted display device; receiving display data sent by the vehicle-mounted host, wherein the display data is generated by the vehicle-mounted host according to the content type corresponding to the display request and based on the camera image captured by the at least one camera; The to-be-displayed content corresponding to the content type is displayed according to the display data.

14. A display system, characterized in that: The display system includes a vehicle-mounted host and at least one head-mounted display device in a vehicle, wherein the vehicle-mounted host is connected to at least one camera on the vehicle; The vehicle-mounted host is used to establish a communication connection with the at least one head-mounted display device; The head-mounted display device is used to send a display request to the vehicle-mounted host, where different display requests are used to request different contents to be displayed for the head-mounted display device, and the different display requests are triggered by different human-computer interaction operations in the head-mounted display device; The vehicle-mounted host is further configured to generate display data corresponding to the content type based on the camera image captured by the at least one camera according to the content type corresponding to the display request; The vehicle-mounted host is further used to send the display data to the head-mounted display device; The head mounted display device is further configured to display the to-be-displayed content corresponding to the content type according to the display data.

15. A vehicle-mounted host, characterized in that: The vehicle-mounted host includes: a processor and a memory, wherein the memory stores at least one program; the processor is used to execute the at least one program in the memory to implement the display method based on the head-mounted display device as described in any one of claims 1 to 12.

16. A head mounted display device, characterized in that: The head-mounted display device includes: a processor and a memory, wherein the memory stores at least one program; the processor is used to execute the at least one program in the memory to implement the display method based on the head-mounted display device as described in claim 13.

17. A computer-readable storage medium, characterized in that: The readable storage medium stores executable instructions, and the executable instructions are loaded and executed by the processor to implement the display method based on the head-mounted display device as described in any one of claims 1 to 13.

18. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. The processor reads and executes the computer instructions from the computer-readable storage medium to implement the display method based on the head-mounted display device as described in any one of claims 1 to 13.

Images