Vehicle-mounted virtual human generation method and system, vehicle, in-vehicle entertainment product and aerial imaging equipment

Abstract

The invention relates to the technical field of intelligent driving, and discloses a vehicle-mounted virtual human generation method and system, a vehicle, a vehicle-mounted infotainment device and an aerial imaging device.Communication information generated by at least one service module is obtained through the vehicle-mounted infotainment device, so that virtual human imaging parameters capable of reflecting the intention of a vehicle owner and the driving trend of the vehicle are determined; and generating a communication packet based on the virtual human imaging parameters and sending the communication packet to the aerial imaging equipment. After the communication packet is decoded by using the aerial imaging device, rendering is performed according to the obtained virtual human imaging parameters, the vehicle-mounted virtual human is generated, and the rendering effect is good, so that the rendering performance of the vehicle-mounted terminal does not need to be consumed, the energy consumption of the vehicle-mounted terminal is reduced, the virtual human imaging does not depend on the screen of the vehicle-mounted terminal, and the immersion feeling of a user is stronger. In addition, the communication information of different service modules of the vehicle can be used for virtual human imaging, and the imaging effect is richer.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a method, system, vehicle, vehicle computer and aerial imaging equipment for generating a vehicle-mounted virtual human. Background Art

[0002] Nowadays, cockpit functions such as intelligent driving, voice assistants, advanced audio and cinema movies have become an important area to enhance the driving experience. Global automakers are committed to creating the ultimate intelligent cockpit experience and have used new technologies to develop car systems, greatly improving driving safety.

[0003] At present, in-vehicle virtual humans, as a smart cockpit function, can bring users the ultimate driving experience. However, traditional in-vehicle virtual humans can only rely on the car screen for display rendering, and the rendering process is limited by the performance of the car, which easily leads to poor rendering effects and high energy consumption of the car. Summary of the invention

[0004] In view of this, the present invention provides a method, system, vehicle, vehicle computer and aerial imaging device for generating a vehicle-mounted virtual human to solve the problem that traditional vehicle-mounted virtual humans are heavily dependent on the vehicle computer screen, resulting in poor rendering effects and high energy consumption of the vehicle computer.

[0005] In a first aspect, the present invention provides a method for generating a vehicle-mounted virtual human, which is applied to a vehicle computer, and the method comprises:

[0006] Obtaining communication information generated by at least one business module in the vehicle computer;

[0007] Determine virtual human imaging parameters based on the communication information;

[0008] A communication packet is generated based on the virtual human imaging parameters, and the communication packet is sent to an aerial imaging device, so that the aerial imaging device parses the communication packet to obtain the virtual human imaging parameters, and renders based on the virtual human imaging parameters to generate a vehicle-mounted virtual human.

[0009] Beneficial effects: The communication information generated by at least one business module is obtained through the vehicle computer, so as to determine the virtual human imaging parameters that can reflect the driver's intention and the vehicle's driving trend, and then a communication packet is generated based on the virtual human imaging parameters and sent to the aerial imaging device. After the communication packet is decoded by the aerial imaging device, it is rendered according to the obtained virtual human imaging parameters to generate a vehicle-mounted virtual human with good rendering effect. This does not require consuming the rendering performance of the vehicle computer, reduces the energy consumption of the vehicle computer, and the virtual human imaging does not depend on the vehicle computer screen, so the user has a stronger sense of immersion. In addition, the present invention can use the communication information of different business modules of the vehicle to perform virtual human imaging, and the imaging effect is richer.

[0010] In an optional implementation, determining virtual human imaging parameters based on the communication information includes:

[0011] Filtering the communication information to obtain interactive data sent by at least one business module; wherein the business module includes an aerial imaging device setting module, a voice module and a navigation module;

[0012] According to the business instructions corresponding to the interactive data, the virtual human imaging parameters are obtained; wherein the virtual human imaging parameters include imaging trigger type, imaging area, imaging volume, virtual human model parameters, virtual human action parameters and / or virtual human voice parameters.

[0013] Beneficial effects: By extracting and identifying the communication information of each business module of the vehicle, business instructions that can reflect the driver's intention and the vehicle's driving trend are generated, so as to determine the imaging trigger type, imaging area, imaging volume, virtual human model, and the actions and voice broadcasts that the virtual human needs to perform according to the business instructions, so as to realize the interaction between the virtual human and the user and enhance the driving experience.

[0014] In an optional implementation, the business instruction includes an imaging parameter setting instruction, the virtual human action parameter includes the virtual human initial action set by the user, and the virtual human voice parameter includes the virtual human initial voice set by the user; and obtaining the virtual human imaging parameter according to the business instruction corresponding to the interaction data includes:

[0015] Obtaining an imaging parameter setting instruction issued by a user for the aerial imaging device according to the interactive data sent by the aerial imaging device setting module;

[0016] According to the imaging parameter setting instruction, the imaging trigger type, imaging area, imaging volume, virtual human model parameters, virtual human initial action and / or virtual human initial voice set by the user for the virtual human are obtained.

[0017] Beneficial effect: In an embodiment of the present invention, the user is allowed to issue imaging parameter setting instructions through the aerial imaging device setting module to set the imaging trigger type, imaging area, imaging volume, virtual human model parameters, virtual human motion parameters and / or virtual human voice parameters of the virtual human, thereby improving the richness of imaging and the user's experience.

[0018] In an optional implementation, the business instruction further includes a voice instruction, the virtual human action parameter further includes a target action corresponding to the voice instruction, and the virtual human voice parameter further includes a target voice corresponding to the voice instruction; obtaining the virtual human imaging parameter according to the business instruction corresponding to the interaction data further includes:

[0019] Extract and identify the interactive data sent by the voice module to obtain the voice command issued by the user;

[0020] According to the preset mapping relationship between the voice command and the virtual human imaging parameters, the target action and / or target voice corresponding to the voice command is confirmed.

[0021] Beneficial effect: In an embodiment of the present invention, the vehicle computer analyzes the user's intention through the interactive data sent by the voice module, obtains the user's voice command, and queries the pre-stored mapping relationship between the voice command and the virtual human imaging parameter to determine the target action that the virtual human needs to perform and / or the target voice that needs to be broadcast, which facilitates the aerial imaging equipment to render and project, thereby realizing the interaction between the vehicle computer language module and the virtual human, and improving the user's immersive experience during driving.

[0022] In an optional implementation, the business instruction further includes a navigation instruction, the virtual human action parameter further includes a target action corresponding to the navigation instruction, and the virtual human voice parameter further includes a target voice corresponding to the navigation instruction; and obtaining the virtual human imaging parameter according to the business instruction corresponding to the interaction data further includes:

[0023] Obtaining navigation instructions according to the interactive data sent by the navigation module;

[0024] According to the preset mapping relationship between the navigation instruction and the virtual human imaging parameter, the target action and / or target voice corresponding to the navigation instruction is confirmed.

[0025] Beneficial effect: In the embodiment of the present invention, the vehicle computer analyzes the driving trend of the vehicle through the interactive data sent by the navigation module to obtain the navigation instructions, and then queries and obtains the target action and / or target voice corresponding to the navigation instructions according to the pre-stored mapping relationship between the navigation instructions and the virtual human imaging parameters, which facilitates the aerial imaging equipment to perform rendering imaging and projection, thereby realizing the interaction between the vehicle computer navigation module and the virtual human, which is conducive to improving the user's immersive experience during driving.

[0026] In an optional implementation, generating a communication packet based on the virtual human imaging parameters and sending the communication packet to the aerial imaging device includes:

[0027] Byte-encoding the virtual human imaging parameters to generate a communication body, and generating a communication header according to the byte length of the communication body and version information of a preset communication protocol;

[0028] A communication packet is obtained according to the communication head and the communication body, and the communication packet is sent to the aerial imaging device according to the preset communication protocol, so that the aerial imaging device parses the communication packet to obtain the virtual human imaging parameters, and based on the virtual human imaging parameters, the initial state machine is updated, and the model is rendered using the updated state machine to obtain the target virtual human model, and the target virtual human model is projected to generate a vehicle-mounted virtual human.

[0029] Beneficial effect: The virtual human imaging parameters are byte-encoded according to the preset communication protocol to generate a communication body, and a communication header is generated according to the byte length of the communication body and the protocol version information. The communication package containing the communication header and the communication body is sent to the aerial imaging device, which is convenient for the aerial imaging device to unpack and avoid reading erroneous data, thereby improving the accuracy of information transmission.

[0030] In an optional implementation, sending a communication packet to an aerial imaging device according to a preset communication protocol includes:

[0031] Sending a synchronization message to the aerial imaging device for requesting to establish a connection;

[0032] After receiving the synchronization confirmation message returned by the aerial imaging device based on the synchronization message, sending a first confirmation message for confirming the establishment of the connection to the aerial imaging device to establish a connection with the aerial imaging device;

[0033] sending communication packets to aerial imaging equipment;

[0034] When it is detected that the communication packet has been sent, a termination message for requesting to close the connection is sent to the aerial imaging device;

[0035] After receiving the second confirmation message sent by the aerial imaging device to indicate the receipt of the termination message and the termination confirmation message returned based on the termination message, a third confirmation message for confirming the closing of the connection is sent to the aerial imaging device to disconnect the connection with the aerial imaging device.

[0036] Beneficial effect: In the embodiment of the present application, by sending the communication packet to the aerial imaging device according to the preset communication protocol, the reliability, integrity and accuracy of data transmission are improved, and there is no need to develop different communication standards for different aerial imaging equipment manufacturers, thereby reducing the project maintenance cost and development difficulty.

[0037] In a second aspect, the present invention provides a method for generating a vehicle-mounted virtual human, which is applied to an aerial imaging device, and the method comprises:

[0038] Receiving a communication packet sent by the vehicle computer; wherein the communication packet is generated by the vehicle computer by acquiring communication information generated by at least one business module and based on the virtual human imaging parameters corresponding to the communication information;

[0039] Parse the communication packet to obtain the virtual human imaging parameters;

[0040] Rendering is performed based on the virtual human imaging parameters to generate a vehicle-mounted virtual human.

[0041] Beneficial effects: The communication information generated by at least one business module is obtained through the vehicle computer, so as to determine the virtual human imaging parameters that can reflect the driver's intention and the vehicle's driving trend, and then a communication packet is generated based on the virtual human imaging parameters and sent to the aerial imaging device. After the communication packet is decoded by the aerial imaging device, it is rendered according to the obtained virtual human imaging parameters to generate a vehicle-mounted virtual human with good rendering effect. This does not require consuming the rendering performance of the vehicle computer, reduces the energy consumption of the vehicle computer, and the virtual human imaging does not depend on the vehicle computer screen, so the user has a stronger sense of immersion. In addition, the present invention can use the communication information of different business modules of the vehicle to perform virtual human imaging, and the imaging effect is richer.

[0042] In an optional implementation, rendering is performed based on the virtual human imaging parameters to generate a vehicle-mounted virtual human, including:

[0043] Based on the virtual human imaging parameters, the initial state machine is updated; wherein the virtual human imaging parameters include imaging trigger type, imaging area, imaging volume, virtual human model parameters, virtual human motion parameters and / or virtual human voice parameters;

[0044] The updated state machine is used to render the model to obtain the target virtual human model;

[0045] The target virtual human model is projected to generate a vehicle-mounted virtual human.

[0046] Beneficial effect: The virtual human imaging parameters sent by the vehicle computer are used to update the initial state machine of the aerial imaging device, so that the updated state machine can perform graphics drawing and projection according to the virtual human imaging parameters, with good rendering effect. There is no need for the vehicle computer to perform imaging rendering, which reduces the energy consumption of the vehicle computer.

[0047] In an optional implementation, using the updated state machine to render the model to obtain the target virtual human model includes:

[0048] The updated state machine is used to perform geometry drawing, texture mapping, light and shadow processing, color processing, depth testing, and template testing to obtain the target virtual human model.

[0049] Beneficial effects: Use the updated state machine to draw geometry and get the outline of the virtual person, then perform texture mapping to enrich the color, bumpiness, roughness, etc. of the virtual person, which is conducive to improving the authenticity of the virtual person. Then perform light and shadow processing and color processing to determine the illumination color of each voxel in the imaging area. Finally, the obtained rendering effect can be stored in the frame buffer before subsequent projection, reducing the flickering problem of the image and improving the stability of the rendering. In addition, the depth test and template test are used to filter noise and interference factors to further improve the authenticity of the rendering.

[0050] In an optional implementation, after projecting the target virtual human model to generate the in-vehicle virtual human, the method further includes:

[0051] Based on the virtual human action parameters and / or the virtual human voice parameters, obtaining the target action and / or the target voice;

[0052] Control the in-vehicle virtual human to perform target actions and / or broadcast target voice.

[0053] Beneficial effects: According to the virtual human action parameters and / or virtual human voice parameters, the target action that the virtual human needs to perform and / or the target voice that needs to be broadcast are identified, and the on-board virtual human is controlled to perform the target action and / or broadcast the target voice, thereby realizing the interaction process between the virtual human and the driver and passengers, and improving the vehicle experience and immersion of the driver and passengers.

[0054] In an optional implementation, after projecting the target virtual human model to generate the in-vehicle virtual human, the method further includes:

[0055] When no communication packet sent by the vehicle computer is received within a preset time, the vehicle-mounted virtual person is controlled to perform a preset standby action and / or broadcast a preset standby voice.

[0056] Beneficial effect: When no new communication packet is obtained within the preset time, that is, if the virtual person does not interact with the user, the embodiment controls the virtual person to enter a leisure state, controls the in-vehicle virtual person to perform some leisure standby actions, and can also play standby voice to further improve the realism of the in-vehicle virtual person.

[0057] In an optional implementation, after projecting the target virtual human model to generate the in-vehicle virtual human, the method further includes:

[0058] According to the new communication packet sent by the vehicle computer, new virtual human imaging parameters are obtained;

[0059] Adjust the on-vehicle virtual human according to the new virtual human imaging parameters.

[0060] Beneficial effects: In this embodiment, the vehicle will request the aerial imaging device to adjust the imaging area, imaging size, displayed virtual human model and other parameters, and will also issue new business instructions to adjust the motion parameters and voice parameters of the virtual human. After receiving the new communication package sent by the vehicle, the aerial imaging device will make adjustments according to the corresponding parameters to achieve real-time adjustment of the virtual human and enhance the user experience.

[0061] In an optional implementation, adjusting the in-vehicle virtual human according to the new virtual human imaging parameters includes:

[0062] Adjust the projection duration of the in-vehicle virtual person according to the new imaging trigger type;

[0063] And / or, adjust the projection position of the in-vehicle virtual person according to the new imaging area.

[0064] Beneficial effect: In the embodiment of the present invention, the user is allowed to adjust the projection duration and projection position of the in-vehicle virtual person, thereby improving the user experience.

[0065] In a third aspect, the present invention provides a vehicle-mounted virtual human generation system, comprising: a vehicle computer and an aerial imaging device;

[0066] The vehicle computer obtains communication information generated by at least one business module in the vehicle computer; determines virtual human imaging parameters based on the communication information; generates a communication packet based on the virtual human imaging parameters, and sends the communication packet to the aerial imaging device;

[0067] The aerial imaging device receives the communication packets sent by the vehicle computer; parses the communication packets to obtain the virtual human imaging parameters; renders based on the virtual human imaging parameters to generate a vehicle-mounted virtual human.

[0068] Beneficial effects: The vehicle computer is used to obtain communication information generated by at least one business module, so as to determine the virtual human imaging parameters that can reflect the driver's intention and the vehicle's driving trend, and then a communication packet is generated based on the virtual human imaging parameters and sent to the aerial imaging device. After the aerial imaging device decodes the communication packet, it is rendered according to the obtained virtual human imaging parameters to generate a vehicle-mounted virtual human with good rendering effect. In this way, there is no need to consume the rendering performance of the vehicle computer, which reduces the energy consumption of the vehicle computer. In addition, the virtual human imaging does not depend on the vehicle computer screen, and the user has a stronger sense of immersion.

[0069] In an optional embodiment, the aerial imaging device is mounted on the co-pilot seat, the back seat and / or the driving console of the vehicle.

[0070] Beneficial effects: By installing aerial imaging equipment in multiple areas of the vehicle, the imaging range is expanded, so that virtual human images can be presented in multiple areas of the vehicle, which is beneficial to improving the driving experience of multiple drivers and passengers.

[0071] In a fourth aspect, the present invention provides a vehicle, comprising the vehicle-mounted virtual human generation system of the third aspect or any corresponding embodiment thereof.

[0072] In a fifth aspect, the present invention provides a vehicle computer, which includes a first memory and a first processor, the first memory and the first processor are communicatively connected to each other, the first memory stores first computer instructions, and the first processor executes the in-vehicle virtual human generation method of the above-mentioned first aspect or any corresponding embodiment thereof by executing the first computer instructions.

[0073] In a sixth aspect, the present invention provides an aerial imaging device, which includes a second memory and a second processor, the second memory and the second processor are communicatively connected to each other, the second memory stores second computer instructions, and the second processor executes the vehicle-mounted virtual human generation method of the above-mentioned second aspect or any corresponding embodiment thereof by executing the second computer instructions.

[0074] The beneficial effects of the present invention are:

[0075] The communication information generated by at least one business module in the vehicle computer is obtained through the vehicle computer, so as to determine the virtual human imaging parameters that can reflect the driver's intention and the vehicle's driving trend, and then generate a communication packet based on the virtual human imaging parameters and send it to the aerial imaging device. After the communication packet is decoded by the aerial imaging device, it is rendered according to the obtained virtual human imaging parameters to generate a vehicle-mounted virtual human with good rendering effect. In this way, there is no need to consume the rendering performance of the vehicle computer, which reduces the energy consumption of the vehicle computer, and the virtual human imaging does not depend on the vehicle computer screen, so the user has a stronger sense of immersion. In addition, the present invention can use the communication information of different business modules of the vehicle to perform virtual human imaging, and the imaging effect is richer. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0077] Figure 1 is a structural block diagram of a vehicle-mounted virtual human generation system according to an embodiment of the present invention;

[0078] Figure 2 is a schematic diagram of the installation position of an aerial imaging device according to an embodiment of the present invention;

[0079] Figure 3 is a schematic diagram of the interaction process of the vehicle-mounted virtual human generation system according to an embodiment of the present invention;

[0080] Figure 4 is another interactive process schematic diagram of the vehicle-mounted virtual human generation system according to an embodiment of the present invention;

[0081] Figure 5 is a schematic diagram of a process of aerial imaging according to an embodiment of the present invention;

[0082] Figure 6 is a schematic diagram of a TCP communication process according to an embodiment of the present invention;

[0083] Figure 7 is a communication flow chart of a vehicle machine and an aerial imaging device according to an embodiment of the present invention;

[0084] Figure 8 is a schematic diagram of virtual human rendering imaging according to an embodiment of the present invention;

[0085] Fig. 9 is a schematic diagram of another virtual human rendering imaging according to an embodiment of the present invention;

[0086] Fig.10 is a schematic diagram of the architecture of a vehicle-mounted virtual human generation system according to an embodiment of the present invention;

[0087] Fig.11 is a structural block diagram of a vehicle according to an embodiment of the present invention;

[0088] Fig.12 is a structural schematic diagram of a vehicle computer according to an embodiment of the present invention;

[0089] Fig.13 is a schematic structural diagram of an aerial imaging device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0090] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0091] At present, when there is no significant development in automobile engine technology, enriching cockpit functions such as intelligent driving, voice assistants, advanced audio, cinema movies, etc. has become an important area to enhance the driving experience. Global automobile manufacturers are spending a lot of money to create the ultimate intelligent cockpit experience, and have redeveloped traditional car system modules using new technologies. For example, 2D car desktops are redeveloped into 3D car desktops using 3D rendering technology, allowing car owners to use car functions in a more intuitive and WYSIWYG way; another example is the Advanced Driver Assistance System (ADAS), which uses 3D world restoration technology to render the surrounding environment of the vehicle on the car screen, greatly improving driving safety.

[0092] As a smart cockpit function, in-vehicle virtual humans can bring users an ultimate driving experience. However, there are some problems with traditional in-vehicle virtual humans: 1) They can only rely on the car screen for display rendering and cannot be off-screen; 2) The rendering effect is related to the screen size. The effect is poor on a small screen, unclear and insufficient details; the effect is good on a large screen, but the performance consumption is high; 3) Limited by the performance of the car, if multiple high-consumption modules (such as voice, navigation) are run at the same time, the image rendering program will be stuck; 4) It is easy to conflict with other APPs in the rendering area, such as the virtual human image will cover the map navigation content; 5) Since the image is only displayed on a 2D screen, it cannot interact with specific objects in the cockpit (such as air conditioning vents).

[0093] The implementation of a new science and technology often brings great changes to related industries. Air Imaging Technology or Free-space Imaging Technology is an advanced display technology that can display images in the air without physical media (such as screens or projection screens). This technology uses the principles of reflection, refraction or scattering of light to suspend images in the air through specific optical devices, creating a visual effect of suspending in the air. Free-space imaging technology directly brings a large number of scenes that can only be seen in science fiction movies to people's real lives, and its application areas involve almost all aspects of life. If it can be used in the cockpit of a car, one or more 3D realistic virtual human images can be projected on demand in multiple areas of the cockpit. If the virtual human image can be customized by the owner, given a specific tone and action, and accompanied throughout the journey and provided with multi-faceted driving assistance, it will definitely greatly enhance the driving experience and make the user feel like they are in a future scene.

[0094] Therefore, an embodiment of the present invention provides a solution for generating a vehicle-mounted virtual human, which transmits the communication data collected by the vehicle computer to an aerial imaging device. The aerial imaging device generates a vehicle-mounted virtual human based on medium-free aerial imaging technology, and projects one or more realistic 3D virtual human images in multiple areas in the cabin as needed, thereby greatly improving the experience of the driver and passengers.

[0095] According to an embodiment of the present invention, a vehicle-mounted virtual human generation system 100 is provided, including: a vehicle computer 101 and an aerial imaging device 102, wherein the vehicle computer 101 can be used as a client, and the aerial imaging device 102 can be used as a server, and the client and the server communicate according to a preset communication protocol. It should be noted that the vehicle computer 101, that is, the client, can include various modules of the vehicle computer central control system (provided that the vehicle computer is connected to the upper-layer application development SDK), such as an aerial imaging device setting module, a voice module, a navigation module, and a sensor module, etc., which can be specifically set according to actual application requirements, and the present invention is not limited thereto.

[0096] The vehicle computer 101 obtains communication information generated by at least one service module in the vehicle computer; determines virtual human imaging parameters based on the communication information; generates a communication packet based on the virtual human imaging parameters, and sends the communication packet to the aerial imaging device 102.

[0097] The aerial imaging device 102 receives the communication packet sent by the vehicle computer 101; parses the communication packet to obtain the virtual human imaging parameters; and renders based on the virtual human imaging parameters to generate the vehicle-mounted virtual human.

[0098] The specific working principles and working processes of the vehicle computer 101 and the aerial imaging device 102 are described in the following method embodiment and will not be repeated here.

[0099] In some optional implementations, the aerial imaging device 102 may be mounted on the co-pilot seat, the rear seat and / or the driving console of the vehicle. The aerial imaging device 102 is usually installed by the vehicle manufacturer. Figure 2 As shown, the areas that can be installed are: co-pilot, back seat, and driving platform. In this embodiment, the images projected on the co-pilot and back seat are about the same size as real people, while the image projected on the driving platform is about 30cm*30cm*30cm.

[0100] In this embodiment, there are two strategies for the imaging position of the aerial imaging device 102: selective imaging and on-demand imaging, and the driver can switch between these two strategies at any time during driving. Selective imaging means that the central control system of the vehicle 101 will automatically traverse multiple imaging areas and select the first idle area without passengers for projection, so that the imaging area may be different each time; on-demand imaging means that each imaging area will have a unique identification ID, and the driver will select a specific imaging area for projection through the central control system of the vehicle 101, regardless of whether there are passengers in the imaging area.

[0101] This embodiment expands the imaging range by installing aerial imaging devices in multiple areas in the vehicle, so that multiple areas in the vehicle can present virtual human images, which is beneficial to improving the driving experience of multiple drivers and passengers.

[0102] In some optional implementations, the imaging triggering mechanism of the aerial imaging device 102 can be divided into two types: permanent projection and on-demand projection, and the driver can switch between these two mechanisms at any time during driving. Permanent projection means that as long as the owner starts the car, the vehicle central control system will project the image according to the imaging area strategy described above, and the projection will always exist until the car is turned off or the driver actively changes the imaging area strategy through the vehicle central control system. At this time, if the virtual person does not interact with humans (such as responding to voice commands, navigation commands, chatting and greeting passengers or drivers), the virtual person will enter a leisure state, do some leisure standby actions, and play standby voice.

[0103] Specifically, on-demand projection means that imaging will only be performed when there is a need to interact with the virtual person (such as responding to voice commands, navigation instructions, chatting and greeting passengers or drivers), and imaging will be projected according to the imaging area strategy described above. Once the interaction is completed, the virtual person will disappear and wait for the next interaction opportunity.

[0104] The vehicle-mounted virtual human generation system provided by the embodiment of the present invention uses the vehicle computer 101 to obtain communication information generated by at least one business module, so as to determine the virtual human imaging parameters that can reflect the driver's intention and the vehicle driving trend, and then generates a communication packet based on the virtual human imaging parameters and sends it to the aerial imaging device 102. After the aerial imaging device 102 decodes the communication packet, it renders according to the obtained virtual human imaging parameters to generate a vehicle-mounted virtual human with good rendering effect. In this way, there is no need to consume the rendering performance of the vehicle computer, which reduces the energy consumption of the vehicle computer. In addition, the virtual human imaging does not depend on the vehicle computer screen, and the user has a stronger sense of immersion.

[0105] According to an embodiment of the present invention, an embodiment of a method for generating a vehicle-mounted virtual human is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0106] In this embodiment, a method for generating a vehicle-mounted virtual person is provided. The method for generating a vehicle-mounted virtual person can be used as follows: Figure 1 The vehicle machine 101 and the aerial imaging device 102 are shown. Figure 3 1 is a schematic diagram of the interaction process of the vehicle-mounted virtual human generation system according to an embodiment of the present invention, wherein the vehicle computer 101 is used to execute steps S101 to S103, and the aerial imaging device 102 is used to execute steps S201 to S203. The specific interaction process between the vehicle computer 101 and the aerial imaging device 102 is as follows:

[0107] Step S101, obtaining communication information generated by at least one service module in the vehicle computer.

[0108] Specifically, the vehicle computer can include multiple service modules, as long as the communication information of each service module can be obtained. The service module can be a voice module, a navigation module, a sensor module, a parking module, a multimedia module, an ADAS module, a radar sensing module, a brake module, etc. These service modules can communicate with the vehicle computer, and the vehicle computer will uniformly arbitrate and process the communication information of these service modules.

[0109] Step S102: determining virtual human imaging parameters based on the communication information.

[0110] Specifically, the vehicle will identify, extract and process the communication information sent by each business module, analyze the user's usage intention and the vehicle's driving trend, and obtain virtual human imaging parameters that can reflect the user's intention or the vehicle's driving trend, so that imaging can be performed based on the virtual human imaging parameters.

[0111] For example, the voice module of the car computer will collect the owner's speech through a microphone and identify the owner's intention through a preset AI voice recognition module. The navigation module of the car computer will obtain the vehicle's driving trend based on the car's position and the map navigation route. The car computer will further analyze the virtual human imaging parameters such as the actions that the virtual human needs to make and the voices that they need to make based on the owner's intention and the vehicle's driving trend.

[0112] Step S103: Generate a communication packet based on the virtual human imaging parameters, and send the communication packet to the aerial imaging device.

[0113] Specifically, since there may be multiple manufacturers of aerial imaging equipment, each with different hardware and SDK development methods, if there is no unified communication standard, then each project will need to develop different communication protocols for different aerial imaging equipment manufacturers, which will undoubtedly increase the maintenance cost and development difficulty of the project. Therefore, an embodiment of the present invention provides a set of preset communication protocols for a unified business layer that is independent of the aerial imaging equipment, and the communication between the vehicle computer and the aerial imaging equipment is achieved according to the preset communication protocol, so that the vehicle computer can send the virtual human imaging parameters to the aerial imaging equipment. In addition, the present invention can use the communication information of different business modules of the vehicle to perform virtual human imaging, and the imaging effect is richer.

[0114] Step S201, receiving a communication packet sent by the vehicle computer.

[0115] Since there may be multiple aerial imaging equipment manufacturers, each with different hardware and SDK development methods, if there is no unified communication standard, each project needs to develop different communication protocols for different aerial imaging equipment manufacturers, which will undoubtedly increase the maintenance cost and development difficulty of the project. Therefore, the embodiment of the present invention implements communication between the vehicle computer and the aerial imaging equipment according to the preset communication protocol, so that the aerial imaging equipment can receive the communication package sent by the vehicle computer.

[0116] Step S202, parsing the communication packet to obtain virtual human imaging parameters.

[0117] Specifically, after receiving the communication packet, the aerial imaging device will unpack the communication packet according to the communication header and the communication body in the communication packet, decode it, and obtain detailed virtual human imaging parameters.

[0118] Step S203 , rendering is performed based on the virtual human imaging parameters to generate a vehicle-mounted virtual human.

[0119] Specifically, a three-dimensional graphics rendering library can be preset in the aerial imaging device to draw complex graphics in the air. The aerial imaging device performs rendering according to the virtual human imaging parameters and projects and generates a virtual human.

[0120] The in-vehicle virtual human generation method provided in this embodiment obtains communication information generated by at least one business module through the vehicle computer, thereby determining the virtual human imaging parameters that can reflect the driver's intention and the vehicle's driving trend, and then generates a communication packet based on the virtual human imaging parameters and sends it to the aerial imaging device. After the communication packet is decoded by the aerial imaging device, it is rendered according to the obtained virtual human imaging parameters to generate an in-vehicle virtual human with good rendering effect. In this way, the rendering performance of the vehicle computer does not need to be consumed, the energy consumption of the vehicle computer is reduced, and the virtual human imaging does not depend on the vehicle computer screen, so the user has a stronger sense of immersion.

[0121] In this embodiment, a method for generating a vehicle-mounted virtual person is provided. The method for generating a vehicle-mounted virtual person can be used as follows: Figure 1 The vehicle machine 101 and the aerial imaging device 102 are shown. Figure 4 1 is a schematic diagram of the interaction process of the vehicle-mounted virtual human generation system according to an embodiment of the present invention, wherein the vehicle computer 101 is used to execute steps S301 to S305, and the aerial imaging device 102 is used to execute steps S401 to S203. The specific interaction process between the vehicle computer 101 and the aerial imaging device 102 is as follows:

[0122] Step S301, obtain the communication information generated by at least one business module in the vehicle computer. Figure 3 Step S101 of the illustrated embodiment will not be described in detail here.

[0123] Step S302, filtering the communication information to obtain interactive data sent by at least one service module; wherein the service module includes an aerial imaging device setting module, a voice module and a navigation module.

[0124] Specifically, Figure 5 As shown, the vehicle can be equipped with hardware devices such as 360 cameras, infrared equipment, microphones, etc. to collect communication information such as external environmental information, so that various business modules of the vehicle computer (such as voice module, navigation module, etc.) can send the corresponding communication information to the vehicle computer.

[0125] Specifically, since some communication information does not contain data that can be used for imaging, it is necessary to first filter the communication information to select the interactive data that can interact with the virtual person in each business module. Among them, information filtering can be based on keyword type, type of communication information sender, etc. If the communication information contains preset keywords and senders, the corresponding information is used as interactive data that can be imaged, which is convenient for analyzing the virtual person imaging parameters and improves the data processing efficiency.

[0126] Step S303, obtaining virtual human imaging parameters according to the business instructions corresponding to the interaction data.

[0127] Among them, the virtual human imaging parameters may include imaging trigger type, imaging area, imaging volume, virtual human model parameters, virtual human action parameters and / or virtual human voice parameters, which can be set according to the actual scenario.

[0128] In the embodiment of the present invention, the types and descriptions of virtual human imaging parameters are shown in Table 1 below:

[0129] Table 1

[0130]

[0131] It should be noted that the virtual human voice parameters may include the virtual human's timbre, volume, and voice broadcast content, etc., which can be set specifically according to the actual application scenario.

[0132] In an optional embodiment, the business instruction may include a voice instruction, the virtual human action parameter may further include a target action corresponding to the voice instruction, and the virtual human voice parameter may further include a target voice corresponding to the voice instruction. First, the interactive data sent by the voice module may be extracted and identified to obtain the voice instruction issued by the user. Then, according to the preset mapping relationship between the voice instruction and the virtual human imaging parameter, the target action and / or target voice corresponding to the voice instruction may be confirmed.

[0133] Specifically, since the interactive data sent by the voice module may contain some interference data such as background sounds and noise, the key information in the interactive data can be extracted and identified through keyword extraction algorithms, filtering algorithms, etc. to filter out the background noise, thereby accurately identifying the target user's intentions and obtaining language instructions.

[0134] For example, the car machine voice module will collect the owner's speech content through the microphone, and recognize the owner's intention through the AI ​​voice recognition module. For example, if the owner says "set the air conditioning temperature to 25 degrees", the business layer of the car machine client voice module will generate a voice command "set the air conditioning temperature to 25 degrees". The car machine will pre-store the mapping relationship between voice commands and virtual human imaging parameters. The car machine will query the pre-stored mapping relationship based on the voice command and obtain the target action corresponding to the voice command "pointing the left hand to the air outlet of the air conditioner".

[0135] In an embodiment of the present invention, the vehicle computer analyzes the user's intention through the interactive data sent by the voice module, obtains the user's voice command, and queries the pre-stored mapping relationship between the voice command and the virtual human imaging parameter to determine the target action that the virtual human needs to perform and / or the target voice that needs to be broadcast, so as to facilitate the aerial imaging equipment to perform rendering, imaging and projection, thereby realizing the interaction between the vehicle computer language module and the virtual human, and improving the user's immersive experience during driving.

[0136] In some optional implementations, the business instruction may include a navigation instruction, the virtual human action parameter may further include a target action corresponding to the navigation instruction, and the virtual human voice parameter may further include a target voice corresponding to the navigation instruction. The vehicle computer first obtains the navigation instruction according to the interactive data sent by the navigation module. Then, according to the preset mapping relationship between the navigation instruction and the virtual human imaging parameter, the target action and / or target voice corresponding to the navigation instruction is confirmed.

[0137] For example, the vehicle navigation module recognizes in advance that it needs to turn right at the next intersection based on the vehicle position and the map navigation route, so the corresponding navigation instruction can be "turn right at the intersection ahead". The vehicle computer queries and obtains the target action "right hand pointing to the right front" and the target language "turn right at the intersection ahead" corresponding to the navigation instruction based on the pre-stored mapping relationship between the navigation instruction and the virtual human imaging parameters.

[0138] In an embodiment of the present invention, the vehicle computer analyzes the driving trend of the vehicle through the interactive data sent by the navigation module to obtain navigation instructions, and then queries and obtains the target action and / or target voice corresponding to the navigation instructions based on the pre-stored mapping relationship between the navigation instructions and the virtual human imaging parameters, which facilitates the aerial imaging device to perform rendering, imaging and projection, thereby realizing the interaction between the vehicle computer navigation module and the virtual human, which is conducive to improving the user's immersive experience during driving.

[0139] In some optional implementations, the business instruction may include an imaging parameter setting instruction, the virtual human action parameter includes the virtual human initial action set by the user, and the virtual human voice parameter includes the virtual human initial voice set by the user. The vehicle computer may obtain the imaging parameter setting instruction issued by the user for the aerial imaging device according to the interactive data sent by the aerial imaging device setting module. According to the imaging parameter setting instruction, the imaging trigger type, imaging area, imaging volume, virtual human model parameters, virtual human initial action and / or virtual human initial voice set by the user for the virtual human are obtained.

[0140] For example, an aerial imaging device setting module corresponding to the aerial imaging device can be provided at the vehicle computer application layer, such as an operation APP, through which the user can select the imaging area, imaging volume, and specific model of the virtual person and preset the initial action and / or initial voice of the virtual person. At the same time, the operation APP will generate corresponding imaging parameter setting instructions, so that the vehicle computer can determine the imaging parameters of the virtual person according to the imaging parameter setting instructions. Of course, the vehicle computer can also determine the imaging area, imaging volume, specific model of the virtual person and other parameters according to the space occupancy in the vehicle, so as to improve the flexibility of imaging and the utilization rate of the space in the vehicle.

[0141] In some optional embodiments, the user can first make preliminary settings for the virtual human imaging parameters through the aerial imaging device setting module, such as setting the imaging area, imaging trigger type, the virtual human model to be used, etc., so that the aerial imaging device can preliminarily generate a virtual human. In the subsequent driving process, the vehicle computer can combine the voice commands issued by the user through the voice module, the navigation commands corresponding to the navigation module, etc., to re-determine the actions that the virtual human needs to perform and / or the voices to be broadcast, thereby realizing the virtual human's action and / or voice feedback on the user's intentions or the vehicle's driving intentions during driving, thereby achieving the purpose of improving the user experience.

[0142] In an embodiment of the present invention, a user is allowed to issue an imaging parameter setting instruction through an aerial imaging device setting module to set the imaging trigger type, imaging area, imaging volume, virtual human model parameters, virtual human motion parameters and / or virtual human voice parameters of a virtual human, thereby improving the richness of imaging and the user's experience.

[0143] In some optional implementations, the mapping relationship between voice commands, navigation commands and virtual human imaging parameters can be stored not only in the vehicle computer, but also in the aerial imaging device. Accordingly, the vehicle computer can directly use voice commands, navigation commands and other business commands as specific virtual human action parameters and virtual human voice parameters, and the aerial imaging device will query according to the preset mapping relationship to determine the action that the virtual human needs to perform or the voice to be broadcast. For example, for the business command "Set the air conditioning temperature to 25 degrees", the vehicle computer can package it and send it to the aerial imaging device according to the preset communication protocol. After receiving the communication package, the aerial imaging device decodes it and determines that the action performed by the virtual human is "pointing to the air outlet of the air conditioner with the left hand" according to the preset mapping relationship between the business command and the virtual human action; for another example, for the business command "Play news", the vehicle computer can directly send the business command as a virtual human voice parameter to the aerial imaging device. Next, the aerial imaging device will access the corresponding news APP interface according to the preset mapping relationship between the business command and the virtual human voice to obtain the voice content that the virtual human needs to broadcast.

[0144] In an embodiment of the present invention, by extracting and identifying the communication information of each business module of the vehicle, business instructions that can reflect the driver's intention and the vehicle's driving trend are generated, so as to determine the imaging trigger type, imaging area, imaging volume, virtual human model, and the actions and voices that the virtual human needs to perform and broadcast according to the business instructions, so as to realize the interaction between the virtual human and the user and enhance the driving experience.

[0145] Step S304, byte-encoding the virtual human imaging parameters to generate a communication body, and generating a communication header according to the byte length of the communication body and the version information of the preset communication protocol.

[0146] Specifically, the basic format of the preset communication protocol is communication header + communication body. The communication body is mainly specific business data such as virtual human imaging parameters. These data will actually be compiled into byte streams during the transmission process. The specific business format can be a common data format such as json, xml, etc., or a custom data format, and the embodiments of the present invention are not limited thereto.

[0147] Furthermore, the communication header will at least contain the version information of the communication protocol and the byte length of the communication body, where the version information is mainly used for protocol upgrades and compatibility strategies between various protocol versions. The byte length of the communication body refers to the byte length of specific business data in the communication flow. This data is mainly used for packet sticking and unpacking in the communication flow to avoid reading data that does not belong to this message.

[0148] Step S305, obtaining a communication packet according to the communication header and the communication body, and sending the communication packet to the aerial imaging device according to a preset communication protocol.

[0149] Specifically, the communication packet is synthesized according to the basic format of communication header + communication body. In the subsequent communication process, the vehicle computer or aerial imaging device reads byte data of a specific length from the communication stream according to the byte length of the communication header, and then the business layer decodes the communication body to obtain a common data format (json, xml), thus completing a communication.

[0150] In some optional implementations, the vehicle computer sends a synchronization message to the aerial imaging device for requesting to establish a connection. After receiving a synchronization confirmation message returned by the aerial imaging device based on the synchronization message, the vehicle computer sends a first confirmation message to the aerial imaging device for confirming the establishment of the connection, thereby establishing a connection with the aerial imaging device. Then, the vehicle computer sends a communication packet to the aerial imaging device, and when it detects that the communication packet has been sent, it sends a termination message to the aerial imaging device for requesting to close the connection. Finally, after receiving a second confirmation message sent by the aerial imaging device to indicate that the termination message has been received and a termination confirmation message returned based on the termination message, the vehicle computer sends a third confirmation message to the aerial imaging device for confirming the closure of the connection, thereby disconnecting the connection with the aerial imaging device.

[0151] In this embodiment, the communication standard can be built based on the Transmission Control Protocol (TCP). Figure 6 The following is a basic flow chart of TCP communication. TCP is a connection-oriented, reliable, stream-based communication protocol that is widely used for data transmission on the Internet and local area networks. The principle of using TCP to achieve communication can be divided into the following steps (steps A to C):

[0152] Step A: TCP establishes a connection through a three-way handshake. The process of establishing a TCP connection is called a three-way handshake, and the process is shown in steps a1 to a3:

[0153] Step a1, first handshake (client sends SYN).

[0154] Specifically, the client (ie, the vehicle computer) sends a SYN (Synchronize) message to the server (ie, the aerial imaging device) to indicate that it wishes to establish a connection and sets an initial sequence number (Sequence Number), wherein the message segment: SYN=1, sequence number=x.

[0155] Step a2, second handshake (server responds with SYN-ACK).

[0156] Specifically, after receiving the SYN message from the client, the server replies with a SYN-ACK message (synchronization confirmation message) to indicate that it agrees to establish a connection and also sets an initial sequence number. Among them, the message segment: SYN=1, ACK=1, sequence number=y, confirmation number (Acknowledgment Number)=x+1.

[0157] Step a3, the third handshake (the client responds with ACK).

[0158] Specifically, after receiving the SYN-ACK message from the server, the client replies with an ACK message (the first confirmation message) to confirm the establishment of the connection. Among them, the message segment: ACK = 1, sequence number = x + 1, confirmation number = y + 1. At this point, the three-way handshake is completed and the TCP connection is established.

[0159] Step B, data transmission. After establishing a TCP connection, the client and server can perform data transmission. TCP provides full-duplex communication, and both parties can send and receive data at the same time. The data transmission process includes:

[0160] Step b1, data segmentation and reassembly: the data is divided into multiple segments or communication packets for transmission, and the receiver reassembles the segments in sequence according to the sequence numbers.

[0161] Step b2, flow control: TCP uses a sliding window mechanism to control the sender's data sending speed to prevent network congestion.

[0162] Step b3, reliable transmission: Ensure reliable data transmission through confirmation mechanism (ACK) and retransmission mechanism. The sender waits for the receiver's confirmation after sending the data. If the confirmation is not received within a certain period of time, the sender will retransmit the data.

[0163] Step C, TCP waves four times to disconnect. The process of disconnecting a TCP connection is called four waves, and the steps are as follows:

[0164] Step c1, the first wave (the client sends FIN).

[0165] Specifically, the client sends a FIN (Finish) message (termination message) to indicate that it wishes to close the connection and stop sending data, wherein the message segment includes: FIN = 1, sequence number = u.

[0166] Step c2, wave again (server responds with ACK).

[0167] Specifically, after receiving the FIN message, the server replies with an ACK message (second confirmation message) to confirm receipt of the closing request, wherein the message segment is: ACK=1, sequence number=v, confirmation number=u+1.

[0168] Step c3, the third wave (the server sends FIN).

[0169] Specifically, after the server has processed all the data, it sends a FIN message (termination confirmation message) to indicate that it agrees to close the connection and stop sending data. Among them, the message segment: FIN = 1, sequence number = w.

[0170] Step c4, the fourth wave (the client responds with ACK). After receiving the FIN message from the server, the client replies with an ACK message (the third confirmation message) to confirm the closing request. Among them, the message segment: ACK = 1, sequence number = u+1, confirmation number = w+1. At this point, the four waves are completed and the TCP connection is disconnected.

[0171] In an embodiment of the present application, by sending the communication packet to the aerial imaging device according to a preset communication protocol, the reliability, integrity and accuracy of data transmission are improved, and there is no need to develop different communication standards for different aerial imaging equipment manufacturers, thereby reducing project maintenance costs and development difficulty.

[0172] In an embodiment of the present invention, the virtual human imaging parameters are byte-encoded according to a preset communication protocol to generate a communication body, and a communication header is generated according to the byte length of the communication body and the protocol version information, and a communication packet including the communication header and the communication body is sent to an aerial imaging device, so that the aerial imaging device can unpack the packet, avoid reading erroneous data, and improve the accuracy of information transmission.

[0173] Step S401, receiving the communication packet sent by the vehicle computer. For details, please refer to Figure 3 Step S201 of the illustrated embodiment will not be described in detail here.

[0174] Step S402: parse the communication packet to obtain virtual human imaging parameters. Figure 3 Step S202 of the illustrated embodiment will not be described in detail here.

[0175] Step S403: updating the initial state machine based on the virtual human imaging parameters.

[0176] like Figure 7 As shown, an embodiment of the present invention provides a rendering imaging interface driver layer in an aerial imaging device. The rendering imaging interface driver layer is a three-dimensional graphics rendering library running on a medium-free aerial imaging device, which provides a set of functions for drawing complex graphics in the air.

[0177] Specifically, the rendering imaging interface driver layer is a state machine, and its behavior depends on the current state, where the current state of the state machine includes the current color, current texture, current matrix, etc. These states can be set and modified through related functions. All driver layer states are stored through the context environment, which can be created and managed by the upper-level application, and the imaging rendering area is defined in the context environment.

[0178] Furthermore, the rendering imaging interface driver layer uses a graphics pipeline to process graphics data. This pipeline includes multiple stages, from vertex processing to fragment processing (also called voxel processing). Each stage can be controlled by programming (shader). Its main stages include: defining model vertex data, compiling vertex shaders, coordinate conversion (model space to projection space, rasterization (will generate voxels in projection space, similar to 2D pixels), compiling fragment shaders (mainly calculating lighting and determining the color of each voxel). For details, please refer to the description of related technologies, which will not be repeated here.

[0179] In this embodiment, the initial state machine can be updated using the virtual human imaging parameters and the functions in the three-dimensional graphics rendering library, so that the updated state machine can be used for rendering and imaging.

[0180] Step S404: Utilize the updated state machine to perform model rendering to obtain a target virtual human model.

[0181] Specifically, when using the rendering imaging interface driver layer for rendering imaging, first create and initialize the driver layer context environment. Then, using the virtual human imaging parameters and the functions in the 3D graphics rendering library, update the initial state machine, set the projection area (define the projection position and the size of the stereo), set the projection matrix and the view matrix, define the vertex data (the vertex data defines the shape of the geometric body, which may include information such as position, color, normal, texture coordinates, etc.), then write and compile the shader (the shader is a small program running on the aerial imaging device to control the processing of each voxel), and finally perform the rendering cycle (usually execute rendering commands in a cycle, including clearing the screen, drawing geometric bodies, swapping buffers, etc.), and obtain the target virtual human model.

[0182] It should be noted that the projection matrix is ​​mainly a transformation matrix that projects the 3D model into a specific aerial area. The view matrix can also be divided into translation, rotation and scaling. The view matrix is ​​the inverse matrix of the model matrix of the observer's transformation in the world after the observer is regarded as a model. If the observer's coordinate translation is (tx, ty, tz), the view matrix is ​​as follows. It can be seen that if the view matrix is ​​regarded as the model matrix of the entire world, it is equivalent to the entire world being translated by (-tx, -ty, -tz):

[0183]

[0184] If the observer rotates around the z-axis by an angle θ, the view matrix is ​​as follows, which is equivalent to the entire world rotating around the z-axis by -θ degrees:

[0185]

[0186] If the observer is proportionally reduced by s times in three directions, the view matrix is ​​as follows, which is equivalent to magnifying the entire world by s times:

[0187]

[0188] The embodiment of the present invention utilizes the virtual human imaging parameters sent by the vehicle computer to update the state of the initial state machine of the aerial imaging device, so that the updated state machine can perform graphics drawing and projection according to the virtual human imaging parameters, with good rendering effect. There is no need for the vehicle computer to perform imaging rendering, thereby reducing the energy consumption of the vehicle computer.

[0189] In some optional implementations, the updated state machine may be used to perform geometry rendering, texture mapping, light and shadow processing, color processing, depth testing, and template testing to obtain a target virtual human model.

[0190] Specifically, geometry drawing is the basic drawing commands provided by the rendering imaging interface driver layer, such as drawArrays and drawElements, which are used to draw geometric bodies such as points, lines, and triangles; texture mapping is used to map images on the surface of geometric bodies, and it supports a variety of texture operations, including texture loading, texture binding, setting texture parameters, texture filtering, etc.; light and shadow processing uses shaders and lighting models (such as the Phong model) to achieve realistic lighting and shadow effects; color processing is used to control translucent effects, which is achieved by mixing different color values; frame buffer (FBO) is used for off-screen rendering, allowing the rendering results to be stored in textures or rendering buffers instead of directly displaying them on the imaging area; depth testing is used to hide occluded objects; template testing is used for complex clipping and blending effects.

[0191] For example, if you need to render a human figure, you can follow the steps below:

[0192] Step d1, first provide a 3D model file .fbx, .obj, .gltf, etc. These files are composed of many three-dimensional points (Vector3), which constitute the outline of the character image and are necessary input for the geometry drawing process.

[0193] Step d2, provide several image textures (basic color map, normal map, roughness map, ...). The basic color map is used to determine the color of the character model, such as black hair, yellow skin, etc. The normal map is used to represent the surface concavity and convexity of the model, lighting, etc. The roughness map indicates where the surface of the model is smooth and where it is rough. Using texture mapping, these texture maps can be perfectly covered on the model surface. These maps are optional inputs of the rendering pipeline to improve the realism of the rendering results.

[0194] Step d3, after receiving the above input, the aerial imaging device will perform lighting calculations on the virtual human model according to the algorithm that generates lighting and shadows (such as the PBR algorithm, the Phone algorithm), and obtain the lighting color of each voxel in the imaging area (which can be understood as the color we see in the end), producing the same lighting effect as in reality. After the lighting calculation, each voxel in the imaging area has a color, and some culling may be required through depth testing and template testing. For example, when observing from the front of the person, the occluded voxels such as the internal organs and the back cannot be seen. These colors need to be culled to prevent interference with the rendering results, and on the other hand, it can also save performance.

[0195] Step d4, through the above steps, a model with a three-dimensional rendering effect will be obtained, or a lot of 3D voxel color arrays. These color arrays cannot be projected directly, but are first stored in a frame buffer (FBO). When the imaging device refreshes the imaging area next time, the content of the frame buffer is projected, which can reduce the flickering problem of the image and obtain a stable rendering effect. In this way, the aerial imaging device completes the rendering imaging process.

[0196] In the embodiment of the present application, the updated state machine is used to draw the geometry to obtain the outline of the virtual person, and then texture mapping is performed to enrich the color, concavity, roughness, etc. of the virtual person, which is conducive to improving the authenticity of the virtual person. Then light and shadow processing and color processing are performed to determine the illumination color of each voxel in the imaging area. Finally, the obtained rendering effect is stored in the frame buffer before subsequent projection, which reduces the flickering problem of the image and improves the stability of the rendering. In addition, noise and interference factors are filtered out through depth testing and template testing to further improve the authenticity of the rendering.

[0197] Step S405: Project the target virtual human model to generate a vehicle-mounted virtual human.

[0198] Specifically, the aerial imaging device projects the target virtual human model to the target area according to the imaging area, imaging volume and other parameters to generate the vehicle-mounted virtual human. The aerial imaging device can be mounted on the co-pilot seat, rear seat and / or driving platform of the vehicle, etc. Figure 7The vehicle computer can generate a communication packet containing virtual human imaging parameters based on the communication information of different business modules, and send it to the rendering imaging interface of multiple aerial imaging devices according to the standard TCP communication protocol. Each aerial imaging device uses its own rendering imaging interface driver layer to perform imaging rendering, thereby projecting the on-board virtual human in different areas of the vehicle, such as the co-pilot seat and the back row.

[0199] Step S406, based on the action parameters of the virtual human and / or the voice parameters of the virtual human, a target action and / or a target voice is obtained, and the in-vehicle virtual human is controlled to perform the target action and / or broadcast the target voice.

[0200] For example, the car voice module will collect the owner's speech content through the microphone, and recognize the owner's intention through the AI ​​voice recognition module. For example, if the owner says "set the air conditioning temperature to 25 degrees", the business layer of the car voice module will generate a communication body consisting of a communication header + a byte array of "set the air conditioning temperature to 25 degrees", and then call the communication interface to send this message to the aerial imaging device. The aerial imaging device will receive and unpack it, decode it to obtain the specific executable message content. According to product requirements, this message will make the virtual person play a gesture of pointing to the air outlet of the air conditioner with the left hand and showing the OK gesture with the right hand, indicating that the operation has been completed, giving the owner a concrete interactive feedback.

[0201] For another example, when driving, the car relies on many sensors and cameras to perceive the surrounding environment, such as a simple case of turning right at the intersection ahead. The navigation module recognizes in advance that it will turn right at the next intersection based on the car's position and the map navigation route. The virtual human needs to play specific actions and sounds to prompt the owner to change to the right lane in advance. Then the car navigation module will generate a communication header + "turn right at the intersection ahead" communication body. The car navigation business layer calls the communication interface to send this message to the aerial imaging device. The aerial imaging device on the server receives the message and decodes it to obtain the intention to play specific actions and sounds, and performs rendering to complete this interaction.

[0202] In this embodiment, by controlling the in-vehicle virtual human to perform actions and / or broadcast voices according to the virtual human action parameters and / or virtual human voice parameters, the interaction process between the virtual human and the driver and passengers is realized, thereby improving the vehicle experience and immersion of the driver and passengers.

[0203] In some optional implementations, when the communication packet sent by the vehicle computer is not received within a preset time, the onboard virtual person is controlled to perform a preset standby action and / or broadcast a preset standby voice. For example, if the imaging trigger mechanism of the aerial imaging device is always displayed, that is, the vehicle computer aerial system will always project when the vehicle is started, if the vehicle computer does not send a new communication packet within a preset time, that is, if the virtual person does not interact with the user, then the virtual person is controlled to enter a leisure state, and the onboard virtual person is controlled to perform some leisure standby actions, and the standby voice can also be played to further improve the realism of the onboard virtual person.

[0204] In some optional implementations, the aerial imaging device may also obtain new virtual human imaging parameters based on a new communication packet sent by the vehicle computer, and adjust the onboard virtual human based on the new virtual human imaging parameters.

[0205] Specifically, the projection duration of the in-vehicle virtual person can be adjusted according to the new imaging trigger type. The imaging trigger types mainly include permanent projection and on-demand projection. The projection duration corresponding to permanent projection is the time from when the vehicle computer is started to when the vehicle computer is turned off. That is, when the vehicle computer is started, the aerial imaging device will continue to run and image until the vehicle computer is turned off, while on-demand projection allows users to set the projection duration and projection trigger timing by themselves. In addition, the projection position of the in-vehicle virtual person can also be adjusted according to the new imaging area.

[0206] In the embodiment of the present invention, the user is allowed to adjust the projection duration and projection position of the in-vehicle virtual person, thereby improving the user experience.

[0207] For example, the car computer will request the aerial imaging device to adjust parameters such as the imaging size and the displayed virtual human model, and will also issue new business instructions to adjust the motion parameters and voice parameters of the virtual human. After receiving the new communication package sent by the car computer, the aerial imaging device will make adjustments based on the corresponding parameters to achieve real-time adjustment of the virtual human and enhance the user experience.

[0208] The vehicle-mounted virtual human generation method provided in this embodiment targets the pain points of traditional virtual humans and uses medium-free aerial imaging to realize a virtual human driving assistant. The present invention customizes the standard communication protocol between the aerial imaging device and the vehicle computer and the rendering imaging interface running on the vehicle computer end and the aerial imaging end through the TCP protocol. The vehicle computer is responsible for collecting communication information such as data and status of various business modules of the car, and transmits the collected data and status to the aerial imaging device through the standard communication protocol. The aerial imaging device displays images in a specific area through the rendering imaging interface, such as Figure 8 and Fig. 9 And the vehicle computer can directly call the rendering imaging interface to directly control the aerial imaging device to display images or change the image status.

[0209] The present invention also provides an SDK for upper layer application development, such as Fig.10 As shown in the figure, this part is in the JNI interface layer, which is developed entirely in C++. It not only meets the needs of the upper-layer application for high-speed interface calls, but also makes it easier to call the hardware capabilities of the aerial imaging device (because the underlying capabilities provided by the imaging device manufacturer are usually provided to the outside in C++). The JNI interface layer will include the implementation of the communication standard and the calls of most of the interfaces of the rendering layer, and encapsulate the interfaces required for the upper-layer application development.

[0210] The present invention provides an SDK for upper-layer application development, which mainly integrates the capabilities of communication standards and rendering interfaces. In this way, when developing the upper layer, there is no need to worry about how to work with the principles of aerial imaging equipment, but only needs to focus on development business needs, which can greatly improve development efficiency.

[0211] The present invention provides a set of standard rendering and imaging interfaces, which are mainly used to bridge the driver layer of the upper application call requirements and the rendering capabilities of the aerial imaging device, so that the devices produced by different aerial imaging manufacturers can use this set of standards to render aerial images. In the present invention, the aerial imaging device is mounted to the area where the cockpit needs to be projected. The automobile manufacturer can reasonably arrange the aerial imaging device to the correct position according to the layout of the vehicle cockpit, and provide a projection position fine-tuning function.

[0212] Compared with traditional technologies, the present invention does not need to consume the rendering performance of the vehicle computer. The vehicle computer only needs to be responsible for the communication with the aerial imaging equipment and the interactive development of the rendering effect. The traditional rendering of virtual people needs to be rendered on the vehicle computer, which consumes a lot of performance. In addition, the interaction of the virtual people in the present invention can reach any object in reality, while the traditional virtual people only exist on the vehicle computer screen and cannot leave the screen, so they cannot interact with objects in reality.

[0213] The embodiment of the present invention further provides a vehicle, such as Fig.11 As shown, the vehicle has Figure 1 The in-vehicle virtual human generation system 100 is shown.

[0214] See also Fig.12 , Fig.12 is a structural schematic diagram of a vehicle computer provided by an optional embodiment of the present invention, such as Fig.12As shown, the vehicle computer includes: one or more first processors 10, a first memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed in the vehicle computer, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple devices can be connected, each device providing part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.12 A first processor 10 is taken as an example.

[0215] The first processor 10 may be a central processing unit, a network processor or a combination thereof. The first processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0216] The first memory 20 stores instructions executable by at least one first processor 10, so that the at least one first processor 10 executes the method shown in the above embodiment.

[0217] The first memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created according to the use of the vehicle computer, etc. In addition, the first memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the first memory 20 may optionally include a memory remotely arranged relative to the first processor 10, and these remote memories may be connected to the vehicle computer via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0218] The first memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the first memory 20 may also include a combination of the above types of memory.

[0219] The vehicle computer further includes a first communication interface 30, which is used for the vehicle computer to communicate with other devices or a communication network.

[0220] See also Fig.13 , Fig.13 is a schematic diagram of the structure of an aerial imaging device provided by an optional embodiment of the present invention, such as Fig.13 As shown, the vehicle computer includes: one or more second processors 40, a second memory 50 and a second communication interface 60. For details about the specific working principles of the second processor 40, the second memory 50 and the second communication interface 60, please refer to the description of the above vehicle computer embodiment, which will not be repeated here.

[0221] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0222] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0223] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for generating a vehicle-mounted virtual human, characterized in that: Applied to a vehicle computer, the method includes: Obtaining communication information generated by at least one business module in the vehicle computer; Based on the communication information, determining virtual human imaging parameters; A communication packet is generated based on the virtual human imaging parameters, and the communication packet is sent to an aerial imaging device, so that the aerial imaging device parses the communication packet to obtain the virtual human imaging parameters, and renders based on the virtual human imaging parameters to generate a vehicle-mounted virtual human.

2. The method according to claim 1, characterized in that The determining of virtual human imaging parameters based on the communication information includes: Filtering the communication information to obtain interactive data sent by at least one business module; wherein the business module includes an aerial imaging device setting module, a voice module and a navigation module; According to the business instructions corresponding to the interaction data, virtual human imaging parameters are obtained; wherein the virtual human imaging parameters include imaging trigger type, imaging area, imaging volume, virtual human model parameters, virtual human action parameters and / or virtual human voice parameters.

3. The method according to claim 2, characterized in that The business instruction includes an imaging parameter setting instruction, the virtual human action parameter includes an initial action of the virtual human set by a user, and the virtual human voice parameter includes an initial voice of the virtual human set by a user; and obtaining the virtual human imaging parameter according to the business instruction corresponding to the interaction data includes: Obtaining, according to the interactive data sent by the aerial imaging device setting module, an imaging parameter setting instruction issued by the user for the aerial imaging device; According to the imaging parameter setting instruction, the imaging trigger type, imaging area, imaging volume, virtual human model parameters, virtual human initial action and / or virtual human initial voice set by the user for the virtual human are obtained.

4. The method according to claim 3, characterized in that The business instruction also includes a voice instruction, the virtual human action parameter also includes a target action corresponding to the voice instruction, and the virtual human voice parameter also includes a target voice corresponding to the voice instruction; the obtaining of the virtual human imaging parameter according to the business instruction corresponding to the interaction data also includes: Extract and identify the interactive data sent by the voice module to obtain the voice command issued by the user; According to the preset mapping relationship between the voice command and the virtual human imaging parameter, the target action and / or target voice corresponding to the voice command is confirmed.

5. The method according to claim 3, characterized in that: The business instruction also includes a navigation instruction, the virtual human action parameter also includes a target action corresponding to the navigation instruction, and the virtual human voice parameter also includes a target voice corresponding to the navigation instruction; the step of obtaining the virtual human imaging parameter according to the business instruction corresponding to the interaction data also includes: Obtaining navigation instructions according to the interactive data sent by the navigation module; According to the preset mapping relationship between the navigation instruction and the virtual human imaging parameter, the target action and / or target voice corresponding to the navigation instruction is confirmed.

6. The method according to any one of claims 1 to 5, characterized in that The generating of a communication packet based on the virtual human imaging parameters and sending the communication packet to an aerial imaging device comprises: Performing byte encoding on the virtual human imaging parameters to generate a communication body, and generating a communication header according to the byte length of the communication body and version information of a preset communication protocol; A communication packet is obtained according to the communication head and the communication body, and the communication packet is sent to the aerial imaging device according to a preset communication protocol, so that the aerial imaging device parses the communication packet to obtain virtual human imaging parameters, updates the initial state machine based on the virtual human imaging parameters, uses the updated state machine to perform model rendering to obtain a target virtual human model, and projects the target virtual human model to generate a vehicle-mounted virtual human.

7. The method according to claim 6, characterized in that The step of sending the communication packet to the aerial imaging device according to a preset communication protocol includes: Sending a synchronization message for requesting to establish a connection to the aerial imaging device; After receiving the synchronization confirmation message returned by the aerial imaging device based on the synchronization message, sending a first confirmation message for confirming the establishment of the connection to the aerial imaging device to establish a connection with the aerial imaging device; sending the communication packet to the aerial imaging device; When it is detected that the communication packet has been sent, a termination message for requesting to close the connection is sent to the aerial imaging device; After receiving the second confirmation message sent by the aerial imaging device to indicate the receipt of the termination message and the termination confirmation message returned based on the termination message, a third confirmation message for confirming the closing of the connection is sent to the aerial imaging device to disconnect the connection with the aerial imaging device.

8. A method for generating a vehicle-mounted virtual human, characterized in that: Applied to aerial imaging equipment, the method comprises: Receiving a communication packet sent by the vehicle computer; wherein the communication packet is generated by the vehicle computer by acquiring communication information generated by at least one business module and based on the virtual human imaging parameters corresponding to the communication information; Parsing the communication packet to obtain virtual human imaging parameters; Rendering is performed based on the virtual human imaging parameters to generate a vehicle-mounted virtual human.

9. The method according to claim 8, characterized in that The rendering based on the virtual human imaging parameters to generate a vehicle-mounted virtual human includes: Based on the virtual human imaging parameters, the initial state machine is updated; wherein the virtual human imaging parameters include imaging trigger type, imaging area, imaging volume, virtual human model parameters, virtual human motion parameters and / or virtual human voice parameters; The updated state machine is used to render the model to obtain the target virtual human model; The target virtual human model is projected to generate a vehicle-mounted virtual human.

10. The method according to claim 9, characterized in that The method of using the updated state machine to render the model to obtain the target virtual human model includes: The updated state machine is used to perform geometry drawing, texture mapping, light and shadow processing, color processing, depth testing, and template testing to obtain the target virtual human model.

11. The method according to claim 9 or 10, characterized in that: After projecting the target virtual human model to generate a vehicle-mounted virtual human, the method further includes: Based on the virtual human action parameters and / or the virtual human voice parameters, obtaining a target action and / or a target voice; Control the in-vehicle virtual person to perform the target action and / or broadcast the target voice.

12. The method according to claim 9 or 10, characterized in that: After projecting the target virtual human model to generate a vehicle-mounted virtual human, the method further includes: When no communication packet sent by the vehicle computer is received within a preset time, the vehicle-mounted virtual person is controlled to perform a preset standby action and / or broadcast a preset standby voice.

13. The method according to claim 9 or 10, characterized in that: After projecting the target virtual human model to generate a vehicle-mounted virtual human, the method further includes: According to the new communication packet sent by the vehicle computer, new virtual human imaging parameters are obtained; The vehicle-mounted virtual human is adjusted according to the new virtual human imaging parameters.

14. The method according to claim 13, characterized in that The step of adjusting the vehicle-mounted virtual human according to the new virtual human imaging parameters includes: According to the new imaging trigger type, adjusting the projection duration of the vehicle-mounted virtual person; And / or, adjusting the projection position of the vehicle-mounted virtual person according to the new imaging area.

15. A vehicle-mounted virtual human generation system, characterized in that: include: Car and aerial imaging equipment; The vehicle computer obtains communication information generated by at least one service module in the vehicle computer; Based on the communication information, determining virtual human imaging parameters; Generate a communication packet based on the virtual human imaging parameters, and send the communication packet to an aerial imaging device; The aerial imaging device receives the communication packet sent by the vehicle computer; parses the communication packet to obtain the virtual human imaging parameters; and renders based on the virtual human imaging parameters to generate the vehicle-mounted virtual human.

16. The system according to claim 15, characterized in that The aerial imaging device is mounted on the co-pilot seat, the rear seat and / or the driving console of the vehicle.

17. A vehicle, characterized in that: A vehicle-mounted virtual human generation system comprising any one of claims 15 to 16.

18. A vehicle computer, characterized in that: The vehicle computer includes a first memory and a first processor; The first memory and the first processor are communicatively connected to each other, the first memory stores a first computer instruction, and the first processor executes the in-vehicle virtual human generation method according to any one of claims 1 to 7 by executing the first computer instruction.

19. An aerial imaging device, characterized in that: The aerial imaging device includes a second memory and a second processor; The second memory and the second processor are communicatively connected to each other, the second memory stores second computer instructions, and the second processor executes the in-vehicle virtual human generation method according to any one of claims 8 to 14 by executing the second computer instructions.

Images