In-vehicle human-machine interaction methods, devices, equipment and storage media

Abstract

This application discloses an in-vehicle human-machine interaction method, device, equipment, and storage medium, belonging to the field of intelligent vehicle technology. An in-vehicle human-machine interaction method includes: acquiring a target image; inputting the target image into an artificial intelligence (AI) model to generate a 3D model of the target element; and rendering the 3D model of the target element using a rendering engine based on the environmental information of the target vehicle's human-machine interface, so as to display the 3D visual image of the target element within a preset area of ​​the human-machine interface. The in-vehicle human-machine interaction solution provided by this application can generate a 3D model of the target element based on an AI model and display it on the human-machine interface, thereby improving the driving experience.

Description

Technical Field

[0001] This application belongs to the field of intelligent vehicle technology, specifically relating to an in-vehicle human-machine interaction method, device, equipment, and storage medium. Background Technology

[0002] Artificial intelligence (AI) is a branch of computer science that aims to develop systems or programs that can simulate human intelligence. Its core goal is to enable machines to learn, reason, perceive, understand, and make decisions, in order to perform tasks that typically require human intelligence.

[0003] With the rapid development of AI technology, its applications are becoming increasingly widespread, such as in healthcare, education, and office work, helping to improve the intelligence level and service efficiency of various industries. Currently, against the backdrop of the rapid evolution of intelligent connected vehicles, the introduction of AI technology into in-vehicle human-machine interaction is receiving increasing attention.

[0004] Therefore, how to stably and efficiently integrate the content generation and cognitive capabilities of AI models into in-vehicle human-machine interaction to improve the driving experience is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide an in-vehicle human-computer interaction method, device, equipment, and storage medium to stably and efficiently integrate the content generation and cognitive ability of the AI ​​model into the in-vehicle human-computer interaction, thereby improving the driving experience.

[0006] To solve the above-mentioned technical problems, this application provides the following technical solution: Firstly, a vehicle-mounted human-machine interaction method is provided, including: Acquire a target image, wherein the target image contains target elements; The target image is input into an artificial intelligence (AI) model to generate a three-dimensional model of the target element; The rendering engine renders the three-dimensional model of the target element based on the environmental information of the target vehicle's human-machine interface, so as to display the three-dimensional visual image of the target element within a preset area of ​​the human-machine interface.

[0007] Optionally, rendering the 3D model of the target element using a rendering engine based on the environmental information of the target vehicle's human-machine interface includes: The 3D model of the target element is analyzed by the rendering engine to obtain the vertex data of the target element; Based on the vertex data, a mesh is constructed; The mesh is rendered based on the environmental information of the target vehicle's human-machine interface.

[0008] Optionally, rendering the mesh based on the environmental information of the target vehicle's human-machine interface includes: According to the preset material, the obtained texture is applied to the mesh; Bind the lighting parameters in the environmental information of the target vehicle's human-machine interface to the material associated with the target element.

[0009] Optionally, binding the lighting parameters in the environmental information of the target vehicle's human-machine interface to the material associated with the target element includes: Based on the illumination parameters in the environmental information of the target vehicle's human-machine interface and the vertex data, calculate the ambient light component, diffuse reflection component and specular highlight component. The ambient light component, diffuse component, and specular highlight component are superimposed on the texture color of the target element in the material associated with the target element.

[0010] Optionally, the calculation of ambient light component, diffuse component, and specular highlight component based on the illumination parameters in the environmental information of the target vehicle's human-machine interface and the vertex data includes: In the fixed-function pipeline, the vertex shader performs spatial transformation on the vertex data to obtain interpolated vertex data, and inputs the interpolated vertex data into the fragment shader. In the fragment shader, based on the lighting parameters in the environmental information of the target vehicle's human-machine interface and the interpolated vertex data, the ambient light component, diffuse component, and specular highlight component are calculated. Alternatively, the calculation of ambient light components, diffuse components, and specular highlight components based on the illumination parameters in the environmental information of the target vehicle's human-machine interface and the vertex data includes: In the programmable pipeline, the vertex shader obtains vertex attributes based on the vertex data, performs custom rendering calculations on the vertex attributes through programming to complete the spatial transformation, and passes the spatial transformation results to the fragment shader. In the fragment shader, based on the lighting parameters in the environmental information of the target vehicle's human-machine interface, the ambient light component, diffuse component, and specular highlight component are calculated through the lighting model.

[0011] Optionally, the method further includes: In response to an interactive command for a 3D visual image of the target element, the spatial transformation information associated with the target element is adjusted to control the rotation, scaling, or displacement of the 3D visual image of the target element.

[0012] Optionally, the method further includes: In response to an activity command for the three-dimensional visual image of the target element, the three-dimensional visual image of the target element is driven to move.

[0013] Optionally, driving the 3D visual image of the target element to move includes: Obtain target skeletal animation data that matches the type of the target element, with each skeletal animation data corresponding to multiple frames of skeletal transformation images; The multi-frame skeletal transformation images corresponding to the target skeletal animation data are passed into the material associated with the target element; Based on the data recorded in each frame of the skeleton transformation image corresponding to the target skeleton animation data, the three-dimensional visual image of the target element is driven to move.

[0014] Optionally, the target image is an image provided by the user or selected from an in-vehicle image library, or the target image is an image of the environment surrounding the target vehicle acquired in real time.

[0015] Optionally, when the target image is an image provided by the user or selected from the in-vehicle image library, the scene in which the three-dimensional visual image of the target element is located is determined based on at least one of the real-time geographical location data of the target vehicle, user operation habit data, user instructions, and images provided or selected by the user.

[0016] Optionally, when the target image is an image of the environment surrounding the target vehicle acquired in real time, the target element is an element in the target image that is not fully represented by a preset 3D model. The step of rendering the 3D model of the target element using a rendering engine based on the environmental information of the target vehicle's human-machine interface includes: The rendering engine performs spatial alignment and fusion rendering of the 3D model of the target element and the basic scene constructed based on the target image and the preset 3D model, according to the environmental information of the target vehicle's human-machine interface.

[0017] Optionally, the method further includes: Based on the three-dimensional model of the target element, determine at least one of the following: type, size, or spatial location of the target element; Determine whether to output an alarm message based on at least one of the target element's type, size, or spatial location.

[0018] Secondly, an in-vehicle human-machine interaction device is provided, comprising: The acquisition module is used to acquire a target image, wherein the target image contains target elements; The generation module is used to input the target image into the artificial intelligence (AI) model to generate a three-dimensional model of the target element; The rendering module is used to render the three-dimensional model of the target element based on the environmental information of the target vehicle's human-machine interface through the rendering engine, so as to display the three-dimensional visual image of the target element in a preset area of ​​the human-machine interface.

[0019] Thirdly, an apparatus is provided, the apparatus comprising: Memory, used to store computer programs; A processor, configured to implement the steps of the method as described in the first aspect when executing the computer program; or, The device includes the in-vehicle human-machine interaction device as described in the second aspect.

[0020] Fourthly, a readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0021] Fifthly, a program product is provided, the program product including instructions stored in a readable storage medium and adapted to be read and executed by a processor to cause a device having the processor to perform the steps of the method as described in the first aspect.

[0022] By applying the technical solution provided in the embodiments of this application, after obtaining the target image, the target image is input into the AI ​​model to generate a three-dimensional model of the target element in the target image. The three-dimensional model of the target element is rendered by the rendering engine according to the environmental information of the human-machine interface of the target vehicle, so as to display the three-dimensional visual image of the target element in the preset area of ​​the human-machine interface. Different target images will result in different three-dimensional visual images of the target element. By using AI technology and rendering engine, the human-machine interface is no longer just a preset interface, but can present a dynamically generated interface, realizing the transformation from static preset to dynamic generation. This allows the content generation and cognitive ability of the AI ​​model to be stably and efficiently integrated into the in-vehicle human-machine interaction, thereby improving the driving experience.

[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1This is a flowchart illustrating the implementation of an in-vehicle human-machine interaction method according to an embodiment of this application. Figure 2 This is a schematic diagram of the three-dimensional model rendering process in an embodiment of this application; Figure 3 This is a schematic diagram of the fragment color determination process in an embodiment of this application; Figure 4 This is a schematic diagram of a three-dimensional VPA in an embodiment of this application; Figure 5 This is a schematic diagram of a three-dimensional scene in an embodiment of this application; Figure 6 This is a schematic diagram showing the virtual image before and after rotation in an embodiment of this application; Figure 7 This is a schematic diagram obtained after fusion rendering in an embodiment of this application; Figure 8 This is another schematic diagram obtained after fusion rendering in the embodiments of this application; Figure 9 This is a schematic diagram of a processing architecture in an embodiment of this application; Figure 10 This is a schematic diagram of a processing flow in an embodiment of this application; Figure 11 This is a schematic diagram of the structure of an in-vehicle human-machine interaction device according to an embodiment of this application; Figure 12 This is a schematic diagram of the structure of a device according to an embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0028] The following description, in conjunction with the accompanying drawings, details a vehicle-mounted human-computer interaction method provided by this application through specific embodiments and application scenarios.

[0029] See Figure 1 The diagram shown is an implementation flowchart of an in-vehicle human-machine interaction method provided in this application embodiment. The method may include the following steps: S110: Acquire the target image.

[0030] The target image contains target elements.

[0031] In this embodiment, the target image can originate from various sources. Optionally, the target image can be an image provided or selected by the user, such as a photo uploaded by the user via a mobile terminal or vehicle system containing elements such as specific items, people, pets, or dolls, or a picture selected from the vehicle's photo album. Optionally, the target image can be an image of the surrounding environment of the target vehicle acquired in real time, such as an image of the surrounding environment of the target vehicle acquired in real time by a vehicle-mounted image acquisition device (such as a front-view camera, surround-view camera, etc.), which includes elements such as roads, vehicles, traffic signs, pedestrians, green belts, and temporary or sudden obstacles (such as fallen goods, rolling balls, temporary construction facilities, etc.). Target elements can be one or more elements contained in the target image.

[0032] S120: Input the target image into the artificial intelligence (AI) model to generate a 3D model of the target elements.

[0033] In this embodiment of the application, the AI ​​model can be pre-trained and has the ability to understand content and generate a 3D model based on input data.

[0034] After acquiring the target image, it can be input into an AI model. The AI ​​model analyzes the input image and identifies the target elements that need to be modeled. For example, for a user-uploaded pet photo, the target element could be the pet in the photo. For real-time environmental images, the target element could be an element that is not fully represented by a pre-defined 3D model. The AI ​​model can then generate a 3D model of the target element.

[0035] S130: The rendering engine renders the 3D model of the target element based on the environmental information of the target vehicle's human-machine interface, so as to display the 3D visual image of the target element in the preset area of ​​the human-machine interface.

[0036] After generating a 3D model of the target element using an AI model, the 3D model can be further rendered using a rendering engine to display a 3D visual image of the target element within a preset area of ​​the target vehicle's Human Machine Interface (HMI). The rendering engine can render the 3D model of the target element based on the environmental information of the HMI, ensuring that the lighting and shadow effects of the rendered 3D visual image match the environmental information. Generating a 3D model of the target element based on an AI model and displaying it on the HMI helps improve the driving experience.

[0037] Optionally, the rendering engine can be the Kanzi User Interface (UI) rendering engine. The Kanzi UI rendering engine is an advanced toolchain focused on the development of automotive human-machine interfaces. It mainly provides graphics rendering and interaction solutions for smart cockpits and multi-screen digital cockpits. It has high-fidelity graphics rendering capabilities and supports smooth two-dimensional (2D) / three-dimensional (3D) dynamic effects. Targeting the needs of the automotive industry, it can optimize UI animation response speed, improve startup efficiency, and enhance system stability.

[0038] The human-machine interface (HMI) can be a central control screen, dashboard, or augmented reality head-up display (ARHUD). Displaying a 3D visual image of a target element within a preset area of ​​the HMI allows the user to see a three-dimensional element with a sense of depth and realism. For example, an interactive 3D virtual pet image generated from a user-uploaded photo can be presented within the preset area of ​​the HMI, or 3D virtual roadblocks representing temporary obstacles can be overlaid on a real-time video stream or map displayed on the HMI.

[0039] The environmental information of the human-computer interface can include lighting parameters, reflection parameters (such as reflection intensity), and environment maps. Based on its own rendering capabilities and the environmental information of the current human-computer interface, the rendering engine can render the 3D model of the target element and automatically generate shadows. The current human-computer interface can be the default interface, an interface generated based on an image uploaded or selected by the user, or an interface generated based on the currently defined scene.

[0040] The preset area used to display the 3D visual image of the target element can be pre-marked by the user on the human-machine interface. Correspondingly, the rendering engine needs to consider parameters such as lighting and reflection within the preset area during rendering. Alternatively, the preset area can be a default setting, such as one configured at the factory. Users can adjust the preset area. For example, a user can define an area (such as a circle or square) by touching the screen and then adjust its size and / or position by dragging. Alternatively, a user can make gestures in the air (such as drawing circles, squares, or irregular shapes), and the in-vehicle system can recognize the gestures and generate a preset area based on the shape of the gesture.

[0041] A target element can include one or more elements. Using an AI model, 3D models of different elements can be generated separately. If the target element includes multiple elements, when rendering the 3D model of the target element, their 3D models can be integrated into the same scene for rendering based on the spatial relationships, semantic associations, or user instructions of the multiple elements in the target image, thereby constructing a scene containing multiple related elements. Alternatively, if the target element includes multiple elements, when rendering the 3D model of the target element, the 3D model of each element can be rendered independently, and the user can operate on them individually.

[0042] By applying the method provided in this application embodiment, after obtaining the target image, the target image is input into the AI ​​model to generate a three-dimensional model of the target element in the target image. The three-dimensional model of the target element is rendered by the rendering engine according to the environmental information of the human-machine interface of the target vehicle, so as to display the three-dimensional visual image of the target element in a preset area of ​​the human-machine interface. Different target images will result in different three-dimensional visual images of the target element. By using AI technology and rendering engine, the human-machine interface is no longer just a preset interface, but can present a dynamically generated interface, realizing the transformation from static preset to dynamic generation. This allows the content generation and cognitive ability of the AI ​​model to be stably and efficiently integrated into the in-vehicle human-machine interaction, thereby improving the driving experience.

[0043] In some embodiments of this application, rendering a 3D model of a target element using a rendering engine based on the environmental information of the target vehicle's human-machine interface may include the following steps: The 3D model of the target element is analyzed by the rendering engine to obtain the vertex data of the target element; A mesh is constructed based on vertex data; The mesh is rendered based on the environmental information of the target vehicle's human-machine interface.

[0044] For ease of description, the above steps will be combined for explanation.

[0045] In this embodiment, after generating a 3D model of the target element using an AI model, the 3D model of the target element can be parsed by a rendering engine to obtain the vertex data of the target element. The vertex data is the basic data used to describe the 3D shape of the target element, including vertex coordinates in 3D space, vertex connection relationships, etc.

[0046] Based on the vertex coordinates and vertex connectivity data, vertices can be connected to form triangular faces, resulting in multiple triangular faces. These multiple triangular faces together construct a complete mesh representing the outer surface of the target element. The mesh can then be rendered based on the environmental information of the target vehicle's human-machine interface.

[0047] Optionally, the rendering engine can parse the 3D model of the target element to obtain at least one image resource, or it can obtain at least one image resource based on the input image. Image resources are data used to define the surface appearance of the target element, such as base color and normals.

[0048] Textures can be generated from at least one image resource. Optionally, textures can be generated by calling the OpenGL (Graphics Library, GL) standard interface. Rendering is then performed using the texture according to the preset material.

[0049] like Figure 2 As shown, the 3D model of the target element generated using the AI ​​model is a GL Transmission Format Binary (GLB) format 3D model. Parsing the 3D model according to the GLB format yields the vertex data of the target element and at least one image-type resource, such as base color and normals. Based on the vertex data, multiple triangles are obtained, such as triangle 1, ..., triangle N. These triangles are used to construct a mesh. Based on at least one image-type resource, a texture is generated. The mesh is then rendered using OpenGL, and the texture is applied according to the material to achieve the rendering of the 3D model.

[0050] The above operations enable the 3D models generated using AI models to be rendered efficiently and stably.

[0051] In some embodiments of this application, rendering the mesh based on the environmental information of the target vehicle's human-machine interface may include the following steps: Apply the obtained texture to the mesh according to the preset material; Bind the lighting parameters from the environmental information of the target vehicle's human-machine interface to the material associated with the target element.

[0052] For ease of description, the above steps will be combined for explanation.

[0053] In this embodiment, the scene in which the three-dimensional visual image of the target element is located can be determined based on at least one of the following: real-time geographical location data of the target vehicle, user operation habit data, user instructions, or images provided or selected by the user. This scene can be a two-dimensional scene or a three-dimensional scene.

[0054] The real-time geographic location data of the target vehicle can be obtained through the in-vehicle Global Positioning System (GPS), the BeiDou Navigation Satellite System, or a fusion positioning technology combining cellular networks and Inertial Measurement Units (IMUs). This data may include, but is not limited to, latitude and longitude coordinates, altitude, road information, points of interest (such as company premises, residential areas, commercial centers, campsites, etc.), and area information. User operation habit data can be extracted from the background logs of the intelligent cockpit system. This data may include, but is not limited to, high-frequency operation data (such as navigating to location X, playing news, etc.), operation data with time patterns (such as users typically activating "camping mode" on rest days), and preference settings data (such as preferred music genres, ambient light colors, etc.). User commands refer to real-time commands input by the user through voice, gestures, or touch. For example, a user might input "I want to see the scenery by the sea" via voice or click the "Generate Camping Mode" control in the in-vehicle human-machine interface of the intelligent cockpit system. The images provided or selected by the user can be images uploaded by the user through a mobile application, or images selected by the user in the car's photo album, such as a seaside landscape photo or a picture of relaxing at home.

[0055] The above data can be used independently or combined to define a scene. This scene may include 3D visual images of multiple elements, as well as frequently used functional controls within the scene. This scene can be understood as a dynamically generated set of exclusive, high-frequency functions and visual content, which can be called a Reality Space. This space not only integrates the geographical environment but also incorporates user behavior habits to create contextualized scenes and functional entry points.

[0056] For example, when a user drives their vehicle to the vicinity of the company, the scene displayed through the human-machine interface can be an office environment, and it can also aggregate frequently used functions such as music, navigation, and meeting reminders.

[0057] For example, when a user drives their vehicle to their vicinity, the human-machine interface displays a warm and inviting home scene, complete with features such as pet interaction and family reminders.

[0058] For example, when users drive out for leisure, the scenes displayed through the human-machine interface include natural scenery, campsites, and rest areas, creating an immersive travel atmosphere.

[0059] The environmental information of the target vehicle's human-machine interface can include the environmental information of the scene in which the 3D visual image of the target element is located. Based on the vertex data of the target element, after constructing a mesh, the obtained texture can be applied to the mesh according to the preset material. Then, the lighting parameters in the environmental information of the human-machine interface are bound to the material associated with the target element. In this way, as the intensity or color of the scene lighting system changes, the visual appearance of the 3D visual image of the target element will also change, maintaining consistency with the scene lighting environment and achieving a natural and realistic visual integration effect.

[0060] It should be noted that textures can be obtained by the rendering engine parsing the 3D model of the target element, or by the rendering engine based on the input image, or by the default settings.

[0061] In some embodiments of this application, binding the lighting parameters in the environmental information of the target vehicle's human-machine interface to the material associated with the target element may include the following steps: Based on the lighting parameters and vertex data in the environmental information of the target vehicle's human-machine interface, calculate the ambient light component, diffuse reflection component, and specular highlight component. The ambient light component, diffuse component, and specular highlight component are superimposed on the texture color of the target element in the material associated with the target element.

[0062] For ease of description, the above steps will be combined for explanation.

[0063] In this embodiment of the application, vertex data of the target element can be obtained by parsing the three-dimensional model of the target element.

[0064] Based on the lighting parameters and vertex data in the environmental information of the target vehicle's human-machine interface, the ambient light component, diffuse component, and specular highlight component can be calculated. There are two lighting calculation modes: one is through a fixed-function pipeline, and the other is through a programmable pipeline.

[0065] Optionally, in the fixed-function pipeline, the vertex data is spatially transformed in the vertex shader to obtain interpolated vertex data. The interpolated vertex data is then input into the fragment shader, where the ambient light component, diffuse component, and specular highlight component are calculated based on the lighting parameters in the environmental information of the target vehicle's human-machine interface and the interpolated vertex data.

[0066] Optionally, in the programmable pipeline, the vertex shader obtains vertex attributes based on vertex data, performs custom rendering calculations on the vertex attributes through programming to complete spatial transformation, and passes the spatial transformation results to the fragment shader. In the fragment shader, based on the lighting parameters in the environmental information of the target vehicle's human-machine interface, the ambient light component, diffuse component, and specular highlight component are calculated through the lighting model.

[0067] The fixed-function pipeline can perform lighting calculations during the color transformation and lighting stages. The vertex attributes passed can include vertex position, normals, texture coordinates, etc.

[0068] After obtaining the ambient light component, diffuse component, and specular highlight component, these components can be overlaid with the texture color of the target element in the material associated with the target element to obtain the rendering effect of the target element. This allows the 3D visual image of the target element to change with the intensity or color changes of the lighting system.

[0069] Optionally, the lighting parameters, environment map, normals, base color, and reflection intensity of the human-computer interface can be passed into the material through the OpenGL standard interface. Based on the vertex data and lighting parameters of the target element, the ambient light component, diffuse component, and specular highlight component are calculated. Then, the ambient light component, diffuse component, and specular highlight component are superimposed with the texture color of the target element in the material associated with the target element. The final color output is completed in the fragment shader to obtain the rendering effect of the target element.

[0070] exist Figure 3 In the illustrated embodiment, taking OpenGL as an example, the OpenGL lighting system can be set up first to obtain the vertex data of the target element and the lighting parameters from the environment information of the human-computer interface. There are two lighting calculation modes: one is lighting calculation through the fixed-function pipeline, and the other is lighting calculation through the programmable pipeline. When performing lighting calculation through the fixed-function pipeline, it is done during the color transformation and lighting stage. The interpolated vertex data is input into the fragment shader. The fragment shader calculates the ambient light component, diffuse component, and specular highlight component based on the lighting parameters and vertex data, and then superimposes the texture color of the target element to output the final fragment color. When performing lighting calculations through a programmable pipeline, the vertex shader obtains vertex attributes such as position, normal, and texture coordinates. Custom rendering calculations are performed on these vertex attributes through programming to complete spatial transformations. The spatial transformation results are then passed to the fragment shader. Based on lighting parameters, the fragment shader calculates the ambient light component, diffuse component, and specular highlight component using a lighting model (such as Phong (one lighting model) or Blinn-Phong (another lighting model)). The texture color of the target element is then superimposed to output the final fragment color.

[0071] In some embodiments of this application, the method may further include the following steps: In response to interactive commands for the 3D visual image of the target element, adjust the spatial transformation information associated with the target element to control the rotation, scaling, or displacement of the 3D visual image of the target element.

[0072] In this embodiment, after displaying a 3D visual image of a target element in the human-machine interface, the user can issue interactive commands for the 3D visual image of the target element. These interactive commands can be triggered through various methods, including touch gestures (such as dragging, pinching, and rotating the 3D visual image of the target element in the human-machine interface), voice commands (such as saying "zoom in on this model" or "rotate 90° clockwise"), controls (such as left arrows, right arrows, and rotation icons), and visual or gesture recognition (such as recognizing specific user gestures via an in-vehicle camera, such as air grabbing or moving).

[0073] In response to an interactive command regarding a 3D visual image of a target element, the spatial transformation information associated with the target element can be adjusted to control the rotation, scaling, or translation of the 3D visual image of the target element. The spatial transformation information may include translation information, scaling information, rotation information, etc. Optionally, the translation of the 3D visual image of the target element can be controlled by adjusting the displacement information in the spatial transformation information associated with the target element. Optionally, the scaling of the 3D visual image of the target element can be controlled by adjusting the scaling information in the spatial transformation information associated with the target element. Optionally, new rotation information can be obtained by performing a calculation between the rotation information in the spatial transformation information associated with the target element and the rotation information generated according to the rotation command; the rotation of the 3D visual image of the target element can be controlled based on this new rotation information.

[0074] Upon receiving user interaction commands, by adjusting the spatial transformation information associated with the target element, the rotation, scaling, or displacement of the target element's 3D visual image can be controlled, providing users with an intuitive control experience and enhancing user engagement.

[0075] It should be noted that the process of controlling the rotation, scaling, or displacement of the 3D visual image of the target element can be understood as the rendering engine performing real-time rendering based on the environmental information of the human-computer interface and the adjusted spatial transformation information associated with the target element. The rendered 3D visual image has lighting and shadow effects that match the environmental information.

[0076] In some embodiments of this application, the method may further include the following steps: In response to an activity command for a 3D visual image of a target element, drive the 3D visual image of the target element to perform activities.

[0077] Optionally, driving the 3D visual image of the target element to move may include the following steps: Retrieve target skeletal animation data that matches the type of the target element; each skeletal animation data corresponds to multiple frames of skeletal transformation images. Pass the multi-frame skeletal transformation images corresponding to the target skeletal animation data into the material associated with the target element; Based on the data recorded in each frame of the skeleton transformation image corresponding to the target skeleton animation data, the 3D visual image of the target element is driven to move.

[0078] For ease of description, the above steps will be combined for explanation.

[0079] In this embodiment, if the 3D visual image of the target element is a virtual image, such as a virtual character, virtual pet, or virtual doll, it can be driven to move through activity commands, producing animation effects. Activity commands can be issued by the user via touch gestures, voice, or can be generated by default based on the current scene; for example, a greeting action command can be generated after the 3D model of the target element is rendered. After displaying the 3D visual image of the target element within a preset area, in response to the activity command for the 3D visual image of the target element, the 3D visual image of the target element can be driven to move, producing animation effects.

[0080] To animate a target element's 3D visual image, a skeleton needs to be generated. This can be achieved using skeleton baking technology, which bakes existing skeleton information into an image, converts the image into a texture, modifies the material associated with the target element, reads the color information from the texture, converts it into numbers, and alters the vertices of the target element's 3D visual image, causing them to shift and thus animate the target element's 3D visual image. In other words, skeleton-driven animation essentially controls the vertex displacement of the corresponding model.

[0081] Skeletal animation data can be pre-stored, with each skeletal animation data corresponding to multiple frames of skeletal transformation images. For example, a model with 30 frames and 50 bones can have its movement, rotation, scaling, and other information (i.e., matrices) recorded for all 50 bones across those 30 frames. This information is then stored in the skeletal transformation images, which can be in EXR (an image format) (EXR format images can store 32 bits or more and can record float type data). Information such as the image's region and frame rate can also be recorded. By repeating the process of storing information in the skeletal transformation images, various animations can be recorded to obtain the corresponding skeletal animation data.

[0082] Upon receiving an activity command for a 3D visual image of a target element, target skeletal animation data matching the type of the target element can be retrieved. For example, matching target skeletal animation data can be found in pre-stored skeletal animation data. Then, the multi-frame skeletal transformation images corresponding to the target skeletal animation data are passed to the material associated with the target element and played in the material according to the frame rate. For example, when playing the first frame, the data of the first frame is parsed according to the corresponding recorded area, and the data is assigned to the corresponding vertices in the vertex shader to reproduce the action. In other words, based on the data recorded in each frame of the skeletal transformation images corresponding to the target skeletal animation data, the 3D visual image of the target element is driven to move and the animation is executed.

[0083] This can bring the 3D visual image of the target element to life, making the target element more vivid.

[0084] like Figure 4 As shown, the target image is a two-dimensional image of a water droplet or a doll. Using an AI model, a three-dimensional model of the water droplet or doll is generated. Then, the three-dimensional model of the water droplet or doll is rendered, which can present a replaceable virtual personal assistant (VPA) with skeletal animation in the human-computer interface, realizing a customized image.

[0085] like Figure 5 The image shown is a schematic diagram of a camping scene, which includes elements such as tents, beach chairs, campfires, butterflies, signs, and dolls. The dolls are virtual characters that can be animated using skeletal animation technology.

[0086] like Figure 6 The image shown is a schematic diagram of a camping scene, which includes a plush toy. By responding to interactive commands to the plush toy, it can be rotated, changing its posture from the left image to the right image.

[0087] In some embodiments of this application, the target image is an image provided by the user or selected from an in-vehicle image library. The target image is input into an AI model to generate a 3D model of the target element. The 3D model of the target element is rendered by a rendering engine based on the environmental information of the target vehicle's human-machine interface, so as to display the 3D visual image of the target element within a preset area of ​​the human-machine interface, thereby achieving static enhancement of in-vehicle human-machine interaction.

[0088] In some embodiments of this application, the target image is an image of the surrounding environment of the target vehicle acquired in real time. The target image is input into an AI model to generate a 3D model of the target element. A rendering engine then renders the 3D model of the target element based on the environmental information of the target vehicle's human-machine interface, displaying the 3D visual image of the target element within a preset area of ​​the human-machine interface, thus achieving dynamic enhancement of in-vehicle human-machine interaction. This dynamic enhancement scheme can be applied to intelligent driving scenarios.

[0089] In some embodiments of this application, when the target image is an image of the environment surrounding the target vehicle acquired in real time, the target element is an element in the target image that is not fully represented by a preset 3D model. Rendering the 3D model of the target element using a rendering engine based on the environmental information of the target vehicle's human-machine interface may include the following steps: The rendering engine performs spatial alignment and fusion rendering of the 3D model of the target element and the basic scene constructed based on the target image and the preset 3D model, according to the environmental information of the target vehicle's human-machine interface.

[0090] In this embodiment of the application, the target image is an image of the surrounding environment of the target vehicle acquired in real time. Some elements in the target image may be fully represented by a preset three-dimensional model, while some elements may not be fully represented by the preset three-dimensional model. Elements that cannot be fully represented (or accurately expressed) by the preset three-dimensional model (such as overpasses, guardrails, construction barriers, temporary roadblocks, and sudden objects on the road surface (such as running animals, rolling balls, temporarily parked tricycles, scattered luggage, etc.)) can be considered as target elements. A three-dimensional model of the target elements is generated using an AI model.

[0091] After generating a 3D model of the target element using an AI model, the 3D model of the target element can be fused and rendered with a base scene constructed based on the pre-defined 3D model. Optionally, the 3D model of the target element and the base scene constructed based on the pre-defined 3D model can be spatially aligned during the fusion rendering, enabling the 3D visual image of the target element to be spatially aligned with the base scene. For example... Figure 7 , Figure 8 The image shown is a schematic diagram obtained after fusing and rendering the 3D model of the target element with a base scene constructed based on a preset 3D model. The object indicated by the AI ​​is the 3D visual image of the target element. Figure 7 In Chinese, AVM stands for Around View Monitor. Figure 8 MAP in the text refers to a map.

[0092] By leveraging the real-time generation capabilities of AI models, three-dimensional models can be generated for dynamic or complex elements in target images that are difficult to represent using traditional algorithms, thereby enhancing the vehicle's environmental perception capabilities and the information dimension of human-computer interaction.

[0093] In some embodiments of this application, the method may further include at least one of the following: Based on the 3D model of the target element, determine at least one of the following: type, size, or spatial location of the target element; Determine whether to output an alarm message based on at least one of the target element's type, size, or spatial location.

[0094] In one embodiment, it is determined whether to output an alarm message based on at least one of the target element's type, size, or spatial location: including when the target element is an obstacle, when the target element's size exceeds a preset value, or when the target element's spatial location conflicts with the vehicle's direction of travel.

[0095] Alarm messages can be delivered via voice broadcast, such as "Obstacle ahead of the vehicle" or "Target ahead is too large." Alarm messages can also be displayed as pop-up windows on the human-machine interface.

[0096] For ease of description, the above steps will be combined for explanation.

[0097] In this embodiment of the application, after generating a three-dimensional model of the target element using an AI model, at least one of the following information can be obtained from the three-dimensional model of the target element: type, size, or spatial location. Based on the obtained information, it can be predicted whether there are risk factors (such as predicting risk factors that are not visible in front of the target vehicle but can be inferred). If there are risk factors, an alarm message can be output.

[0098] For example, if the target element is goods scattered on the road, an alarm message can be output.

[0099] For example, by using an AI model to generate a 3D model of a stone obstacle from a target image, it is possible to infer the presence of a stone obstacle nearby and provide avoidance prompts.

[0100] For example, in blind spots or fog, if an AI model can identify a 3D model of a person's limbs based on a target image, it can determine that someone is nearby and that it is necessary to slow down or stop.

[0101] The 3D model of the target element generated by the AI ​​model can provide timely warnings and improve vehicle driving safety.

[0102] The foregoing has provided a detailed description of the static and dynamic enhancement solutions in the embodiments of this application. In one possible implementation, the AI-enhanced human-computer interface utilizes Kanzi's real-time rendering capabilities and skeletal animation baking technology to provide skeletal and animation support for the virtual character. For example... Figure 9As shown, in one possible architecture, the backend can communicate with the AI ​​model, providing Kanzi with a 3D model and offering interfaces to connect to various AI models, such as large AI 3D generative models. Kanzi performs operations such as parsing and 3D model rendering, where 3D model rendering can include Kanzi rendering materials and binding Kanzi's lighting system. Figure 10 As shown, in one possible workflow, the acquired target image is input into the AI ​​model, which generates a 3D model of multiple elements. Kanzi then determines whether a skeleton needs to be adapted based on the 3D model of each element. If so, textures are baked using pre-set skeletons. Subsequently, a comprehensive rendering of the material, texture, and lighting systems is performed on the 3D model of each element. This can be applied to enhance the human-computer interface of ARHUB, Mycar, Surrounding Reality (SR), maps, etc.

[0103] Kanzi enables high-performance rendering of 3D models generated in real time by AI models, allowing the evolution of 3D scenes from pre-defined content to real-time generation, which helps to expand the experience boundaries of intelligent cockpit systems.

[0104] It should be noted that the vehicle-mounted human-machine interaction method provided in this application embodiment can be executed by a vehicle-mounted human-machine interaction device, or by a control module in the vehicle-mounted human-machine interaction device for executing the vehicle-mounted human-machine interaction method. This application embodiment uses the execution of the vehicle-mounted human-machine interaction method by a vehicle-mounted human-machine interaction device as an example to illustrate the vehicle-mounted human-machine interaction method provided in this application embodiment.

[0105] See Figure 11 The diagram shown is a structural schematic of the in-vehicle human-machine interaction device provided in an embodiment of this application, which includes the following modules: The acquisition module 1110 is used to acquire the target image, which contains target elements; The generation module 1120 is used to input the target image into the artificial intelligence (AI) model to generate a three-dimensional model of the target element; The rendering module 1130 is used to render the three-dimensional model of the target element based on the environmental information of the target vehicle's human-machine interface through the rendering engine, so as to display the three-dimensional visual image of the target element in a preset area of ​​the human-machine interface.

[0106] Using the apparatus provided in this application embodiment, after acquiring the target image, the target image is input into the AI ​​model to generate a 3D model of the target element in the target image. The 3D model of the target element is rendered by the rendering engine according to the environmental information of the human-machine interface of the target vehicle, so as to display the 3D visual image of the target element in a preset area of ​​the human-machine interface. Different target images will result in different 3D visual images of the target element. By using AI technology and rendering engine, the human-machine interface is no longer just a preset interface, but can present a dynamically generated interface, realizing the transformation from static preset to dynamic generation. This allows the content generation and cognitive ability of the AI ​​model to be stably and efficiently integrated into the in-vehicle human-machine interaction, thereby improving the driving experience.

[0107] In some embodiments of this application, the rendering module 1130 is specifically used for: The 3D model of the target element is analyzed by the rendering engine to obtain the vertex data of the target element; A mesh is constructed based on vertex data; The mesh is rendered based on the environmental information of the target vehicle's human-machine interface.

[0108] In some embodiments of this application, the rendering module 1130 is specifically used for: Apply the obtained texture to the mesh according to the preset material; Bind the lighting parameters from the environmental information of the target vehicle's human-machine interface to the material associated with the target element.

[0109] In some embodiments of this application, the rendering module 1130 is specifically used for: Based on the lighting parameters and vertex data in the environmental information of the target vehicle's human-machine interface, calculate the ambient light component, diffuse reflection component, and specular highlight component. The ambient light component, diffuse component, and specular highlight component are superimposed on the texture color of the target element in the material associated with the target element.

[0110] In some embodiments of this application, the rendering module 1130 is specifically used for: In the fixed-function pipeline, the vertex shader performs spatial transformation on the vertex data to obtain interpolated vertex data. The interpolated vertex data is then input into the fragment shader. In the fragment shader, based on the lighting parameters in the environmental information of the target vehicle's human-machine interface and the interpolated vertex data, the ambient light component, diffuse component, and specular highlight component are calculated. Alternatively, in a programmable pipeline, the vertex shader obtains vertex attributes based on vertex data, performs custom rendering calculations on vertex attributes through programming to complete spatial transformation, and passes the spatial transformation results to the fragment shader. In the fragment shader, based on the lighting parameters in the environmental information of the target vehicle's human-machine interface, the ambient light component, diffuse component, and specular highlight component are calculated through a lighting model.

[0111] In some embodiments of this application, the in-vehicle human-machine interaction device 1100 further includes an image transformation module, used for: In response to interactive commands for the 3D visual image of the target element, adjust the spatial transformation information associated with the target element to control the rotation, scaling, or displacement of the 3D visual image of the target element.

[0112] In some embodiments of this application, the in-vehicle human-machine interaction device 1100 further includes an image activity module, used for: In response to an activity command for a 3D visual image of a target element, drive the 3D visual image of the target element to perform activities.

[0113] In some embodiments of this application, the image activity module is specifically used for: Retrieve target skeletal animation data that matches the type of the target element; each skeletal animation data corresponds to multiple frames of skeletal transformation images. Pass the multi-frame skeletal transformation images corresponding to the target skeletal animation data into the material associated with the target element; Based on the data recorded in each frame of the skeleton transformation image corresponding to the target skeleton animation data, the 3D visual image of the target element is driven to move.

[0114] In some embodiments of this application, the target image is an image provided by the user or selected from an in-vehicle image library, or the target image is an image of the surrounding environment of the target vehicle acquired in real time.

[0115] In some embodiments of this application, when the target image is provided by the user or selected from an in-vehicle image library, the scene in which the three-dimensional visual image of the target element is located is determined based on at least one of the following: real-time geographical location data of the target vehicle, user operation habit data, user instructions, and images provided or selected by the user.

[0116] In some embodiments of this application, when the target image is an image of the environment surrounding the target vehicle acquired in real time, the target element is an element in the target image that is not fully represented by a preset 3D model. The rendering module 1130 is specifically used for: The rendering engine performs spatial alignment and fusion rendering of the 3D model of the target element and the basic scene constructed based on the target image and the preset 3D model, according to the environmental information of the target vehicle's human-machine interface.

[0117] In some embodiments of this application, the in-vehicle human-machine interaction device further includes an alarm module for performing at least one of the following: Based on the 3D model of the target element, determine at least one of the following: type, size, or spatial location of the target element; Determine whether to output an alarm message based on at least one of the target element's type, size, or spatial location.

[0118] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the relevant method embodiments, and will not be repeated here to avoid repetition.

[0119] Corresponding to the above method embodiments, this application also provides an apparatus, including: Memory, used to store computer programs; A processor is used to implement the steps of the above method embodiments when executing a computer program.

[0120] like Figure 12 The diagram shows the structural composition of the device, which may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.

[0121] In this embodiment, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0122] The processor 10 can call programs stored in the memory 11. Specifically, the processor 10 can execute the operations in the method embodiment.

[0123] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions: Obtain the target image, which contains the target element; The target image is input into an artificial intelligence (AI) model to generate a 3D model of the target element; The rendering engine renders the 3D model of the target element based on the environmental information of the target vehicle's human-machine interface, so as to display the 3D visual image of the target element in a preset area of ​​the human-machine interface.

[0124] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during use.

[0125] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0126] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.

[0127] Of course, it should be noted that, Figure 12 The structure shown does not constitute a limitation on the device in the embodiments of this application. In practical applications, the device may include more than Figure 12 More or fewer components as shown, or combinations of certain components.

[0128] Alternatively, the device may include an in-vehicle human-machine interface device 1100.

[0129] In one embodiment, the aforementioned device may be a vehicle.

[0130] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0131] The processor is the processor in the above-described device embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] Furthermore, it should be noted that this application also provides a program product or program, which may include instructions that can be stored in a readable storage medium. The processor of the device reads the instructions from the readable storage medium and executes the instructions, causing the device to perform the descriptions in the corresponding method embodiments above; therefore, these will not be repeated here. Additionally, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the program product or program embodiments involved in this application, please refer to the descriptions in the method embodiments of this application.

[0133] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0134] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0135] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0137] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many different implementations and modifications to this application without departing from the spirit and scope of the claims. All such implementations, improvements and modifications are within the scope of protection of this application.

Claims

1. A vehicle-mounted human-computer interaction method, characterized in that, include: Acquire a target image, wherein the target image contains target elements; The target image is input into an artificial intelligence (AI) model to generate a three-dimensional model of the target element; The rendering engine renders the three-dimensional model of the target element based on the environmental information of the target vehicle's human-machine interface, so as to display the three-dimensional visual image of the target element within a preset area of ​​the human-machine interface.

2. The method according to claim 1, characterized in that, The step of rendering the 3D model of the target element using a rendering engine based on the environmental information of the target vehicle's human-machine interface includes: The 3D model of the target element is analyzed by the rendering engine to obtain the vertex data of the target element; Based on the vertex data, a mesh is constructed; The mesh is rendered based on the environmental information of the target vehicle's human-machine interface.

3. The method according to claim 2, characterized in that, The step of rendering the mesh based on the environmental information of the target vehicle's human-machine interface includes: According to the preset material, the obtained texture is applied to the mesh; Bind the lighting parameters in the environmental information of the target vehicle's human-machine interface to the material associated with the target element.

4. The method according to claim 3, characterized in that, The step of binding the lighting parameters in the environmental information of the target vehicle's human-machine interface to the material associated with the target element includes: Based on the illumination parameters in the environmental information of the target vehicle's human-machine interface and the vertex data, calculate the ambient light component, diffuse reflection component and specular highlight component. The ambient light component, diffuse component, and specular highlight component are superimposed on the texture color of the target element in the material associated with the target element.

5. The method according to claim 4, characterized in that, The calculation of ambient light component, diffuse component, and specular highlight component based on the illumination parameters in the environmental information of the target vehicle's human-machine interface and the vertex data includes: In the fixed-function pipeline, the vertex shader performs spatial transformation on the vertex data to obtain interpolated vertex data, and inputs the interpolated vertex data into the fragment shader. In the fragment shader, based on the lighting parameters in the environmental information of the target vehicle's human-machine interface and the interpolated vertex data, the ambient light component, diffuse component, and specular highlight component are calculated. Alternatively, the calculation of ambient light components, diffuse components, and specular highlight components based on the illumination parameters in the environmental information of the target vehicle's human-machine interface and the vertex data includes: In the programmable pipeline, the vertex shader obtains vertex attributes based on the vertex data, performs custom rendering calculations on the vertex attributes through programming to complete the spatial transformation, and passes the spatial transformation results to the fragment shader. In the fragment shader, based on the lighting parameters in the environmental information of the target vehicle's human-machine interface, the ambient light component, diffuse component, and specular highlight component are calculated through the lighting model.

6. The method according to claim 1, characterized in that, The method further includes: In response to an interactive command for a 3D visual image of the target element, the spatial transformation information associated with the target element is adjusted to control the rotation, scaling, or displacement of the 3D visual image of the target element.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: In response to an activity command for the three-dimensional visual image of the target element, the three-dimensional visual image of the target element is driven to move.

8. The method according to claim 7, characterized in that, The process of driving the 3D visual image of the target element to move includes: Obtain target skeletal animation data that matches the type of the target element, with each skeletal animation data corresponding to multiple frames of skeletal transformation images; The multi-frame skeletal transformation images corresponding to the target skeletal animation data are passed into the material associated with the target element; Based on the data recorded in each frame of the skeleton transformation image corresponding to the target skeleton animation data, the three-dimensional visual image of the target element is driven to move.

9. The method according to any one of claims 1 to 6 or 8, characterized in that, The target image is an image provided by the user or selected from the in-vehicle image library, or the target image is an image of the surrounding environment of the target vehicle acquired in real time.

10. The method according to claim 9, characterized in that, When the target image is provided by the user or selected from the in-vehicle image library, the scene in which the three-dimensional visual image of the target element is located is determined based on at least one of the following: the real-time geographical location data of the target vehicle, user operation habit data, user instructions, and the image provided or selected by the user.

11. The method according to claim 9, characterized in that, When the target image is a real-time acquired image of the environment surrounding the target vehicle, the target element is an element in the target image that is not fully represented by a preset 3D model. The step of rendering the 3D model of the target element using a rendering engine based on the environmental information of the target vehicle's human-machine interface includes: The rendering engine performs spatial alignment and fusion rendering of the 3D model of the target element and the basic scene constructed based on the target image and the preset 3D model, according to the environmental information of the target vehicle's human-machine interface.

12. The method according to claim 11, characterized in that, The method further includes: Based on the three-dimensional model of the target element, determine at least one of the following: type, size, or spatial location of the target element; Determine whether to output an alarm message based on at least one of the target element's type, size, or spatial location.

13. A vehicle-mounted human-machine interaction device, characterized in that, include: The acquisition module is used to acquire a target image, wherein the target image contains target elements; The generation module is used to input the target image into the artificial intelligence (AI) model to generate a three-dimensional model of the target element; The rendering module is used to render the three-dimensional model of the target element based on the environmental information of the target vehicle's human-machine interface through the rendering engine, so as to display the three-dimensional visual image of the target element in a preset area of ​​the human-machine interface.

14. A device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the vehicle-mounted human-machine interaction method as described in any one of claims 1 to 12 when executing the computer program; or, The device includes the in-vehicle human-machine interaction device as described in claim 13.

15. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the vehicle-mounted human-machine interaction method as described in any one of claims 1 to 12.

Images