Vehicle-mounted interaction method and vehicle-mounted interaction system

Abstract

The application provides a vehicle-mounted interaction method and a vehicle-mounted interaction system, wherein the method comprises the following steps: obtaining vehicle environment data, user identity information, user interaction information and vehicle state parameters of a smart vehicle; constructing a spatial coordinate model containing the vehicle and the surrounding environment according to the vehicle environment data; then fusing the model with the other three types of information to generate interaction content; and finally projecting the interaction content to the target projection area inside and outside the vehicle based on the spatial coordinate model and the vehicle state parameters when the vehicle is parked. By constructing a unified spatial coordinate model, the virtual and real spaces are aligned, and by fusing multi-dimensional information to generate interaction content, the content can be highly adapted to the scene and user demand, realizing integrated space interaction inside and outside the vehicle and improving the accuracy and immersion of vehicle-mounted interaction.

Description

Technical Field

[0001] This application relates to the field of vehicle control technology, and more specifically, to an in-vehicle interaction method and an in-vehicle interaction system. Background Technology

[0002] As vehicle technology continues to iterate, in-vehicle display and interaction systems are gradually moving away from the traditional single-screen mode and developing towards multi-screen, immersive, and scenario-based approaches. Users have increasingly higher demands for an integrated spatial interaction experience both inside and outside the vehicle.

[0003] In existing technologies, in-vehicle interaction mostly uses instrument panel screens, central control screens, head-up displays, or augmented reality display devices to display information and provide basic entertainment functions. There are also ways to introduce projection display and motion-sensing interaction technologies into the in-vehicle environment to enhance the immersiveness and fun of the interaction.

[0004] However, existing in-vehicle interactive systems still revolve around various display terminals, with the vehicle serving only as an information carrier platform. They lack a unified spatial coordinate system for the vehicle, the user, and the real environment, making it difficult to achieve precise alignment and collaborative presentation of content displayed inside and outside the vehicle. At the same time, existing solutions do not incorporate the vehicle's operating status and physical execution components into the interaction process, thus failing to achieve coordinated linkage between content generation, display output, and vehicle physical feedback, which affects the user's interactive experience. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing an in-vehicle interaction method and an in-vehicle interaction system, thereby solving the problem that in-vehicle interaction systems in the prior art are unable to align the displayed content inside and outside the vehicle, cannot interact in conjunction with vehicle operating parameters, and thus affect the user's interactive experience.

[0006] To achieve the above objectives, the technical solution adopted in this application is as follows: In a first aspect, this application provides an in-vehicle interaction method, the method comprising: Acquire vehicle environmental data, user identity information, user interaction information, and vehicle status parameters; A spatial coordinate model of the vehicle is constructed based on the vehicle environment data. The spatial coordinate model is a virtual model that includes the vehicle and its surrounding environment. Interactive content is generated based on the spatial coordinate model, the user identity information, the user interaction information, and the vehicle status parameters; When the vehicle is in a parked state, the interactive content is projected onto the target projection area of ​​the vehicle based on the spatial coordinate model and the vehicle state parameters.

[0007] Optionally, constructing the spatial coordinate model of the vehicle based on the vehicle environment data includes: The vehicle environmental data is subjected to time synchronization processing and noise filtering processing to obtain processed environmental data; The processed environmental data is segmented according to its region to obtain initial vehicle body region data, initial terrain region data, and initial external object data. A vehicle coordinate system is established based on the vehicle, and the target vehicle area data, target terrain area data, and target external object data corresponding to the initial vehicle area data, the initial terrain area data, and the initial external object data are determined respectively in the vehicle coordinate system. The spatial coordinate model is established based on the coordinate and attribute information of the target vehicle area data, the target terrain area data, and the target external object data.

[0008] Optionally, the step of segmenting the processed environmental data according to its region to obtain initial vehicle area data, initial terrain area data, and initial external object data includes: The processed environmental data is matched with a preset vehicle body structure model to obtain the initial vehicle body area data. The initial terrain region data is obtained by identifying the geometric features of the terrain surface from the processed environmental data; The data in the processed environmental data, excluding the initial vehicle area data and the initial terrain area data, are used as the initial external object data.

[0009] Optionally, generating interactive content based on the spatial coordinate model, the user identity information, the user interaction information, and the vehicle state parameters includes: The user identity information is parsed to obtain a first parsing result, which includes: user permissions and user interaction mode; The spatial coordinate model is analyzed to obtain a second analysis result, which includes: the current environment type and the interactive space range; Information is extracted from the user interaction information to obtain a third parsing result, which includes: interaction instructions; The vehicle state parameters are analyzed to obtain a fourth analysis result, which includes the vehicle's energy level, interaction range, and feedback intensity. The interactive content is generated based on the first parsing result, the second parsing result, the third parsing result, and the fourth parsing result.

[0010] Optionally, generating the interactive content based on the first parsing result, the second parsing result, the third parsing result, and the fourth parsing result includes: The first parsing result, the second parsing result, the third parsing result, and the fourth parsing result are fused to obtain a fusion result. Based on the fusion result, a target content generation strategy is obtained by matching in a preset strategy library. The interactive content is generated based on the target content generation strategy, the first parsing result, the second parsing result, the third parsing result, and the fourth parsing result.

[0011] Optionally, the vehicle includes: a projection unit; The step of projecting the interactive content onto the target projection area of ​​the vehicle based on the spatial coordinate model and the vehicle state parameters includes: The target projection area is determined based on the interactive content, the spatial coordinate model, and the projection parameters of the projection unit. The initial projection image of the interactive content in the target projection area is determined based on the spatial coordinate model. Based on the vehicle state parameters and the spatial coordinate model, the initial projection image is corrected to obtain the target projection image; The brightness of the target projection image is compensated according to the projection incident angle, and the target projection image is projected onto the target projection area of ​​the vehicle.

[0012] Optionally, the step of correcting the initial projected image based on the vehicle state parameters and the spatial coordinate model to obtain the target projected image includes: The ground features of the target projection area are extracted from the spatial coordinate model; Vehicle body features are extracted from the spatial coordinate model and the vehicle state parameters; Based on the ground features and the vehicle body features, each pixel in the initial projection image is corrected to obtain the intermediate projection image; Based on the spatial coordinate model, the occlusion relationship of the intermediate projection image is determined, and an occlusion mask is generated. The target projection image is obtained based on the occlusion mask and the intermediate projection image.

[0013] Optionally, the step of determining the occlusion relationship of the intermediate projected image based on the spatial coordinate model and generating an occlusion mask includes: Traverse each pixel of the intermediate projected image, and for the current pixel that has been traversed, determine the ray in space for the current pixel; The first depth value of the object is obtained by tracing the ray of the current pixel in space to find the object that hits the ray in space. If the first depth value is less than the depth value of the current pixel in the intermediate projected image, then the current pixel is determined to be an occluded pixel. After determining all the occluded pixels, the occlusion mask is generated based on each of the occluded pixels.

[0014] Optionally, the method further includes: Real-time collection of vehicle environmental data, user identity information, user interaction information, and vehicle status parameters; The spatial coordinate model is updated based on the vehicle environment data, user identity information, user interaction information, and vehicle status parameters to obtain the updated model. The target projection area of ​​the vehicle and / or the interactive content are corrected based on the updated model.

[0015] In a second aspect, this application provides an in-vehicle interaction system, the system including a vehicle and at least one projection unit disposed on the vehicle, the system being used to perform the steps of the in-vehicle interaction method as described in the first aspect.

[0016] The beneficial effects of this application are as follows: By acquiring multi-dimensional data, it is possible to ensure that the interaction design fits the actual usage scenario; by constructing a spatial coordinate model that includes the vehicle and its surrounding environment, a unified reference between virtual interactive content and real space is established, solving the problem of misalignment between interactive content and the real scene, and thus enabling spatial alignment of the projection; by integrating multiple types of data to generate customized interactive content, the interactive content is adapted to user attributes, interaction needs, and vehicle status, improving the personalization and adaptability of the content; by projecting based on the spatial coordinate model and vehicle status parameters in the parked state, both the safety of use is ensured, and the interactive content is grounded in the target area inside and outside the vehicle, realizing integrated spatial interaction inside and outside the vehicle and improving the interactive experience in the parking scenario.

[0017] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram illustrating an application scenario provided by an embodiment of this application is shown; Figure 2 A flowchart of an in-vehicle interaction method provided in an embodiment of this application is shown; Figure 3 A flowchart of a method for establishing a spatial coordinate model provided in an embodiment of this application is shown; Figure 4 This document illustrates a flowchart of a region data segmentation method provided in an embodiment of this application. Figure 5 This document illustrates a flowchart of an embodiment of generating interactive content. Figure 6 This document illustrates a flowchart of yet another method for generating interactive content, as provided in an embodiment of this application. Figure 7 A flowchart illustrating a projection method provided in an embodiment of this application is shown; Figure 8 A flowchart illustrating a method for determining a target projection image according to an embodiment of this application is shown; Figure 9 This document illustrates a flowchart of a method for generating an occlusion mask according to an embodiment of this application. Figure 10 This document illustrates a flowchart of an embodiment of interactive content correction provided in this application. Figure 11 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0021] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0022] Existing intelligent vehicle interaction systems have evolved from traditional single-screen displays to multi-screen and immersive ones, realizing information display and entertainment functions through in-vehicle screens, HUDs, and AR devices. There are also solutions that incorporate projection and somatosensory interaction technologies to enrich vehicle interaction forms and break the limitations of single display terminals.

[0023] However, existing vehicle interaction systems still have obvious technical shortcomings. They lack a unified spatial coordinate system for the vehicle, user, and real environment, making it difficult to accurately align and coordinate the display content inside and outside the vehicle. The vehicle only serves as an information carrier platform, without incorporating its operating state and physical execution components into the interaction, and cannot achieve the coordinated linkage of content generation, display output, and vehicle physical feedback, resulting in a fragmented interaction experience.

[0024] Taking the scenario where a vehicle is parked on outdoor grassland / camping ground as an example, in the existing technology, although the vehicle interaction system can achieve simple interactions, it is difficult to combine the outdoor environment with the interaction content. The projected content cannot be aligned with the real scenes such as the camping ground terrain, vehicle body, and tent, and is prone to misalignment and deformation. Secondly, the existing interaction relies on a single projection display and does not adjust the interaction content according to the vehicle state. Therefore, it is difficult to adapt the interaction content according to the vehicle's power and the state of the vehicle's doors and seats. Moreover, the vehicle only serves as a carrier for projection, without incorporating the vehicle for interactive feedback linkage, so there is a lack of immersion.

[0025] Based on this, the present application proposes a vehicle interaction method. By obtaining vehicle environment data to construct a spatial coordinate model including the vehicle and its surrounding environment, it solves the problem of misalignment between projection and real scenes caused by the lack of a unified coordinate, and achieves accurate alignment of the projection space. By combining the spatial coordinate model, user identity and interaction information, and vehicle state parameters to generate interaction content, it solves the problem that the content cannot adapt to the user and vehicle states, making the interaction content more compatible with the camping scene, user needs, and vehicle state. And when parked, based on the spatial coordinate model and vehicle state parameters, project the interaction content to the target area, solve the problem that the vehicle is only a carrier without adaptable projection, and achieve the linkage between the in-vehicle and out-vehicle projections and the vehicle state, enhancing the immersion and adaptability of vehicle interaction.

[0026] Figure 1 is a schematic diagram of an application scenario of the present application. Referring to Figure 1 , this method can be applied to vehicles with at least one projection unit. The projection unit can be an LCoS projection unit, and the projection unit can be set inside or outside the vehicle, such as on Figure 1 the front windshield, rear of the vehicle, side doors of the vehicle, etc. shown in the figure. The specific position is not limited here.

[0027] Next, in combination with Figure 2 explain the vehicle interaction method of the present application. The execution subject of this method can be a vehicle equipped with a projection unit, such as Figure 2 As shown, the method includes: S201. Obtain vehicle environmental data, user identity information, user interaction information, and vehicle status parameters.

[0028] Among these, vehicle environment data refers to spatial data related to the real-world environment surrounding the vehicle, including terrain, external objects, and environmental contours. User identity information is relevant information used to identify user attributes, including user age, number of users, and permissions. User interaction information consists of interaction-related command signals issued by the user to the system, including gestures, gaze, and operational commands. Vehicle status parameters are the vehicle's own operational and structural status data, including parameters such as battery level, seats, doors / tailgate, and vehicle posture.

[0029] Optionally, vehicle-mounted LiDAR, cameras, and depth sensors can be used to collect environmental data such as terrain and external objects around the vehicle. Biometric identification distinguishes between adults and children, image recognition counts the number of users, and user permissions are determined based on the currently logged-in or registered user information.

[0030] Optionally, gesture commands can be captured through motion recognition, eye-tracking can be used to obtain user gaze interaction needs, and voice commands can be obtained by recognizing the user's voice and speech. Vehicle status parameters such as remaining battery power can be obtained through onboard detection units, and seat adjustment status and door / tailgate opening status can be collected through vehicle body sensors.

[0031] Taking a car-parking camping scenario as an example, when a car is parked at a suburban grassland campsite, the in-vehicle interactive system can collect environmental data such as grassland terrain and the location of surrounding trees through LiDAR, identify the parent-child user group in the car through facial recognition, capture the user's "start interactive game" gesture through motion sensors, and collect status parameters such as the vehicle's battery level being 80% and the tailgate being open.

[0032] S202. Construct a spatial coordinate model of the vehicle based on the vehicle environment data. The spatial coordinate model is a virtual model that includes the vehicle and its surrounding environment.

[0033] A spatial coordinate model is a digital virtual model built based on the vehicle, integrating spatial information about the vehicle itself and its surrounding environment. Based on collected vehicle environmental data, through data processing and spatial modeling algorithms, an integrated virtual model including the vehicle and its surrounding environment is constructed, allowing virtual interactive content to have a unified coordinate reference with real space.

[0034] Optionally, the environmental data can be filtered and synchronized, and then the three-dimensional spatial structure of the vehicle and campsite can be reconstructed using a point cloud fusion algorithm to generate a spatial coordinate model. Alternatively, the environmental data can be reconstructed from images, and combined with the vehicle's own three-dimensional parameters to construct a virtual coordinate model that includes terrain, external objects, and the vehicle body as a spatial coordinate model.

[0035] Taking the scenario of vehicle parking and camping as an example, a three-dimensional spatial coordinate virtual model can be constructed based on the collected grassland campsite terrain data, tree location data, tent location data, and vehicle structure data. This model is centered on the vehicle and includes the vehicle itself, the surrounding grassland terrain, and the locations of the tents and trees.

[0036] S203. Generate interactive content based on the spatial coordinate model, user identity information, user interaction information, and vehicle status parameters.

[0037] Interactive content refers to virtual interactive content that adapts to real space, user needs, and vehicle status, including digital content projected in various interactive scenarios such as virtual games, light and shadow interaction, and scenario-based experiences.

[0038] Optionally, the system can parse user identity to determine the interaction mode, parse spatial coordinate model to determine the interactive range, parse user interaction information to determine interaction requirements, and parse vehicle state parameters to determine content output constraints. Finally, based on the parsing results, a content generation algorithm is invoked to generate interactive content. For example, the parsing results are input into a preset content generation algorithm, which matches the spatial location of the content based on the spatial coordinate model, matches the content type based on user information, and matches the content output intensity based on vehicle state, automatically generating interactive content.

[0039] Taking the parking and camping scenario as an example, the system determines the grass in front of the vehicle as an interactive area based on the spatial coordinate model of the campsite. Combining the identities of the parent and child users, the user's "start parent-child game" interaction information, and the vehicle's 80% battery level and tailgate open status parameters, the system generates parent-child light and shadow interactive game content adapted to the grass space, and the spatial position of the content corresponds to the position of the real grass.

[0040] S204. When the vehicle is in a parked state, the interactive content is projected onto the target projection area of ​​the vehicle based on the spatial coordinate model and vehicle state parameters.

[0041] The parking status refers to the state where the vehicle is stopped and the engine / power system is in parking mode. The target projection area is the projection area inside and outside the vehicle that is adapted to the interactive content, including the ground in front of the vehicle, the display medium inside the vehicle, and the area around the doors.

[0042] Optionally, the vehicle can first be determined to be in a parked state, and then the target projection area of ​​the campsite (such as the grass in front of the vehicle) can be determined based on the spatial coordinate model. The projection light path can be adjusted in combination with the vehicle's posture, battery level and other status parameters, and interactive content can be projected through the external projection unit.

[0043] In another possible implementation, when the vehicle is parked, the interactive content can be matched with the projection coordinates of the real space based on the spatial coordinate model. Combined with the status parameters of the vehicle seats and doors, the target projection area inside and outside the vehicle can be determined, and the interactive content can be projected synchronously through multiple projection units.

[0044] In this embodiment, by acquiring multi-dimensional data, it is possible to ensure that the interaction design fits the actual usage scenario; by constructing a spatial coordinate model that includes the vehicle and its surrounding environment, a unified reference between the virtual interactive content and the real space is established, solving the problem of misalignment between the interactive content and the real scene, thereby enabling spatial alignment of the projection; by integrating multiple types of data to generate customized interactive content, the interactive content is adapted to user attributes, interaction needs, and vehicle status, improving the personalization and adaptability of the content; by projecting based on the spatial coordinate model and vehicle status parameters in the parked state, both the safety of use is ensured, and the interactive content is grounded in the target area inside and outside the vehicle, realizing integrated spatial interaction inside and outside the vehicle, and improving the interactive experience in the parking scenario.

[0045] The following is a further explanation of the spatial coordinate model of the vehicle constructed based on the vehicle's environmental data, such as... Figure 3 As shown, the above step S202 includes: S301. Perform time synchronization processing and noise filtering on the vehicle environmental data to obtain processed environmental data.

[0046] Optionally, hardware clock calibration and timestamp matching algorithms can be used to unify the time base of data collected by LiDAR, cameras, and depth sensors, achieving time synchronization of vehicle environmental data. Furthermore, Gaussian filtering and median filtering algorithms can be employed to remove noise from the environmental data, or threshold filtering can be used to remove abnormal environmental data, thus achieving noise filtering of the vehicle environmental data.

[0047] For example, assuming a vehicle is parked at a grassy campsite, the terrain point cloud collected by the lidar and the surrounding images captured by the camera can be time-stamped and synchronized. Then, Gaussian filtering can be used to remove environmental noise in the point cloud, resulting in campsite environmental data without time deviation or invalid interference.

[0048] S302. The processed environmental data is segmented according to the region to obtain the initial vehicle body area data, the initial terrain area data, and the initial external object data.

[0049] Among them, the initial vehicle body area is the original spatial data of the vehicle body, the initial terrain area data is the original spatial data of the terrain around the vehicle, and the initial external object data is the original spatial data of the external objects in the vehicle environment.

[0050] In one possible implementation, a semantic segmentation algorithm can be used to divide the processed environmental data into pixel-level regions and extract the original spatial data corresponding to the vehicle body, terrain, and external objects.

[0051] S303. Establish a vehicle coordinate system based on the vehicle, and determine the target vehicle area data, target terrain area data, and target external object data corresponding to the initial vehicle area data, initial terrain area data, and initial external object data in the vehicle coordinate system.

[0052] Among them, the vehicle coordinate system is a three-dimensional rectangular coordinate system established with the vehicle's preset position as the origin.

[0053] Optionally, the vehicle's center console can be used as the origin of the coordinate system. Through a coordinate transformation matrix, the initial vehicle body area data, initial terrain area data, and initial external object data can be converted into coordinates in the vehicle body coordinate system to obtain the target vehicle body area data, target terrain area data, and target external object data.

[0054] For example, a vehicle coordinate system is established with the center console of the camping vehicle as the origin. The initial spatial data of the vehicle body, grassland terrain, tents and trees are mapped to this coordinate system through a coordinate transformation matrix to obtain the target vehicle area data, target terrain area data and target external object data.

[0055] S304. Establish a spatial coordinate model based on the coordinate and attribute information of the target vehicle area data, target terrain area data, and target external object data.

[0056] Optionally, the coordinate information is the three-dimensional spatial coordinate position information of the target area data in the vehicle coordinate system. The attribute information is the object feature information corresponding to the target area data, such as terrain slope, external object outline, vehicle structure dimensions, etc.

[0057] One possible implementation involves constructing a spatial framework based on the coordinate information of each target data point, reconstructing the object's 3D features using attribute information, and generating a spatial coordinate model through point cloud reconstruction. Alternatively, the coordinates and attribute information of each target data point can be imported into a 3D modeling engine to automatically generate a digital spatial coordinate model containing the vehicle body, terrain, and external objects.

[0058] In this embodiment, the accuracy and effectiveness of the modeling data are ensured by synchronizing the vehicle environment data with time and filtering out noise. The classification and extraction of vehicle and surrounding environment data are achieved through region segmentation, which makes the modeling data structure clearer. A unified vehicle coordinate system is established based on the vehicle and coordinate mapping is completed, realizing the benchmark unification of all spatial data. Finally, the coordinate and attribute information are integrated to build an accurate and complete spatial coordinate model, which provides a reliable unified spatial reference for subsequent interactive content generation and projection.

[0059] The following is a further explanation of how the processed environmental data was segmented according to its region to obtain initial vehicle body region data, initial terrain region data, and initial external object data. Figure 4 As shown, step S302 above includes: S401. Match the processed environmental data according to the preset vehicle body structure model to obtain the initial vehicle body area data.

[0060] The preset vehicle body structure model is a pre-stored digital model of the vehicle's own three-dimensional structure, which includes feature information such as vehicle body size, outline, and spatial position of each component.

[0061] Optionally, a feature point matching algorithm can be used to compare the key feature points of the preset vehicle body structure model with the feature points of the processed environmental data, and the successfully matched areas can be used as the vehicle body region data. Alternatively, a contour matching algorithm can be used to match the contour features of the preset vehicle body structure model with the contour information of the processed environmental data to extract the spatial data corresponding to the vehicle body region.

[0062] S402. Identify the initial terrain area data from the processed environmental data based on the geometric features of the terrain surface.

[0063] Optionally, the geometric features of the terrain surface are the spatial geometric attributes of the terrain surrounding the vehicle, including surface smoothness, normal features, slope, geometric continuity, etc. Terrain region data are the spatial feature data of the terrain surrounding the vehicle in the processed environmental data.

[0064] Optionally, algorithms can be used to extract regions with strong surface geometric continuity and uniform normal features from the processed environmental data, identify them as terrain regions, and extract the corresponding data. Alternatively, based on a slope threshold, planar regions in the processed environmental data with slopes within a preset range can be identified as terrain, and terrain region data can be extracted.

[0065] S403. Use the data in the processed environmental data other than the initial vehicle area data and the initial terrain area data as the initial external object data.

[0066] Optionally, external object data refers to the spatial feature data of various objects around the vehicle, excluding the vehicle body and terrain, within the processed environmental data. External objects are physical objects around the vehicle other than terrain, such as trees, tents, and roadblocks.

[0067] Optionally, in the processed environmental dataset, vehicle body and terrain area data can be removed by data labels, and the remaining unlabeled data can be directly used as external object data.

[0068] In this embodiment, the accuracy of vehicle body data can be ensured by extracting vehicle body area data through matching a preset vehicle body model; and the accuracy of terrain area data can be achieved by identifying and extracting terrain area data based on terrain geometric features, thereby improving the efficiency and accuracy of area segmentation.

[0069] The following is a further explanation of the generation of interactive content based on the spatial coordinate model, user identity information, user interaction information, and vehicle status parameters, such as... Figure 5 As shown, step S203 above includes: S501. Parse the user's identity information to obtain the first parsing result, which includes: user permissions and user interaction mode.

[0070] Optionally, user permissions refer to the scope and level of system interactive functions that can be used, defined based on user identity. User interaction modes are interaction forms matched according to user identity and adapted to user needs, including parent-child mode, multi-person mode, single-person mode, etc.

[0071] As one possible approach, biometric identification can be used to analyze a user's age or identity, granting adults full access and matching them to single-person or multi-person modes, while limiting permissions for children and matching them to parent-child modes.

[0072] As another possible implementation, user registration information and on-site recognition data can be combined to determine the vehicle owner's or visitor's permissions, and then a single-person interaction mode or a multi-person interaction mode can be matched according to the number of people identified.

[0073] S502. Analyze the spatial coordinate model to obtain the second analysis result, which includes the current environment type and the interactive space range.

[0074] Among them, the current environment type is the category of the physical environment around the vehicle determined based on the spatial coordinate model, and the interactive space range is the physical space area that can project interactive content, defined from the spatial coordinate model.

[0075] Optionally, terrain features and external objects of the spatial coordinate model can be extracted, and the environment type can be determined based on the extracted terrain features and external objects; the interactive range can be delineated based on unobstructed and projectable spatial features.

[0076] S503. Extract information from user interaction information to obtain a third parsing result, which includes: interaction instructions.

[0077] Interaction instructions are the interactive operation instructions that the user expects the system to execute, extracted from the user's interactive input information.

[0078] Optionally, a motion recognition algorithm can be used to analyze the user's gestures or body movements and convert them into standardized interaction commands, or semantic recognition can be performed on the user's voice interaction information or touch interaction information to extract operation requirements and generate corresponding interaction commands.

[0079] S504. Analyze the vehicle state parameters to obtain the fourth analysis result, which includes the vehicle's energy level, interaction range, and feedback intensity.

[0080] Among them, energy level is the energy supply level determined based on parameters such as vehicle battery level, which determines the system's ability to output interactive content. Interaction range is the range within which the system can achieve interaction, defined based on vehicle structural state parameters. Feedback intensity is the intensity level of physical feedback that the vehicle can provide, determined based on the state of the vehicle's physical components.

[0081] Optionally, the vehicle's battery level can be mapped to an energy level, with higher battery levels resulting in higher energy levels; the interaction range can be defined based on the status of the doors and tailgate; and the feedback intensity can be determined based on the status of the seat actuators.

[0082] S505. Generate interactive content based on the first parsing result, the second parsing result, the third parsing result, and the fourth parsing result.

[0083] In one possible implementation, the first, second, third, and fourth parsing results can be input into a preset content generation model. The model automatically generates interactive content based on the environment type (scenario), user interaction mode (style), interaction instructions (content), and vehicle parameters (output constraints).

[0084] The following is a further explanation of how the interactive content was generated based on the first, second, third, and fourth parsing results, such as... Figure 6 As shown, the above S505 step includes: S601. The first parsing result, the second parsing result, the third parsing result, and the fourth parsing result are fused to obtain the fused result. The target content generation strategy is obtained by matching the fused result in the preset strategy library.

[0085] The preset strategy library includes a set of spatial interaction content generation strategies for different scenarios, containing content generation rules and logic for various scenarios. The target content generation strategy is a set of content generation rules and logic matched from the strategy library and adapted to the current fusion result.

[0086] Optionally, features can be extracted and quantified from the first, second, third, and fourth parsing results. The results can then be compared with the policy features in the policy library using a similarity algorithm, and the policy with the highest similarity can be selected as the target policy.

[0087] As another possible implementation, scene tags can be set for the first, second, third, and fourth parsing results, and the combined tags can be precisely matched with the strategy tags in the strategy library. The successful match is the target content generation strategy.

[0088] S602. Generate interactive content based on the target content generation strategy, the first parsing result, the second parsing result, the third parsing result, and the fourth parsing result.

[0089] Optionally, the target generation strategy and the four parsing results can be input into the content rendering engine. The engine dynamically adapts the information of each dimension according to the strategy rules, automatically generating and rendering complete interactive content. Alternatively, the target generation strategy can be used as a framework to set the content style based on the first parsing result, define the content space boundaries based on the second parsing result, determine the core content type based on the third parsing result, and limit the content output intensity based on the fourth parsing result, thus constructing and generating interactive content layer by layer.

[0090] Optionally, the vehicle includes a projection unit. The projection unit is a display device mounted on the vehicle for projecting interactive content; for example, it can be built based on an LCoS microdisplay chip. The process described above, based on a spatial coordinate model and vehicle state parameters, projects interactive content onto a target projection area of ​​the vehicle, as follows: Figure 7 As shown, it includes: S701. Determine the target projection area based on the interactive content, spatial coordinate model, and projection parameters of the projection unit.

[0091] Projection parameters are the inherent properties and operating parameters of the projection unit, including installation posture, projection angle, resolution, and projection range. The target projection area is the physical projection area inside and outside the vehicle that is adapted to the interactive content display and matches the spatial coordinate model.

[0092] Optionally, the projection range of the projection unit can be matched based on the spatial dimensions of the interactive content and the unobstructed projectable area of ​​the spatial coordinate model, and the spatial coordinates of the target projection area can be calibrated. Alternatively, based on the installation posture and projection angle of the projection unit, combined with the terrain and vehicle structure of the spatial coordinate model, an area that can fully display the interactive content can be selected as the target projection area.

[0093] S702. Determine the initial projection image of the interactive content in the target projection area based on the spatial coordinate model.

[0094] The initial projected image is a raw digital image formed by mapping interactive content onto the target projection area based on a spatial reference of a spatial coordinate model.

[0095] Optionally, the three-dimensional virtual coordinates of the interactive content can be converted into two-dimensional planar coordinates of the target projection area based on the spatial coordinate model, and the initial projection image can be rendered. Alternatively, the interactive content can be attached to the physical surface of the target projection area based on the terrain and external object spatial features of the spatial coordinate model, and an initial projection image matching the spatial shape can be generated.

[0096] S703. Based on the vehicle state parameters and spatial coordinate model, the initial projected image is corrected to obtain the target projected image.

[0097] The target projection image is a projection image that is corrected by vehicle state parameters and spatial coordinate model and matched with the real physical space and vehicle state.

[0098] Optionally, the initial projected image can be corrected by perspective and stretching based on the vehicle attitude data in the vehicle state parameters and the terrain slope of the spatial coordinate model to obtain the target projected image.

[0099] S704. Perform brightness compensation on the target projection image according to the projection incident angle, and project the target projection image onto the target projection area of ​​the vehicle.

[0100] The projection incident angle is the angle between the light emitted by the projection unit and the surface of the target projection area.

[0101] Optionally, the projection incident angle at each position in the target projection area can be calculated. The larger the incident angle, the higher the brightness of the corresponding pixel. After compensation, the image is projected through the projection unit. Alternatively, a mapping table between the projection incident angle and the brightness compensation coefficient can be pre-established. The mapping relationship is found according to the mapping table, the brightness of the target projection image is adjusted, and then multiple projection units are driven to project synchronously onto the target area.

[0102] In this embodiment, defining the target projection area using multi-dimensional information ensures that the projection area matches the interactive content and hardware capabilities. By combining vehicle status and spatial model to correct the image, projection misalignment and distortion can be eliminated. Optimizing the display effect through brightness compensation achieves accurate and uniform projection of the interactive content, while ensuring spatial consistency between the projected content and the real environment.

[0103] The following is a further explanation of how the initial projected image is corrected based on vehicle state parameters and a spatial coordinate model to obtain the target projected image. Figure 8 As shown, the above step S703 includes: S801. Extract the ground features of the target projection area from the spatial coordinate model.

[0104] Ground features are the spatial geometric and physical properties of the ground within the target projection area, including features such as terrain slope, surface curvature, height distribution, and flatness.

[0105] Optionally, three-dimensional coordinate data of the target projection area in the spatial coordinate model can be extracted, and the terrain slope and height difference can be calculated by the algorithm to obtain ground features. Alternatively, surface feature analysis can be performed on the target projection area in the spatial coordinate model to extract geometric continuity, normal features, etc., and the extracted data can be used as ground features.

[0106] S802. Extract vehicle body features from the spatial coordinate model and vehicle state parameters.

[0107] Vehicle body features are spatial and state attribute information of the vehicle body related to projection, including features such as vehicle body posture, vehicle body tilt angle, and relative changes in the installation posture of the projection unit.

[0108] Optionally, the three-dimensional structural data of the vehicle body can be extracted from the spatial coordinate model, and combined with the vehicle body attitude data in the vehicle state parameters, the vehicle body tilt angle can be calculated to obtain the vehicle body features.

[0109] S803. Based on the ground features and vehicle features, correct each pixel in the initial projected image to obtain the intermediate projected image.

[0110] In one possible implementation, pixel offsets can be calculated based on ground slope and vehicle tilt angle. Coordinate correction and perspective transformation can then be performed on each pixel of the initial projected image to generate an intermediate projected image. Alternatively, a mapping relationship can be established between ground features, vehicle features, and pixel correction. Pixels can then be stretched and rotated according to this relationship to obtain the corrected intermediate projected image.

[0111] S804. Determine the occlusion relationship of the intermediate projected image based on the spatial coordinate model and generate an occlusion mask.

[0112] An occlusion mask is a mask that marks the pixel area in an intermediate projected image that is occluded by a real entity, and is used to block the projected display of the occluded area.

[0113] Optionally, based on the depth data of the spatial coordinate model, it can be determined whether the spatial position of the virtual content corresponding to each pixel in the intermediate projected image is occluded by a real object, and the occluded area can be marked to generate a mask.

[0114] In another possible implementation, a projection ray can be constructed, and the occlusion relationship on the ray path can be determined by combining the entity position of the spatial coordinate model, marking the occluded pixels and generating an occlusion mask.

[0115] S805. Obtain the target projection image based on the occlusion mask and the intermediate projection image.

[0116] Optionally, the occlusion mask can be superimposed on the intermediate projection image to block the pixel areas marked by the mask and retain the image content of the unmasked areas to obtain the target projection image.

[0117] The following is a further explanation of the above method for determining the occlusion relationship of the intermediate projected image based on the spatial coordinate model and generating an occlusion mask, such as... Figure 9 As shown, the above S804 step includes: S901. Traverse each pixel of the intermediate projected image, and for the current pixel that has been traversed, determine the ray in space for the current pixel.

[0118] S902. Search for the object hit by the ray in space along the ray of the current pixel, and obtain the first depth value of the object hit.

[0119] S903. If the first depth value is less than the depth value of the current pixel in the intermediate projected image, then the current pixel is determined to be an occluded pixel.

[0120] S904. After determining all the occluded pixels, generate an occlusion mask based on each occluded pixel.

[0121] Optionally, the pixels of the intermediate projected image can be traversed in row and column order. Based on the installation pose of the projection unit and the pixel coordinates, the direction and starting point of the projected ray are calculated using a spatial geometric algorithm to determine the ray of the pixel in space. Then, the extension direction of the spatial ray is determined, and entity data in the spatial coordinate model is retrieved. The data is sorted from near to far, and the depth value of the first entity is extracted as the first depth value. The first depth value is then compared with the depth value of the current pixel. If the first depth value is smaller, the pixel is directly marked as an occluded pixel. Finally, the coordinates of all occluded pixels are set and labeled to generate a binary mask with the same size as the intermediate projected image. The occluded pixels are marked as mask values, and the rest are displayed values.

[0122] During the interaction, the interactive content can be adjusted in real time based on changes in the vehicle and user information, such as... Figure 10 As shown, the method of this application further includes: S1001: Real-time collection of vehicle environmental data, user identity information, user interaction information, and vehicle status parameters.

[0123] Optionally, a fixed time interval collection frequency can be set to continuously collect environmental data through LiDAR, cameras, etc., while biometric devices, motion sensors, etc. simultaneously collect user and interaction data, and the vehicle detection unit collects vehicle status parameters. Alternatively, a trigger-based collection mechanism can be adopted, which immediately starts collecting the corresponding data when fluctuations in environmental, user, or vehicle status data are detected.

[0124] S1002. Update the spatial coordinate model based on vehicle environment data, user identity information, user interaction information, and vehicle status parameters to obtain the updated model.

[0125] Optionally, the real-time collected change data can be compared with the original model data, the coordinates of the changed areas can be corrected, and an updated model can be generated.

[0126] S1003. Correct the target projection area and / or interactive content of the vehicle based on the updated model.

[0127] If the updated model shows that the original projection area is obstructed, the target projection area can be directly corrected to an unobstructed space; if the environmental or vehicle conditions change slightly, only the spatial position and display brightness of the interactive content can be corrected.

[0128] In another possible implementation, the spatial features of the updated model can be input into the correction algorithm. The algorithm automatically determines whether the target projection area needs to be adjusted. If adjustment is needed, the target projection area of ​​the vehicle and / or the interactive content are corrected.

[0129] In this embodiment, the spatial coordinate model is updated based on real-time data, enabling the model to match the real scene. Then, the projection area and / or interactive content are corrected according to the updated model, realizing the dynamic adaptive adjustment of the system and ensuring that the projection and interactive content always fit the real-time scene, avoiding interaction misalignment caused by scene changes.

[0130] Based on the same inventive concept, this application also provides an in-vehicle interaction system corresponding to the in-vehicle interaction method. Since the principle of the system in this application is similar to that of the in-vehicle interaction method described above, the implementation of the system can refer to the implementation of the method, and the repeated parts will not be described again.

[0131] Figure 11The diagram illustrates the structure of an electronic device according to an embodiment of this application, including a processor 1101, a storage medium 1102, and a bus 1103. The storage medium 1102 stores machine-readable instructions executable by the processor 1101. When the electronic device runs an in-vehicle interaction method as described in the embodiment, the processor 1101 communicates with the storage medium 1102 via the bus 1103. The processor 1101 executes the machine-readable instructions, and the preamble of the method item of the processor 1101 executes the steps in the above-described in-vehicle interaction method.

[0132] This application also provides a computer-readable storage medium storing a computer program that is executed by a processor, which performs the steps in the above-described vehicle interaction method.

[0133] In this embodiment, the computer program, when run by the processor, can also execute other machine-readable instructions to perform other methods as described in the embodiments. For details on the specific execution steps and principles, please refer to the description of the embodiments, which will not be repeated here.

[0134] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0135] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0136] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0137] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0138] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0139] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A vehicle-mounted interaction method, characterized in that, include: Acquire vehicle environmental data, user identity information, user interaction information, and vehicle status parameters; A spatial coordinate model of the vehicle is constructed based on the vehicle environment data. The spatial coordinate model is a virtual model that includes the vehicle and its surrounding environment. Interactive content is generated based on the spatial coordinate model, the user identity information, the user interaction information, and the vehicle status parameters; When the vehicle is in a parked state, the interactive content is projected onto the target projection area of ​​the vehicle based on the spatial coordinate model and the vehicle state parameters.

2. The method according to claim 1, characterized in that, The step of constructing the spatial coordinate model of the vehicle based on the vehicle environment data includes: The vehicle environmental data is subjected to time synchronization processing and noise filtering processing to obtain processed environmental data; The processed environmental data is segmented according to its region to obtain initial vehicle body region data, initial terrain region data, and initial external object data. A vehicle coordinate system is established based on the vehicle, and the target vehicle area data, target terrain area data, and target external object data corresponding to the initial vehicle area data, the initial terrain area data, and the initial external object data are determined respectively in the vehicle coordinate system. The spatial coordinate model is established based on the coordinate and attribute information of the target vehicle area data, the target terrain area data, and the target external object data.

3. The method according to claim 2, characterized in that, The process of segmenting the processed environmental data according to its region yields initial vehicle area data, initial terrain area data, and initial external object data, including: The processed environmental data is matched with a preset vehicle body structure model to obtain the initial vehicle body area data. The initial terrain region data is obtained by identifying the geometric features of the terrain surface from the processed environmental data; The data in the processed environmental data, excluding the initial vehicle area data and the initial terrain area data, are used as the initial external object data.

4. The method according to claim 1, characterized in that, The step of generating interactive content based on the spatial coordinate model, the user identity information, the user interaction information, and the vehicle status parameters includes: The user identity information is parsed to obtain a first parsing result, which includes: user permissions and user interaction mode; The spatial coordinate model is analyzed to obtain a second analysis result, which includes: the current environment type and the interactive space range; Information is extracted from the user interaction information to obtain a third parsing result, which includes: interaction instructions; The vehicle state parameters are analyzed to obtain a fourth analysis result, which includes the vehicle's energy level, interaction range, and feedback intensity. The interactive content is generated based on the first parsing result, the second parsing result, the third parsing result, and the fourth parsing result.

5. The method according to claim 4, characterized in that, The step of generating the interactive content based on the first parsing result, the second parsing result, the third parsing result, and the fourth parsing result includes: The first parsing result, the second parsing result, the third parsing result, and the fourth parsing result are fused to obtain a fusion result. Based on the fusion result, a target content generation strategy is obtained by matching in a preset strategy library. The interactive content is generated based on the target content generation strategy, the first parsing result, the second parsing result, the third parsing result, and the fourth parsing result.

6. The method according to claim 1, characterized in that, The vehicle includes: a projection unit; The step of projecting the interactive content onto the target projection area of ​​the vehicle based on the spatial coordinate model and the vehicle state parameters includes: The target projection area is determined based on the interactive content, the spatial coordinate model, and the projection parameters of the projection unit. The initial projection image of the interactive content in the target projection area is determined based on the spatial coordinate model. Based on the vehicle state parameters and the spatial coordinate model, the initial projection image is corrected to obtain the target projection image; The brightness of the target projection image is compensated according to the projection incident angle, and the target projection image is projected onto the target projection area of ​​the vehicle.

7. The method according to claim 6, characterized in that, The step of correcting the initial projected image based on vehicle state parameters and the spatial coordinate model to obtain the target projected image includes: The ground features of the target projection area are extracted from the spatial coordinate model; Vehicle body features are extracted from the spatial coordinate model and the vehicle state parameters; Based on the ground features and the vehicle body features, each pixel in the initial projection image is corrected to obtain the intermediate projection image; Based on the spatial coordinate model, the occlusion relationship of the intermediate projection image is determined, and an occlusion mask is generated. The target projection image is obtained based on the occlusion mask and the intermediate projection image.

8. The method according to claim 7, characterized in that, The step of determining the occlusion relationship of the intermediate projected image based on the spatial coordinate model and generating an occlusion mask includes: Traverse each pixel of the intermediate projected image, and for the current pixel that has been traversed, determine the ray in space for the current pixel; The first depth value of the object is obtained by tracing the ray of the current pixel in space to find the object that hits the ray in space. If the first depth value is less than the depth value of the current pixel in the intermediate projected image, then the current pixel is determined to be an occluded pixel. After determining all the occluded pixels, the occlusion mask is generated based on each of the occluded pixels.

9. The method according to claim 1, characterized in that, The method further includes: Real-time collection of vehicle environmental data, user identity information, user interaction information, and vehicle status parameters; The spatial coordinate model is updated based on the vehicle environment data, user identity information, user interaction information, and vehicle status parameters to obtain the updated model. The target projection area of ​​the vehicle and / or the interactive content are corrected based on the updated model.

10. An in-vehicle interactive system, characterized in that, The system includes a vehicle and at least one projection unit disposed on the vehicle, the system being used to perform the steps of the in-vehicle interaction method as described in any one of claims 1-9.

Images