A method, virtual device, medium, and equipment for interaction between a vehicle and a user.

Abstract

This specification provides a method, virtual device, medium, and equipment for interaction between a vehicle and a user, applied to a central control device of a vehicle, which also includes a camera device and a projection device. The method includes: determining a user action area and a display area in the external environment of the vehicle; controlling the camera device to continuously capture images of the user action area, and determining the positional range of the user's actions within the user action area by comparing the continuously captured two-dimensional images of the action area; determining a triggered interactive sub-area based on the positional range of the user's actions within the user action area; and controlling the projection device to project onto the display area based on a projection scheme associated with the triggered interactive sub-area. This method provides an interaction scheme for the user.

Description

Technical Field

[0001] This specification relates to the field of human-computer interaction technology, and in particular to an interaction method, virtual device, medium and equipment between a vehicle and a user. Background Technology

[0002] Current vehicles (such as electric vehicles) are equipped with devices to provide driving functions and improve driving safety, such as the vehicle's central control unit, on-vehicle cameras (mainly used to capture road conditions), and on-vehicle projection devices (mainly used to project road condition warning information onto the road surface).

[0003] In practical applications, users often expect to experience more functions based on vehicles in addition to using them to meet their travel needs. Therefore, this application provides an interaction scheme between the vehicle and the user. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this specification provides a method for interaction between a vehicle and a user, a virtual device, a medium, and an equipment.

[0005] According to a first aspect of the embodiments of this specification, a method for interaction between a vehicle and a user is provided, applied to a central control device of the vehicle, wherein the vehicle further includes a camera device and a projection device; the method includes:

[0006] In the external environment of the vehicle, a user action area and a display area are determined; wherein, the user action area includes several interactive sub-areas, and each interactive sub-area is associated with a projection scheme for the display area;

[0007] The camera device is controlled to continuously capture images of the user's action area, and the location range of the user's action is determined in the user's action area by comparing the two-dimensional images of the action area obtained by continuous capture.

[0008] Based on the location range of the user action within the user action area, determine the triggered interactive sub-area;

[0009] Based on the projection scheme associated with the triggered interactive sub-region, the projection device is controlled to project onto the display area.

[0010] According to a second aspect of the embodiments of this specification, an interactive virtual device between a vehicle and a user is provided. The virtual device is applied to a central control device of the vehicle, and the vehicle further includes a camera device and a projection device. The virtual device includes:

[0011] The first determining unit is used to determine the user action area and the display area in the external environment where the vehicle is located; wherein, the user action area includes a plurality of interactive sub-areas, and each interactive sub-area is associated with a projection scheme for the display area;

[0012] The second determining unit is used to control the camera device to continuously capture images of the user's action area, and to determine the position range of the user's action in the user's action area by comparing the two-dimensional images of the action area obtained by continuous capture.

[0013] The third determining unit is used to determine the triggered interactive sub-region based on the location range of the user action in the user action region.

[0014] The projection unit is used to control the projection device to project onto the display area based on the projection scheme associated with the triggered interactive sub-area.

[0015] According to a third aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the interaction method between a vehicle and a user as described in any of the embodiments of the first aspect.

[0016] According to a fourth aspect of the embodiments of this specification, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes steps of an interaction method between a vehicle and a user as described in any of the embodiments of the first aspect.

[0017] The technical solutions provided in the embodiments of this specification may include the following beneficial effects:

[0018] In the embodiments of this specification, in addition to using vehicles to meet travel needs, an interaction scheme between the vehicle and the user is also provided, extending the function of the vehicle to multiple other interactive service dimensions such as life, entertainment, and work.

[0019] Meanwhile, the method for enabling interaction between the vehicle and the user by utilizing the existing central control device, camera device, and projection device in the vehicle, as provided in the embodiments of this specification, determines the position of the user's actions through two-dimensional image comparison. Compared with existing technologies that require real-time identification of the user's limbs, sensing the distance of the user's limbs, and recognizing the user's limb posture, the embodiments of this specification do not require additional sensor devices in the vehicle, thus reducing the vehicle's production cost. Furthermore, the embodiments of this specification do not require real-time tracking of the user's limbs; they only need to determine the position of the user's actions, reducing the computational load on the central control device and improving the interaction response speed without increasing the computational burden on the vehicle.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0022] Figure 1 A flowchart illustrating an interaction method between a vehicle and a user, as provided in an embodiment of this specification, is shown.

[0023] Figure 2a This specification illustrates a first type of regional deployment requirement provided by an embodiment.

[0024] Figure 2b This specification illustrates a second type of regional deployment requirement provided by an embodiment.

[0025] Figure 3 This specification illustrates a third type of regional deployment requirement provided by an embodiment.

[0026] Figure 4 This diagram illustrates a method for determining a triggered interactive sub-region, as provided in an embodiment of this specification.

[0027] Figure 5 This diagram illustrates another method for determining the triggered interactive sub-region, as provided in an embodiment of this specification.

[0028] Figure 6 This specification illustrates a regional deployment requirement for presenting a PowerPoint presentation, as provided in an embodiment.

[0029] Figure 7 Showing the target Figure 6 A diagram illustrating the deployment of interactive sub-regions to determine the triggered interactive sub-regions.

[0030] Figure 8 This specification illustrates a hardware structure diagram of a computer device containing the interactive virtual device provided in an embodiment of this specification. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0032] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” as used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0033] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0034] The vehicle described in this disclosure can be any device capable of carrying one or more users, such as a car, motorcycle, boat, airplane, etc. For ease of description, the following text will primarily use a car (more specifically, a smart electric vehicle) as an example.

[0035] Users typically use vehicles for transportation purposes, and when users don't need to travel, the vehicles are essentially idle. Therefore, how to provide users with more diverse functional services beyond transportation based on these idle vehicles is key for vehicle manufacturers to attract customers.

[0036] Some vehicles on the market often have video recording functions for capturing road conditions and projection functions for projecting road condition warning information. Therefore, a feasible approach is to utilize these existing video recording and projection functions to provide interactive services for users.

[0037] Furthermore, while completing more life interaction services through the existing equipment in the vehicle, due to the limited space in the vehicle, these life interaction services should not be limited to the interior of the vehicle. When extending the interaction services to the exterior of the vehicle, such as open roads, building walls, lakes, and other environments, it is necessary to provide users with a larger display interface, more realistic and broader action interaction needs, and more efficient interaction feedback through the equipment on the vehicle itself.

[0038] The following uses a car as an example to briefly introduce a few interaction scenarios:

[0039] When there is an open ground or wall in the car's external environment, a game interface (or audio / video playback, PPT presentation, or online meeting) can be displayed in a pre-defined area on the ground (or wall). A user action area can also be defined on the ground (or wall) to provide a dedicated interaction location for the user. If the user action area is on the ground, when the user steps on the ground within the user action area, the system will determine the location of the step and trigger a relevant interaction command, which will then be reflected in the corresponding game interface to achieve game interaction (or a corresponding display interface to achieve video switching, PPT page turning, meeting operations, etc.). If the user action area is on the wall, when the user clicks on the wall within the user action area, the system will determine the location of the click and trigger a relevant interaction command, which will then be reflected in the corresponding game interface to achieve game interaction (or a corresponding display interface to achieve video switching, PPT page turning, meeting operations, etc.).

[0040] When a car is parked by a lake or on a bridge, the user action area and display area can be set on the lake surface, the water surface, or the surrounding flat surface, etc., to provide interactive features. For example, a user can fish on the water. The user action area is defined in a specific area on the water surface, and the display area is defined on the water surface, the surface above the water, or the surrounding road or wall. When the user action area is detected and it is determined that the user has placed a fishing float in the user action area, if the user selects the "Real Fishing" option, then the display area will project auxiliary fishing images and information (such as the number of fish caught, the current time, the number of fish in this water area, weather information, and warnings, etc.); if the user selects the "Simulated Fishing" option, then the display area will play fishing tutorial videos, fishing techniques, etc.

[0041] Based on the above-described application scenarios, this specification will now provide a detailed description of the embodiments regarding how to provide a method for interaction between a vehicle and a user based on the equipment mounted on the vehicle itself.

[0042] This specification provides an embodiment of an interaction method between a vehicle and a user, applied to the central control device of the vehicle, which also has a camera device and a projection device. Figure 1 A flowchart illustrating an interaction method between a vehicle and a user, as provided in an embodiment of this specification, is shown. Figure 1 As shown, the method includes the following steps:

[0043] Step 101: Determine the user action area and the display area in the external environment where the vehicle is located; wherein, the user action area includes several interactive sub-areas, and each interactive sub-area is associated with a projection scheme for the display area.

[0044] The central control unit (CCU) enables interaction between the user and the vehicle, between the server and the vehicle, and between the vehicle and other vehicles. The CCU includes the central console, central control system, and central controller within the vehicle. For example, when the vehicle is a car, the CCU could be the car's central console or its in-vehicle infotainment system. Cameras and projection devices are built into the vehicle and capable of capturing two-dimensional images. When the vehicle is a car, cameras could be front-view cameras, rear-view cameras, surround-view cameras, etc.; projection devices include in-vehicle projection equipment (used for projecting audio and video) and projection lights mounted on the car body (used for projecting road condition warning information). The CCU communicates with and controls the camera and projection devices.

[0045] The external environment refers to the environment surrounding the vehicle, such as the outdoor environment where the vehicle travels or the warehouse where the vehicle is placed. The environment surrounding the vehicle may include roads, water surfaces, walls, other vehicles, buildings, etc.

[0046] The user action area is the area where the user interacts with the central control device. The user performs corresponding interactive actions in the user action area, which can be the walls, road surface, water surface, etc., around the car. The display area is the area where content is displayed to the user. The user action area is within the shooting range of the camera device, and the display area is within the projection range of the projection device; both the user action area and the display area are within the user's visible range, and the user action area is within the user's operability range (e.g., it cannot be under the car, or projected onto other vehicles that would obstruct the user action area). The user action area can be the user's own physical actions within the user action area, or actions can be performed through other entities within the user action area (e.g., operating through a tree branch, or placing physical objects such as stones within the user action area). The embodiments in this specification do not limit the methods for determining the user action area and the display area. The user action area can be manually input by the user, preset for different interactive content, or automatically selected by the central control device based on the external environment.

[0047] After determining the user action area, the user action area is divided into several interactive sub-areas, and a projection scheme is associated with each interactive sub-area. This allows the projection scheme of the display area to change accordingly when different user actions trigger different interactive sub-areas. By changing the projection scheme, interactive feedback is provided to the user actions, enabling timely response to user actions.

[0048] Step 102: Control the camera device to continuously capture images of the user's action area, and determine the position range of the user's action within the user's action area by comparing the two-dimensional images of the action area obtained from the continuous captures.

[0049] Specifically, in order to determine whether the user is interacting within the user action area, the central control device controls the camera device to continuously capture images of the user action area. Continuous capturing can be periodic according to a preset cycle, or non-periodic capturing according to specified key time points (such as preset switching time points based on game progress or video progress), thereby capturing the user's actions.

[0050] Furthermore, when continuous shooting is periodic, the period size can be adjusted according to the motion capture accuracy required by the interactive content. If the current interactive content requires high motion capture accuracy, the preset period needs to be set small enough, such as 1ms, 10us, etc. If the current interactive content requires low motion capture accuracy, the preset period can be set larger, such as 1min, 30min, etc.

[0051] After continuous shooting, a two-dimensional image of the motion region containing the user's action area is obtained. This two-dimensional image can be obtained directly from the shot or through post-shot processing. The user's actions in the two-dimensional images of the motion region at different times may be the same or different. Multiple two-dimensional images of the motion region at different times must at least contain the entire action process of the user in the motion region from when no action occurred to when an action occurred. By comparing the two-dimensional images of the motion region at different times, the positional range of the user's action within the user's motion region can be determined.

[0052] For example, when the user's action area is the ground, if the user steps on the ground within the user's action area, the location range is the area where the user's foot is active; if the user lies flat within the user's action area, the location range is the area where the user's body is in contact with the user's action area.

[0053] Step 103: Determine the triggered interactive sub-region based on the location range of the user action in the user action region.

[0054] Specifically, after determining the location range of the user action within the user action area according to step 102, the triggered interactive sub-area is determined based on the relationship between the location range and the interactive sub-areas pre-defined within the user action area. The triggered interactive sub-area can be an interactive sub-area covered by the location range, or it can be an uncovered interactive sub-area that has a specific positional relationship with the location range.

[0055] Step 104: Based on the projection scheme associated with the triggered interactive sub-region, control the projection device to project onto the display area.

[0056] Specifically, after determining the triggered interactive sub-area according to step 103, the projection device is controlled to project onto the display area according to the projection scheme associated with the triggered interactive sub-area.

[0057] Using the above method, when a user action occurs in the user action area, the location range of the user action in the user action area is determined based on the two-dimensional image of the user action area captured by the image. Then, the triggered interactive sub-area and the projection scheme associated with the triggered interactive sub-area are determined by the location range, and the projection is performed on the display area to complete the interactive feedback of the user action.

[0058] It should be noted that for each interactive sub-region, the projection scheme associated with that interactive sub-region at different times can be the same, or it can be continuously adjusted and changed as the game progresses, the audio and video progresses, etc.

[0059] In one feasible implementation, when performing step 101 to determine the user's action area in the external environment where the vehicle is located, the following steps are included:

[0060] Step 1011: Determine a user action area that meets safety requirements in the external environment where the vehicle is located; wherein the safety requirements include: the surface flatness is greater than a first preset degree, and / or the surface tilt relative to the plane is less than a second preset degree.

[0061] Specifically, when the user's action area is an area automatically selected by the central control device on the road surface or wall surface in the external environment, the central control device needs to consider the interactive safety of the user's action area. For example, the flatness of the road surface or wall surface within the user's action area should be relatively high, and the tilt of the wall surface or ground should be relatively small; otherwise, it may interfere with the user's actions or even cause injury to the user. The tilt of the ground is relative to the horizontal plane, and the tilt of the wall surface is relative to the vertical plane.

[0062] The values ​​of the first and second preset levels can be adjusted according to the actual interaction content and items, or according to the user's actual interaction needs.

[0063] When there are roads or walls around the car, the central control unit acquires two-dimensional images of the roadside and walls in the external environment, analyzes the external environment, and automatically determines the user action area that meets safety requirements, and / or automatically determines the display area that meets display requirements.

[0064] The above methods can ensure user safety when performing actions within the designated user action area. Furthermore, to enhance safety, the safety requirements can be supplemented with assessments of road condition information, such as selecting the user action area as the side furthest from oncoming traffic.

[0065] It's important to note that display requirements can also be set for the display area, allowing you to define an area that meets these requirements from the external environment. For example, the color of the display area can be set to not affect the projection display effect (e.g., the color contrast between the display area's planar color and the main background color in the projection scheme is greater than a preset contrast value), the flatness of the display area can be greater than a third preset level, and the size of the display area can meet minimum display requirements. The user action area and the display area can be confirmed simultaneously or separately.

[0066] It is also important to note that when there is no user action area that meets safety requirements in the external environment of the vehicle, or no display area that meets display requirements, or when the user action area / display area entered by the user does not meet safety requirements / display requirements, the central control device provides warning information through sound and light warnings, interface display, etc., thereby reminding the user to change the vehicle parking location or stop interactive services, ensuring the safety of the user's actions in the user action area and the experience of the projection in the display area.

[0067] It's also important to note that when there's no user action area in the external environment that meets safety requirements, warning messages don't need to be sent to the user. Instead, the system can automatically select an interaction scheme (including several projection schemes) that adapts to the flatness and tilt of the external environment, ensuring a seamless content switching service even when the external environment doesn't meet preset requirements (safety or display requirements). Alternatively,

[0068] When there is no user action area in the external environment that meets safety requirements, if a warning message is issued to the user and the user ignores the warning message (or the user refuses to change the vehicle parking position), the system will automatically select an interaction scheme (including several projection schemes) that is suitable for the user based on the flatness and tilt of the plane in the external environment. By fully considering the user's wishes through the above method, the system ensures that the user continues to be provided with interactive services even when the user is aware that the area does not meet safety or display requirements, thereby improving user engagement.

[0069] In one feasible implementation, before performing step 101 to determine the user action area and display area in the external environment of the vehicle, the method further includes the following steps:

[0070] Step 1012: In response to the user's launch command for a specified interactive application, determine the regional deployment requirements associated with the specified interactive application, the interactive sub-region deployment requirements, and the projection scheme associated with each interactive sub-region.

[0071] Specifically, before determining the user action area and display area in step 101, the user pre-selects a specific interactive application they wish to launch. Examples include: a specified interactive game, a video to be played, an online meeting application, etc. Interactions in interactive games include, but are not limited to: performing game actions, placing game props, starting the game, pausing the game, etc.; interactions in videos to be played include, but are not limited to: sending viewing comments, switching video playlists, starting playback, pausing playback, adjusting playback speed, etc.; interactions in online meeting applications include, but are not limited to: joining a meeting, leaving a meeting, speaking, muting, etc.

[0072] Upon selecting a specific interactive application, a launch command for that application is generated. Based on the interactive requirements of the selected application, regional deployment requirements, interactive sub-region deployment requirements, and projection schemes associated with each interactive sub-region are generated. The regional deployment requirements include the positional relationship between the user action area and the display area, the size and shape of the user action area, and the size and shape of the display area. The interactive sub-region deployment requirements include how to divide the user action area into interactive sub-regions, the number of interactive sub-regions, and the positional relationship between each interactive sub-region.

[0073] The user action area and the display area can belong to the same area or different areas. Their positional relationship is determined according to the deployment requirements in the area deployment requirements, and includes at least two cases:

[0074] Scenario 1: The display area and the user action area are taken from the same area (both are the ground, or both are the walls, etc.). Figure 2a This specification illustrates a first type of regional deployment requirement provided by an embodiment, such as... Figure 2a One deployment method shown here divides a large area into a user action area and a display area: Figure 2a The display area and user action area shown are both on the ground, with the display area on the side closer to the vehicle projection device and the user action area on the side farther away from the vehicle projection device, thus avoiding the problem of the user's actions affecting the projection effect of the display area; in another area deployment requirement, the display area and user action area can also be set on the wall.

[0075] Figure 2b This specification illustrates a second type of regional deployment requirement provided by an embodiment, such as... Figure 2bThe diagram illustrates how, when the vehicle is a car and surround-view cameras are installed around the car, a display area and a user action area are defined around the car, as shown in the diagram. Figure 2b As shown, it should be noted that in order to prevent users from constantly moving around in the user action area and thus affecting the projection effect of the display area, the user action area is set outside the display area.

[0076] Furthermore, the projection range of the projection device in the car can cover both the display area and the user's action area; the surround-view camera can capture two-dimensional images of the outer user action area (and can also capture the display area at the same time depending on the angle setting, etc.).

[0077] Scenario 2: The display area and the user action area are taken from different areas (one on the wall, the other on the ground, etc.). Figure 3 This specification illustrates a third type of regional deployment requirement provided by an embodiment, such as... Figure 3 One of the deployment methods introduced is to take the user action area and the display area from different areas, namely: the display area is on the wall and the user action area is on the ground.

[0078] Regional deployment requirements also depend on the content of the specified interactive applications, thus different regional deployment requirements are set according to different types of interactive applications. By pre-setting different regional deployment requirements, interactive sub-region deployment requirements, and projection schemes associated with each interactive sub-region for different specified types of interactive applications, targeted responses to different applications can be achieved, optimizing the interactive experience. Furthermore, the relevant deployment requirements for each application can be automatically obtained through a pre-trained intelligent deployment model.

[0079] It is important to note that the regional deployment requirements for different designated interactive applications may be the same or different; the number and location of interactive sub-regions may be the same or different; all of these can be adjusted according to the content of the designated interactive application and the user's interaction needs. For example, if the designated interactive application remains unchanged, but the user's interaction needs change from single-person interaction to two-person interaction, the regional deployment requirements of the designated interactive application, or the deployment requirements of the interactive sub-regions, or the projection scheme associated with each interactive sub-region may be adjusted accordingly.

[0080] The following is based on Figure 2a Let's take a specific application scenario as an example, such as Figure 2a As shown, both the display area and the user action area are on the ground. When the vehicle is an electric car, the camera device is a surround-view camera on the electric car. The surround-view camera is installed on the outside of the vehicle body to capture the external environment around the vehicle body and generate a two-dimensional image.

[0081] The surround-view cameras are installed on the exterior of the vehicle (deployed around the vehicle body) and are at a certain height from the ground. Figure 2a When projecting in the scenario shown, due to the height difference between the surround-view camera and the ground, objects lower than the installation height of the surround-view camera entering the user's action area will be considered invalid changes in the user's action area. Conversely, when objects higher than the surround-view camera enter the user's action area (e.g., a human walking into the user's action area), since the height of the human body is necessarily higher than the installation height of the surround-view camera, the changes caused by the human body's actions in the two-dimensional image captured by the surround-view camera will be considered valid changes.

[0082] Meanwhile, since the surround-view camera is a camera device capable of capturing two-dimensional images, compared with other solutions that require stereo cameras to capture three-dimensional images in order to detect the human body, this solution has lower functional requirements for the camera device (it only needs to be able to capture two-dimensional images). At the same time, through image comparison, it can still effectively detect the position of the user's actions and quickly provide interactive feedback. This allows the user to have a better interactive experience even without a vehicle equipped with a three-dimensional image camera device (such as a high-precision three-dimensional image camera for autonomous driving) or an infrared sensor for recognizing human movements, through the embodiments in this specification.

[0083] When performing step 101 to determine the user action area and display area in the external environment of the vehicle, the specific steps include:

[0084] Step 1013: Define a user action area and a display area that meet the regional deployment requirements in the external environment where the vehicle is located, and define a number of interactive sub-areas that meet the interactive sub-area deployment requirements in the user action area.

[0085] Specifically, after determining the regional deployment requirements, interactive sub-region deployment requirements, and projection scheme associated with each interactive sub-region according to step 1012, the central control device delineates the user action area and display area according to the regional deployment requirements associated with the interactive application, and then divides several interactive sub-regions in the user action area according to the interactive sub-region deployment requirements, and associates a corresponding projection scheme with each interactive sub-region.

[0086] In one feasible implementation, when executing step 102, controlling the camera device to continuously capture images of the user's action area, the following specific situations are included:

[0087] Scenario 1: Control the camera device to continuously capture images of the user's action area in a fixed posture.

[0088] Specifically, when comparing two-dimensional images of different motion regions, it is necessary to ensure that the static features in these two-dimensional images remain unchanged, so as to obtain the dynamic features that show dynamic changes in the user's motion region. Static features refer to objects that do not change position over time, such as stationary vehicles, ground, buildings, etc. Dynamic features refer to objects that change position at different times, such as a rolling ball or limbs that change with the user's movements.

[0089] When the camera device remains in a fixed posture, the reference coordinate system of the two-dimensional images of the user's action area obtained by continuously capturing images of the user's action area in a fixed posture is consistent. The size, coverage, and alignment position of each two-dimensional image of the action area are the same, that is, the position of the static features in each two-dimensional image of the action area remains unchanged. When the user performs an action, dynamic features with dynamic position changes are generated in each two-dimensional image of the action area. Therefore, by comparing the two-dimensional images of each action area, the position range of the user's action on the user's action area can be determined.

[0090] For example, a fixed posture can be one where the camera's shooting plane is parallel to the plane containing the user's action area (i.e., looking down directly), and the boundary of the two-dimensional image of the action area is the boundary line of the user's action area; or, a fixed posture can be one where the camera's shooting plane forms a specific angle with the plane containing the user's action area (e.g., an oblique angle). By setting the fixed posture of the camera, a two-dimensional image of the action area can be obtained, showing whether the user's action actually contacts the user's action area.

[0091] or:

[0092] Scenario 2: Control the camera device to continuously capture images of the user's action area in a non-fixed shooting posture, and calibrate the two-dimensional images of each action area obtained by continuous shooting to the coordinate reference system under the same shooting posture.

[0093] Specifically, when the camera cannot capture the complete user action area due to factors such as user movement occlusion or the angle of the user action area, the camera's angle and posture need to be adjusted. In this case, to ensure that the 2D images of each action area obtained after adjusting the camera's posture do not cause misidentification, the 2D images of each action area need to be corrected to the same coordinate reference system, and that the positions of all static features are identical in each 2D image. Calibration methods include, but are not limited to, image geometric correction and image fusion. The 2D images of each action area obtained through this method are essentially 2D images of the action areas captured in a fixed posture.

[0094] For example, when shooting from different angles, all 2D images of the action area are calibrated to a unified viewpoint (e.g., a top-down view). When the 2D image of the action area obtained by switching angles cannot contain the complete user action area, two 2D images of the action area to be synthesized from different angles can be quickly captured in a short time and synthesized into a 2D image of the action area from a specific viewpoint. For example, the 2D images of the action area to be synthesized from the left and right viewpoints can be synthesized into a 2D image of the action area from a top-down viewpoint. The 2D image of the action area to be synthesized contains the relationship between the user action and the user action area (not in contact, in contact). The size and shape of the user action area in each 2D image of the action area are the same.

[0095] The above method eliminates interference caused by inconsistent coordinate reference systems of two-dimensional images or inconsistent positions of static features in two-dimensional images, ensuring that the result of determining the position range of the user action in the user action area during step 102 is more accurate.

[0096] It should be noted that, in a feasible implementation, the camera device can be a surround-view camera mounted on the vehicle, or other devices and apparatus with shooting capabilities. The projection device can be a standalone projector, projection equipment, a projection lamp integrated into the vehicle's headlights, a projection lamp on the vehicle body, or a projection device integrated into the central control unit.

[0097] In one feasible implementation, when performing step 102 to determine the location range of the user's action within the user's action region by comparing the two-dimensional images of the action region obtained through continuous shooting, the following steps are included:

[0098] Step 1021: Take at least two two-dimensional images of action regions that are adjacent in the shooting order as a two-dimensional image group to obtain a two-dimensional image group sequence.

[0099] Specifically, the number of images in a 2D image group is determined based on the user's required interaction precision. When high interaction precision is required, there are two 2D images of the action region in each 2D image group, and the two action region 2D images are compared to obtain a comparison result. Furthermore, according to the requirement that each 2D image group contains two 2D images of the action region, the number of 2D image groups in the 2D image group sequence is further determined based on the total number of captured action region 2D images.

[0100] To meet increasing accuracy requirements, the number of two-dimensional images of the motion region in a two-dimensional image group can be increased, such as 3, 7, etc. When generating a sequence of two-dimensional image groups, each two-dimensional image of the motion region can appear only once in that sequence of two-dimensional image groups, or it can appear multiple times in different two-dimensional image groups.

[0101] For example:

[0102] Example 1: Suppose four two-dimensional images of the action area are captured: two-dimensional image one, two-dimensional image two, two-dimensional image three, and two-dimensional image four. The division of the two-dimensional image group and the sequence of two-dimensional image groups can be as follows:

[0103] Scenario 1:

[0104] Two-dimensional image group one: two-dimensional image one, two-dimensional image two.

[0105] Two-dimensional image group two: two-dimensional image three, two-dimensional image four.

[0106] Two-dimensional image group sequence: Two-dimensional image group one, two-dimensional image group two.

[0107] Scenario 2:

[0108] Two-dimensional image group one: two-dimensional image one, two-dimensional image two, two-dimensional image three.

[0109] Two-dimensional image group two: two-dimensional image two, two-dimensional image three, two-dimensional image four.

[0110] Two-dimensional image group sequence: Two-dimensional image group one, two-dimensional image group two.

[0111] Scenario 3:

[0112] Two-dimensional image group one: two-dimensional image one, two-dimensional image two.

[0113] Two-dimensional image group two: two-dimensional image two and two-dimensional image three.

[0114] Two-dimensional image group three: two-dimensional image three and two-dimensional image four.

[0115] Two-dimensional image group sequence: Two-dimensional image group one, two-dimensional image group two, two-dimensional image group three.

[0116] Scenario 4:

[0117] Two-dimensional image group one: two-dimensional image one, two-dimensional image two, two-dimensional image three.

[0118] Two-dimensional image group two: two-dimensional image two, two-dimensional image three, two-dimensional image four.

[0119] Two-dimensional image group three: two-dimensional image three and two-dimensional image four.

[0120] Two-dimensional image group sequence: Two-dimensional image group one, two-dimensional image group two, two-dimensional image group three.

[0121] Based on the above scenarios, the accuracy of capturing user actions is adjusted to adapt to different interaction accuracy requirements. Compared with scenario one, scenario two maintains the same number of 2D image group sequences, but increases the number of 2D images analyzed in each group, resulting in higher processing accuracy. Compared with scenario one, scenario three maintains the same number of 2D images of the action region in the 2D image group, but increases the number of 2D image group sequences, resulting in higher processing accuracy. Compared with scenario two, scenario four achieves higher processing accuracy by adding a group of 2D image groups.

[0122] Step 1022: For the i-th two-dimensional image group, compare the difference between the first action region two-dimensional image of the two-dimensional image group and each other action region two-dimensional image; if the comparison result between the first action region two-dimensional image and each other action region two-dimensional image in the two-dimensional image group is greater than the first preset difference threshold, then the position range corresponding to the corresponding difference image part is taken as the position range of the user action; i = 1, 2, ..., N; N is the number of two-dimensional image groups.

[0123] Specifically, the first preset difference threshold is determined based on the degree of contact between the user's action and the user's action area. When the comparison result is greater than the first preset difference threshold, the difference image portion is considered to be the actual contact position between the user's action and the user's action area. The sequence of two-dimensional image groups contains N sequentially ordered two-dimensional image groups. For each two-dimensional image group, the first action area two-dimensional image in the group is compared with the other action area two-dimensional images. The portion where the comparison result between the first action area two-dimensional image and each of the other action area two-dimensional images is greater than the first preset difference threshold is determined. This portion is taken as the difference portion in the two-dimensional image, i.e., the difference image portion. The position range corresponding to this difference image portion is determined as the position range where the user's action is performed.

[0124] For example, if the user's interaction action is to step on a certain position in the user's action area with one foot, then continuously shooting the user's action area can obtain at least two two-dimensional images of the action area: a two-dimensional image of the action area before the user steps out, and a two-dimensional image of the action area after the user steps out.

[0125] When comparing these two two-dimensional images, the position where the user's foot is placed will be identified as the difference image part where the comparison result is greater than the first preset difference threshold. The position range of the difference image part (the position range covered by a single foot) will then be determined as the position range where the user's action is performed.

[0126] It is important to note that when a 2D image group includes three 2D images: one of the user's movement area before stepping out, one of the user's movement area during stepping out, and one of the user's movement area after stepping out. Since the 2D image of the movement area during the user's movement is actually an image of the movement area where the user's foot makes a stepping motion but does not actually touch the user's movement area, the comparison result between this 2D image and the 2D image of the movement area before stepping out in the difference image portion is less than a first preset difference threshold. Therefore, in the verification of this 2D image group, the comparison result between the first movement area 2D image in the 2D image group and each of the other movement area 2D images is not always greater than the first preset difference threshold, and the location range cannot be identified in the verification of this 2D image group.

[0127] In the next round of verification and recognition of the two-dimensional image group, according to the division scheme of the two-dimensional image group in Case 2 of Example 1 above, the next two-dimensional image group includes three two-dimensional images: the two-dimensional image of the action area during the user's foot movement, the two-dimensional image of the first action area after the user's foot movement, and the two-dimensional image of the second action area after the user's foot movement.

[0128] Based on the division scheme of the three two-dimensional image groups in Case 4 of Example 1 above, the next two-dimensional image group includes two two-dimensional images: a two-dimensional image of the action area during the user's foot movement, and a two-dimensional image of the first action area after the user's foot movement. In both cases, the location range can be identified within this round of two-dimensional image groups.

[0129] As can be seen, by comparing the two-dimensional images of the motion region in each two-dimensional image group obtained by continuous shooting, the location range of the user's action can be determined, and the specific time of the user's action can be determined based on the two-dimensional image group that has determined the user's action. By adjusting the precision of the two-dimensional image group division and the two-dimensional image group sequence, the amount of computation and the precision of motion capture can be adjusted.

[0130] In one feasible implementation, before performing step 102, which controls the camera device to continuously capture images of the user's action area, the method further includes the following steps:

[0131] Step 1023: Control the projection device to project a reference pattern onto the user's action area.

[0132] Specifically, each interactive sub-area contains a portion of the reference pattern, or each small area divided from the user action area contains a portion of the reference pattern. The reference pattern can be a graphic, character, text, etc. Different colors can also be used for differentiation within the reference pattern.

[0133] After performing step 1023, when performing step 1022 to compare the difference between the first action region two-dimensional image of the two-dimensional image group and each of the other action region two-dimensional images, the specific steps include:

[0134] Step 1024: Compare whether each of the other motion region two-dimensional images in the two-dimensional image group produces a new stereo distortion relative to the first motion region two-dimensional image; wherein, the stereo distortion includes the distortion produced when the reference pattern is occluded by a stereo object and the two-dimensional image obtained by taking a picture of the reference pattern is compared with the two-dimensional image obtained by taking a picture of the reference pattern before it was occluded.

[0135] Specifically, stereo distortion refers to the distortion of a reference pattern that should not be distorted at the obstructed location when a three-dimensional object (such as a user's limb) obstructs the user's movement area during projection. When the obstructing object is non-three-dimensional or difficult to identify (such as other patterns sprayed on the ground or paper), it will not cause stereo distortion of the reference pattern (e.g., from a straight line to a curve, from a square to an irregular shape, etc.). Therefore, if stereo distortion can be identified, and its location determined, it can effectively eliminate situations where other changes besides stereo distortion cause significant differences between the first action region 2D image and each of the other action region 2D images in a 2D image group (for example, when a newspaper is blown into the user interaction area by the wind, the newspaper's 3D shape is not distinct enough, so theoretically, even if the newspaper is blown flat by the wind and lies flat in the user interaction area, it will not affect the projection effect of the reference pattern, and therefore will not cause projection distortion of the reference pattern, i.e., it will not be considered as stereo distortion. Thus, even if the newspaper presents a dynamic effect while being blown by the wind, it will not be identified as a user action, eliminating some interaction interference and improving interaction accuracy).

[0136] Other motion region 2D images are any other motion region 2D images in this 2D image group besides the first motion region 2D image.

[0137] Since the central control device itself has the original shape of the reference pattern and multiple two-dimensional images of the user's action area, if each two-dimensional image of the action area in the two-dimensional image group contains the same stereo distortion at the same location, then the two-dimensional images of the action area in the two-dimensional image group are not considered to have generated new stereo distortion. Only when they contain different stereo distortions are they considered to have generated new stereo distortion.

[0138] Example 2: In the first two-dimensional image group, the two-dimensional image group includes: a first two-dimensional image of the user's foot at position A, and a second two-dimensional image of the user's foot at position A. Although both the first and second two-dimensional images exhibit stereo distortion at position A, the second two-dimensional image is not considered a new stereo distortion compared to the first two-dimensional image.

[0139] If the two-dimensional image group also includes a third two-dimensional image with one foot on position A and the other foot on position B, then compared with the first two-dimensional image, the stereo distortion caused by the foot being on position B in the third two-dimensional image is considered a new stereo distortion.

[0140] Step 1025: If each of the other two-dimensional motion regions in the two-dimensional image group produces a new stereo distortion relative to the first two-dimensional motion region image, then it is determined that the comparison result between the first two-dimensional motion region image and each of the other two-dimensional motion regions in the two-dimensional image group is greater than the first preset difference threshold.

[0141] Specifically, in step 1024, if it is determined that each other action region two-dimensional image produces a new stereo distortion relative to the first action region two-dimensional image, and the first action region two-dimensional image and each other action region two-dimensional image produce a new stereo distortion, then it is considered that the comparison result between the first action region two-dimensional image and each other action region two-dimensional image is greater than the first preset difference threshold, and the part with the new stereo distortion is taken as the difference image part, and then the position range of the new stereo distortion acting on the user action region is determined as the position range acting on the action.

[0142] In one feasible implementation, after determining the user action area in step 101, the user action area is divided into multiple embedding point areas.

[0143] Step 102 involves comparing the two-dimensional images of the action area obtained from continuous shooting to determine the location range of the user's action within the user's action area, including the following steps:

[0144] Step 201: A two-dimensional image group is formed by taking at least two adjacent two-dimensional images of motion regions in the shooting sequence. For the i-th two-dimensional image group, the difference between the first motion region two-dimensional image in the group and the image portion corresponding to the j-th embedded point region in each of the other motion region two-dimensional images is compared. If the comparison result between the first motion region two-dimensional image in the group and the image portion corresponding to the j-th embedded point region in each of the other motion region two-dimensional images is greater than a second preset difference threshold, then the j-th embedded point region is assigned to the region set corresponding to the i-th two-dimensional image group; i = 1, 2, ..., N; N is the number of two-dimensional image groups; j = 1, 2, ..., M; M is the number of embedded point regions.

[0145] Specifically, the division of two-dimensional image groups and two-dimensional image group sequences is explained in Example 1, and will not be repeated here. Set up tracking areas for user action areas. The size of the tracking areas can be adjusted according to user needs. The tracking areas can be evenly distributed in the user action area, or they can be unevenly distributed in the user action area according to different interaction content (for example, set dense, small-sized tracking areas in places where frequent interaction is required, and set scattered, large-sized tracking areas in places where frequent interaction is not required).

[0146] After dividing the sequence into two-dimensional image groups, for each two-dimensional image group, since each action region two-dimensional image in the two-dimensional image group contains the same number of embedded point regions, and the positions of each embedded point region are the same, based on the one-to-one correspondence of the embedded point regions in each action region two-dimensional image, each embedded point region in the first action region two-dimensional image is compared with each embedded point region in each of the other action region two-dimensional images to determine whether there is a difference. If the comparison result between the j-th embedded point region of the first action region and the j-th embedded point region of each of the other action region two-dimensional images is greater than the second preset difference threshold, then it is considered that the j-th embedded point region has indeed generated a user action, and the j-th embedded point region is assigned to the region set corresponding to the i-th two-dimensional image group.

[0147] Step 202: If the number of embedded regions in the region set corresponding to the i-th two-dimensional image group is greater than the preset number, then the location range formed by all embedded regions in the region set corresponding to the i-th two-dimensional image group is taken as the location range of a user action.

[0148] Specifically, each two-dimensional image group contains embedded regions that meet the above conditions. When the number of embedded regions in the region set corresponding to the i-th two-dimensional image group exceeds the preset number (or the total area of ​​the embedded regions exceeds the specified area), the location range formed by all embedded regions in the region set is determined as the location range where the user action is performed.

[0149] When the embedding area is small, the preset quantity is relatively large; when the embedding area is large, the preset quantity is relatively small. That is, the preset quantity can be adjusted adaptively according to the size of the embedding area.

[0150] By dividing the user action area into embedded regions, since the embedded regions carry location attributes, it is possible to quickly determine whether each position in the user action area is affected by the user action after determining that the comparison result of the embedded region is greater than the second preset difference threshold. Furthermore, by setting the number of embedded regions, motion interference caused by non-human or non-three-dimensional object actions can be eliminated, thereby improving recognition accuracy.

[0151] It should be noted that whether there is a difference in the embedded area can be determined by pre-set judgment criteria or by a pre-trained intelligent recognition model.

[0152] In a feasible implementation, when the user action area is divided into multiple embedded point areas, before executing step 102 to control the camera device to continuously capture images of the user action area, the method further includes the following steps:

[0153] Step 203: Control the projection device to project a reference pattern onto each embedded point area in the user action area.

[0154] Specifically, when the user action area is divided into multiple embedded point areas, the projection device can project a complete reference pattern into each embedded point area, or it can project a partial reference pattern of a reference pattern into each embedded point area. This ensures that each embedded point area has at least a partial reference pattern, thereby enabling the determination of whether the embedded point area has undergone stereo distortion based on the reference pattern or partial reference pattern projected by the projection device.

[0155] When performing step 201, comparing the differences between the image portion corresponding to the j-th embedding point region in the first action region two-dimensional image of the two-dimensional image group and each other action region two-dimensional image, the following steps are included:

[0156] Step 204: Compare the image portion of the j-th embedded point region of each other motion region 2D image in the 2D image group with the image portion of the j-th embedded point region of the first motion region 2D image to see if a new stereo distortion occurs; wherein, the stereo distortion includes the distortion produced when the reference pattern is occluded by a stereo object, and the 2D image obtained by taking a picture of the reference pattern is compared with the 2D image obtained by taking a picture of the reference pattern before it is occluded; if the image portion of the j-th embedded point region of each other motion region 2D image in the 2D image group with the image portion of the j-th embedded point region of the first motion region 2D image all exhibit stereo distortion, then it is determined that the comparison result between the first motion region 2D image and each other motion region 2D image in the 2D image group is greater than the second preset difference threshold.

[0157] Specifically, for each embedded region, it is determined whether the j-th embedded region of the first action region 2D image in the 2D image group produces a new stereo distortion compared to the j-th embedded region of the other action region 2D images. If the j-th embedded region of each other action region 2D image produces stereo distortion relative to the j-th embedded region of the first action region 2D image, then it is determined that the comparison result between the first action region 2D image and each other action region 2D image in the 2D image group is greater than the second preset difference threshold, and the j-th embedded region is assigned to the region set corresponding to the i-th 2D image group. Then, step 202 above is executed.

[0158] The comparison of the embedded areas and the determination of whether 3D distortion has occurred involved in step 204 are described in steps 1024 and 201 above, and will not be repeated here. When identifying whether 3D distortion has occurred in the embedded area through the intelligent recognition model, the intelligent recognition model can be trained in advance with a large number of training samples containing 3D distortion, thereby improving the accuracy of 3D distortion judgment.

[0159] In another feasible implementation, in step 103, the triggered interactive sub-region is determined based on the location range of the user action within the user action region, including the following two cases:

[0160] Case 1: One or more interactive sub-regions covered by the location range where the user action is applied in the user action area are taken as the triggered interactive sub-regions.

[0161] Specifically, when the location range uniquely covers an interactive sub-region within the user action area, the interactive sub-region covered by that location range is the triggered interactive sub-region. Figure 4 This specification illustrates a schematic diagram of determining a triggered interactive sub-region according to an embodiment of the present specification, such as... Figure 4 As shown, the location range determined by step 102 is 401. An interactive sub-region 402 is pre-set in the user action area. Therefore, in this schematic diagram, the location range 401 uniquely covers an interactive sub-region 402, and the interactive sub-region 402 is the triggered interactive sub-region.

[0162] If the location range covers multiple interactive sub-regions and these sub-regions do not share a common projection scheme, then the interactive sub-region closest to the midpoint of the location range is determined as the triggered interactive sub-region. Alternatively, if the location range covers multiple interactive sub-regions and these sub-regions share a common projection scheme, then all interactive sub-regions are collectively determined as the triggered interactive sub-region.

[0163] Alternatively, in scenario two, based on a preset mapping rule between location range and interactive sub-region, one or more interactive sub-regions mapped to the location range where the user action is performed in the user action area are determined as the triggered interactive sub-regions.

[0164] Specifically, when there is only one location range that encompasses multiple interactive sub-regions, or when there are multiple location ranges, each encompassing at least one interactive sub-region, if all multiple interactive sub-regions are designated as triggered interactive sub-regions, the number of triggered interactive sub-regions will be large, potentially leading to inaccurate interactions due to multiple projection schemes being fed back. In this case, to improve interaction accuracy, a mapping rule between the location range and the interactive sub-regions can be set. For example, the interactive sub-region triggered by the center point of the location range (or the center point of the area jointly formed by multiple location ranges) can be designated as the triggered interactive sub-region, and the projection scheme associated with each interactive sub-region can be adjusted according to the set mapping rule.

[0165] Even when the location range does not cover any interactive sub-region, it is still necessary to rely on the mapping relationship to determine the triggered interactive sub-region. In this case, the mapping relationship can be: determine the interactive sub-region closest to the center point of the location range in each interactive sub-region of the user action area as the triggered interactive sub-region.

[0166] Figure 5 This specification illustrates another schematic diagram of determining the triggered interactive sub-region, as provided in an embodiment. Figure 5 As shown, the location ranges determined by step 102 include: location range 501 and location range 502, and the interactive sub-regions pre-determined in the interactive action area include: interactive sub-region 503, interactive sub-region 504 and interactive sub-region 505.

[0167] Location range 501 covers interactive sub-region 503, location range 502 covers interactive sub-region 504, and the center point of location range 501 and location range 502 covers interactive sub-region 505. Therefore, interactive sub-region 505 is determined as the triggered interactive sub-region.

[0168] Figure 6 This specification illustrates an example of a regional deployment requirement for displaying a PowerPoint presentation (e.g., ...). Figure 3 In the area division method shown, in the scenario where a PPT is played in the display area, when the content played in the display area is the PPT presentation content, such as... Figure 6 The user action area shown contains two interactive sub-areas: interactive sub-area 601 and interactive sub-area 602. When interactive sub-area 601 is triggered, it flips the PPT slide forward; when interactive sub-area 602 is triggered, it flips the PPT slide backward. Other interactive sub-areas can also be deployed and configured with corresponding functions, such as automatic playback, pause, and end playback, according to the PPT presentation requirements.

[0169] Figure 7 Showing the target Figure 6The deployed interactive sub-regions are illustrated in the diagram indicating the triggered interactive sub-regions. For example... Figure 7 As shown, when the position range 701 is determined in the user action area according to step 102, since the center point of the position range 701 is closer to the interactive sub-area 601, the interactive sub-area 601 is the triggered interactive sub-area. At the same time, since the interactive sub-area 601 is set to flip the page forward, when the interactive sub-area 601 is triggered, the projection scheme associated with the interactive sub-area 601 at this moment displays the content of the previous page of the PPT. Therefore, according to the projection scheme, the previous page of the PPT is displayed in the display area.

[0170] In one feasible implementation, the vehicle further includes an audio playback device, and the projection scheme associated with the triggered interactive sub-area is further associated with an audio playback scheme; when performing step 104, which controls the projection device to project onto the display area based on the projection scheme associated with the triggered interactive sub-area, the following steps are included:

[0171] Based on the aforementioned audio playback scheme, the audio playback device is controlled to play audio.

[0172] Specifically, the audio playback solution is linked to the projection solution, so that when relevant images are projected into the display area, the audio playback device in the vehicle can switch between music, sound effects, and audio. This method enhances the user's interactive experience.

[0173] In one feasible implementation, the designated interactive application includes: an interactive game application; in executing step 104, controlling the projection device to project onto the display area based on the projection scheme associated with the triggered interactive sub-area includes the following steps:

[0174] Based on the game screen control scheme associated with the triggered interactive sub-region, the projection device is controlled to project the corresponding game screen onto the display area.

[0175] Specifically, when the designated interactive application is an interactive game application, the projection scheme associated with each interactive sub-area is the game screen control scheme, and the corresponding game screen is projected onto the display area according to the game screen control scheme.

[0176] In one feasible implementation, this application provides an interactive virtual device between a vehicle and a user. The virtual device is applied to the vehicle's central control unit, and the vehicle also includes a camera device and a projection device. The virtual device includes:

[0177] The first determining unit is used to determine the user action area and the display area in the external environment where the vehicle is located; wherein, the user action area includes a plurality of interactive sub-areas, and each interactive sub-area is associated with a projection scheme for the display area.

[0178] The second determining unit is used to control the camera device to continuously capture images of the user's action area, and to determine the position range of the user's action in the user's action area by comparing the two-dimensional images of the action area obtained by continuous capture.

[0179] The third determining unit is used to determine the triggered interactive sub-region based on the location range of the user action in the user action region.

[0180] The projection unit is used to control the projection device to project onto the display area based on the projection scheme associated with the triggered interactive sub-area.

[0181] In one feasible implementation, when the first determining unit determines the user's action area in the external environment where the vehicle is located, it is specifically used for:

[0182] Determine the user action area that meets safety requirements in the external environment where the vehicle is located.

[0183] The safety requirements include: the surface flatness is greater than a first preset degree, and / or the surface tilt relative to the plane is less than a second preset degree.

[0184] In one feasible implementation, the virtual device further includes:

[0185] The response unit is used to determine the area deployment requirements associated with the specified interactive application, the interactive sub-area deployment requirements, and the projection scheme associated with each interactive sub-area in response to the user's launch command for the specified interactive application before determining the user's action area and display area in the external environment where the vehicle is located.

[0186] The first determining unit, when determining the user action area and display area in the external environment where the vehicle is located, is specifically used for:

[0187] In the external environment where the vehicle is located, a user action area and a display area that meet the deployment requirements of the area are defined, and in the user action area, a number of interactive sub-areas that meet the deployment requirements of the interactive sub-areas are defined.

[0188] In one feasible implementation, when the second determining unit is used to control the camera device to continuously capture images of the user's action area, it is specifically used for:

[0189] The camera device is controlled to continuously capture images of the user's action area in a fixed posture.

[0190] Alternatively, the camera device can be controlled to continuously capture images of the user's action area in a non-fixed shooting posture, and the two-dimensional images of each action area obtained by continuous shooting can be calibrated to the coordinate reference system under the same shooting posture.

[0191] In one feasible implementation, the second determining unit, when determining the position range of the user action within the user action region by comparing two-dimensional images of the action region obtained through continuous shooting, is specifically used for:

[0192] Two-dimensional image groups are obtained by taking at least two adjacent two-dimensional images of action regions in the shooting sequence as a two-dimensional image group;

[0193] For the i-th two-dimensional image group, compare the difference between the first action region two-dimensional image in the two-dimensional image group and each other action region two-dimensional image; if the comparison result between the first action region two-dimensional image in the two-dimensional image group and each other action region two-dimensional image is greater than the first preset difference threshold, then the position range corresponding to the corresponding difference image part is taken as the position range of the user action; i = 1, 2, ..., N; N is the number of two-dimensional image groups.

[0194] In one feasible implementation, the virtual device further includes:

[0195] A first control unit is configured to control the projection device to project a reference pattern onto the user's action area before controlling the camera device to continuously capture images of the user's action area.

[0196] The second determining unit, when comparing the difference between the first motion region two-dimensional image of the two-dimensional image group and each of the other motion region two-dimensional images, is specifically used for:

[0197] Compare whether each other motion region two-dimensional image in the two-dimensional image group produces a new stereo distortion relative to the first motion region two-dimensional image; wherein, the stereo distortion includes the distortion produced when the reference pattern is occluded by a stereo object and the two-dimensional image obtained by taking the reference pattern is compared with the two-dimensional image obtained by taking the reference pattern before it is occluded.

[0198] If each of the other two-dimensional motion regions in the two-dimensional image group produces a new stereo distortion relative to the first two-dimensional motion region image, then it is determined that the comparison result between the first two-dimensional motion region image and each of the other two-dimensional motion regions in the two-dimensional image group is greater than the first preset difference threshold.

[0199] In one feasible implementation, the user action area is divided into multiple data point areas.

[0200] When the second determining unit determines the position range of the user's action within the user's action region by comparing two-dimensional images of the action region obtained through continuous shooting, it is specifically used for:

[0201] Two-dimensional image groups are obtained by taking at least two adjacent two-dimensional images of action regions in the shooting sequence as a two-dimensional image group.

[0202] For the i-th two-dimensional image group, compare the difference between the first action region two-dimensional image in the two-dimensional image group and the image portion corresponding to the j-th embedded point region in each other action region two-dimensional image; if the comparison result between the first action region two-dimensional image in the two-dimensional image group and the image portion corresponding to the j-th embedded point region in each other action region two-dimensional image is greater than the second preset difference threshold, then the j-th embedded point region is assigned to the region set corresponding to the i-th two-dimensional image group; i = 1, 2, ..., N; N is the number of two-dimensional image groups; j = 1, 2, ..., M; M is the number of embedded point regions.

[0203] If the number of embedded regions in the region set corresponding to the i-th two-dimensional image group is greater than the preset number, then the location range formed by all embedded regions in the region set corresponding to the i-th two-dimensional image group is taken as the location range of a user action.

[0204] In one feasible implementation, the virtual device further includes:

[0205] The second control unit is used to control the projection device to project a reference pattern onto each embedded point area in the user's action area before controlling the camera device to continuously capture images of the user's action area.

[0206] The differences between the first action region 2D image of this 2D image group and the image portion corresponding to the j-th embedding point region in each of the other action region 2D images include:

[0207] Compare the j-th embedded point region of each other action region 2D image in the 2D image group with the image portion of the j-th embedded point region of the first action region 2D image to see if a new stereo distortion occurs; wherein, the stereo distortion includes the distortion produced when the reference pattern is occluded by a stereo object, and the 2D image obtained by taking a picture of the reference pattern is compared with the 2D image obtained by taking a picture of the reference pattern before it is occluded.

[0208] If the j-th embedded point region of each other motion region 2D image in the 2D image group produces stereo distortion relative to the j-th embedded point region of the first motion region 2D image, then it is determined that the comparison result between the first motion region 2D image and each other motion region 2D image in the 2D image group is greater than the second preset difference threshold.

[0209] In one feasible implementation, when the third determining unit determines the triggered interactive sub-region based on the location range of the user action within the user action region, it is specifically used for:

[0210] One or more interactive sub-regions covered by the location range where the user action is applied within the user action area are designated as the triggered interactive sub-regions.

[0211] Alternatively, based on a preset mapping rule between location range and interactive sub-region, one or more interactive sub-regions mapped to the location range where the user action is performed in the user action area are determined as the triggered interactive sub-regions.

[0212] In one feasible implementation, the vehicle also has an audio playback device, and the projection scheme associated with the triggered interactive sub-area is further associated with an audio playback scheme.

[0213] The virtual device also includes:

[0214] An audio playback unit is configured to control the audio playback device to play audio based on the audio playback scheme when the projection device projects onto the display area based on the projection scheme associated with the triggered interactive sub-area.

[0215] In one feasible implementation, the specified interactive application includes: an interactive game application.

[0216] When the projection unit controls the projection device to project onto the display area based on the projection scheme associated with the triggered interactive sub-region, it is specifically used for:

[0217] Based on the game screen control scheme associated with the triggered interactive sub-region, the projection device is controlled to project the corresponding game screen onto the display area.

[0218] The embodiments of the interactive virtual device described in this specification can be applied to computer devices, such as servers or terminal devices. The device embodiments can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor reading the corresponding computer program instructions from non-volatile memory into memory and executing them. When the processor executes the computer program instructions, it implements the steps of the interaction method between the vehicle and the user as described in any of the above embodiments. From a hardware perspective, Figure 8 This specification illustrates a hardware structure diagram of a computer device housing the interactive virtual device provided in an embodiment, such as... Figure 8 As shown, except Figure 8In addition to the processor 810, memory 830, network interface 820, and non-volatile memory 840 shown, the server or electronic device where the device 831 (i.e., the interactive virtual device) is located in the embodiment may also include other hardware depending on the actual function of the computer device, which will not be described in detail here.

[0219] Accordingly, this specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the interaction method between the vehicle and the user as described in any of the above embodiments.

[0220] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0221] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0222] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0223] Other embodiments of this specification will readily occur to those skilled in the art upon consideration of the specification and practice of the invention claimed herein. This specification is intended to cover any variations, uses, or adaptations that follow the general principles of this specification and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this specification are indicated by the following claims.

[0224] It should be understood that this specification is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this specification is limited only by the appended claims.

[0225] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A method of interaction between a vehicle and a user, the method comprising: A central control device applied to a carrier, the carrier also having a camera device and a projection device; the method comprises: determining a user action area and a display area in an external environment where the carrier is located; wherein the user action area comprises a plurality of interaction sub-areas, each interaction sub-area being associated with a projection scheme for the display area; controlling the camera device to continuously capture the user action area, and determining a position range of a user action in the user action area by comparing two-dimensional images of the user action area obtained by continuous capturing; determining a triggered interaction sub-area according to the position range of the user action in the user action area; controlling the projection device to project onto the display area based on the projection scheme associated with the triggered interaction sub-area.

2. The method of claim 1, wherein, Determining a user action area in an external environment where the carrier is located comprises: Determining a user action area that meets safety requirements in an external environment where the carrier is located; wherein the safety requirements include: a surface flatness greater than a first preset degree, and / or a surface inclination relative to a plane less than a second preset degree.

3. The method of claim 1, wherein, Before determining a user action area and a display area in an external environment where the carrier is located, the method comprises: in response to a user's start instruction for a specified interaction class application, determining the area deployment requirements associated with the specified interaction class application, the interaction sub-area deployment requirements, and the projection scheme associated with each interaction sub-area; Determining a user action area and a display area in an external environment where the carrier is located comprises: demarcating a user action area and a display area that meet the area deployment requirements in an external environment where the carrier is located, and demarcating a plurality of interaction sub-areas in the user action area that meet the interaction sub-area deployment requirements.

4. The method of claim 1, wherein, Controlling the camera device to continuously capture the user action area comprises: controlling the camera device to continuously capture the user action area in a fixed posture; or controlling the camera device to continuously capture the user action area in a non-fixed posture, and calibrating each two-dimensional image of the user action area obtained by continuous capturing to a coordinate reference system in the same posture.

5. The method of claim 1, wherein, Determining a position range of a user action in the user action area by comparing two-dimensional images of the user action area obtained by continuous capturing comprises: obtaining a two-dimensional image group sequence by taking at least two two-dimensional images of the user action area adjacent in the shooting sequence as a two-dimensional image group; for the i-th two-dimensional image group, comparing the first two-dimensional image of the user action area with each other two-dimensional image of the user action area; if the comparison result of the first two-dimensional image of the user action area and each other two-dimensional image of the user action area in the two-dimensional image group is greater than a first preset difference threshold, then the position range corresponding to the difference image part is taken as a position range of a user action; i=1, 2, …, N; N is the number of two-dimensional image groups.

6. The method of claim 5, wherein, Before controlling the camera device to continuously capture the user action area, the method further comprises: controlling the projection device to project a reference pattern onto the user action area; comparing the first action region two-dimensional image of the two-dimensional image group with each other action region two-dimensional image, including: comparing whether each other action region two-dimensional image of the two-dimensional image group produces new stereoscopic distortion relative to the first action region two-dimensional image; wherein the stereoscopic distortion includes the distortion produced by the two-dimensional image obtained by shooting the reference pattern after the reference pattern is blocked by the stereoscopic object relative to the two-dimensional image obtained by shooting the reference pattern before being blocked; if each other action region two-dimensional image of the two-dimensional image group produces new stereoscopic distortion relative to the first action region two-dimensional image, it is determined that the comparison result of the first action region two-dimensional image and each other action region two-dimensional image in the two-dimensional image group is greater than the first preset difference threshold.

7. The method of claim 1, wherein, wherein, the user action region is divided into a plurality of buried point regions; determining the position range of the user action in the user action region by comparing the action region two-dimensional images obtained by continuous shooting, including: taking at least two action region two-dimensional images adjacent in shooting order as a two-dimensional image group to obtain a two-dimensional image group sequence; for the i-th two-dimensional image group, comparing the difference between the first action region two-dimensional image of the two-dimensional image group and the image part corresponding to the j-th buried point region in each other action region two-dimensional image; if the comparison result of the first action region two-dimensional image and each other action region two-dimensional image corresponding to the j-th buried point region in the two-dimensional image group is greater than the second preset difference threshold, the j-th buried point region is included in the region set corresponding to the i-th two-dimensional image group; i=1, 2, …, N; N is the number of two-dimensional image groups; j=1, 2, …, M; M is the number of buried point regions; if the number of buried point regions in the region set corresponding to the i-th two-dimensional image group is greater than the preset number, the position range formed by all buried point regions in the region set corresponding to the i-th two-dimensional image group is taken as the position range of a user action.

8. The method of claim 7, wherein, Before controlling the camera to continuously shoot the user action region, the method further includes: controlling the projection device to project the reference pattern to each buried point region in the user action region; comparing the first action region two-dimensional image of the two-dimensional image group with each other action region two-dimensional image corresponding to the j-th buried point region, including: comparing whether the j-th buried point region of each other action region two-dimensional image of the two-dimensional image group produces new stereoscopic distortion relative to the j-th buried point region of the first action region two-dimensional image; wherein the stereoscopic distortion includes the distortion produced by the two-dimensional image obtained by shooting the reference pattern after the reference pattern is blocked by the stereoscopic object relative to the two-dimensional image obtained by shooting the reference pattern before being blocked; if the j-th buried point region of each other action region two-dimensional image of the two-dimensional image group produces stereoscopic distortion relative to the j-th buried point region of the first action region two-dimensional image, it is determined that the comparison result of the first action region two-dimensional image and each other action region two-dimensional image in the two-dimensional image group is greater than the second preset difference threshold.

9. The method of claim 1, wherein, determine the triggered interaction sub-area according to a position range of the user action in the user action area, including: determine one or more interaction sub-areas covered by the position range of the user action in the user action area as the triggered interaction sub-area; or determine one or more interaction sub-areas mapped by the position range of the user action in the user action area as the triggered interaction sub-area based on a preset mapping rule between the position range and the interaction sub-area.

10. The method of claim 1, wherein, Wherein, the vehicle further has an audio playing device, and the projection scheme associated with the triggered interaction sub-area further has an audio playing scheme; when the projection device is controlled to project onto the display area based on the projection scheme associated with the triggered interaction sub-area, the method further includes: control the audio playing device to play audio based on the audio playing scheme.

11. The method of claim 3, wherein, Wherein, the specified interaction type application includes an interaction type game application; controlling the projection device to project onto the display area based on the projection scheme associated with the triggered interaction sub-area includes: controlling the projection device to project a corresponding game screen onto the display area based on a game screen control scheme associated with the triggered interaction sub-area.

12. An interactive virtual device between a vehicle and a user, comprising: the virtual device is applied to a central control device of a vehicle, and the vehicle further has a camera device and a projection device; the virtual device includes: a first determination unit configured to determine a user action area and a display area in an external environment of the vehicle; the user action area includes a plurality of interaction sub-areas, and each interaction sub-area is associated with a projection scheme for the display area; a second determination unit configured to control the camera device to continuously capture the user action area, and determine a position range of a user action in the user action area by comparing two-dimensional images of the user action area obtained by continuous capturing; a third determination unit configured to determine a triggered interaction sub-area according to the position range of the user action in the user action area; a projection unit configured to control the projection device to project onto the display area based on the projection scheme associated with the triggered interaction sub-area.

13. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method of any one of claims 1-11.

14. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the steps of the method of any one of claims 1-11.

Images