In-vehicle image display method, storage medium and program product

Abstract

One or more embodiments of the invention provide an in-vehicle image display method, a storage medium and a program product, the method is applied to AR glasses, and the method comprises the following steps: obtaining an in-vehicle space model of a target vehicle and relative position information of a user and the target vehicle; under the condition of determining that the target vehicle is in the visual field visible area of the user, determining an in-vehicle image presented by the in-vehicle space model at the visual angle of the user according to the relative position information; and overlapping and displaying the in-vehicle image at a corresponding position of the target vehicle in the visual field visible area.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of augmented reality (AR) technology, and more particularly to a method for displaying in-vehicle images, a storage medium, and a program product. Background Art

[0002] When driving a vehicle, users often need to leave the vehicle temporarily for a period of time, such as parking to shop or do some errands. At this time, users may lose real-time control of the situation inside the vehicle, making it difficult to discover and deal with potential problems in a timely manner, increasing the risk of accidents inside the vehicle. Therefore, finding a convenient way to understand the situation inside the vehicle has become an important requirement for improving user experience and safety.

[0003] In the related technology, mobile devices such as mobile phones are usually relied on to monitor the situation inside the car, but this method is not only cumbersome to operate, but also particularly inconvenient when the vehicle is still within the user's field of vision. Users often need to frequently unlock their phones, open applications, and check the images from the camera inside the car. This series of operations is not only time-consuming, but may also distract users and affect their observation of the surrounding environment. Summary of the invention

[0004] In view of this, one or more embodiments of this specification provide the following technical solutions:

[0005] According to a first aspect of one or more embodiments of this specification, a method for displaying an in-vehicle image is proposed, which is applied to AR glasses; the method includes:

[0006] Acquire the interior space model of the target vehicle and the relative position information between the user and the target vehicle;

[0007] In the case where it is determined that the target vehicle is in the visible area of ​​the user's field of vision, determining an in-vehicle image presented by the in-vehicle space model from the user's perspective according to the relative position information;

[0008] The in-vehicle image is superimposed and displayed on the corresponding position of the target vehicle in the visual area of ​​the field of view.

[0009] Optionally, determining the in-vehicle image presented by the in-vehicle space model from the user's perspective according to the relative position information includes:

[0010] Determining the model size and angle of the in-vehicle space model relative to the user according to the relative position information;

[0011] The in-vehicle space model is projected onto a two-dimensional plane to obtain the corresponding in-vehicle image.

[0012] Optionally, the target vehicle corresponds to at least one in-vehicle object; the method further includes:

[0013] In the case where the at least one in-vehicle object is blocked by the target vehicle, a transparent gradient process is performed on the blocked portion of the in-vehicle image corresponding to the target vehicle, and an image enhancement process is performed on the in-vehicle object in the in-vehicle image.

[0014] Optionally, the target vehicle corresponds to at least one in-vehicle object; the method further includes:

[0015] determining whether the at least one in-vehicle object is in an abnormal state according to the in-vehicle space model;

[0016] If the judgment result is yes, the corresponding risk warning information is sent to the user.

[0017] Optionally, the AR glasses are equipped with an environmental sensor and a posture sensor; and determining that the target vehicle is in a visible area of ​​the user's field of view includes:

[0018] Acquire surrounding environmental information according to the environmental sensor, wherein the environmental information is used to characterize surrounding objects and their positions;

[0019] Determine the user's line of sight direction according to the posture sensor, and determine the visual area of ​​the field of view according to the line of sight direction and the environmental information;

[0020] The target vehicle is determined to be in the visible area of ​​the field of view according to the objects and their positions represented by the environmental information.

[0021] Optionally, the method further includes:

[0022] Get the monitoring picture of the target vehicle;

[0023] When it is determined that the target vehicle is not in the visible area of ​​the user's field of view, or the user enters the target vehicle, the in-vehicle space model and / or the monitoring screen is displayed.

[0024] Optionally, obtaining the interior space model of the target vehicle includes:

[0025] Initiate a model acquisition request to the target vehicle to obtain the interior space model returned by the target vehicle; or

[0026] When the target vehicle meets the model pushing condition, the vehicle interior space model pushed by the target vehicle is received.

[0027] Optionally, the relative position information is generated by the target vehicle or the AR glasses according to a corresponding sensor.

[0028] According to a second aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0029] According to a third aspect of one or more embodiments of the present specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the method described in the first aspect when executed by a processor.

[0030] In the technical solution provided in this specification, the interior space model of the target vehicle and the relative position information between the user and the target vehicle can be obtained through AR glasses. When the target vehicle appears in the visible area of ​​the user's field of view, the interior image of the above-mentioned interior space model presented from the user's perspective can be determined according to the relative position information, and displayed to the user. Among them, through the device of AR glasses, the user only needs to simply look at the target vehicle to see the details inside the car. The whole process is simple and convenient, and no other additional operations are required. In addition, the displayed interior image is superimposed on the corresponding position of the target vehicle in the visible area of ​​the field of view, so that the user can intuitively see the interior image while observing the target vehicle, which enhances the visualization effect of information and reduces the occurrence of unexpected situations in the car. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 It is a schematic diagram of the architecture of a system for displaying in-vehicle images provided by an exemplary embodiment;

[0033] Figure 2 is a flowchart of a method for displaying an in-vehicle image provided by an exemplary embodiment;

[0034] Figure 3 is a flowchart of another method for displaying an in-vehicle image provided by an exemplary embodiment;

[0035] Figure 4 is a schematic structural diagram of an electronic device provided by an exemplary embodiment;

[0036] Figure 5 It is a structural schematic diagram of a device for displaying in-vehicle images provided by an exemplary embodiment. DETAILED DESCRIPTION

[0037] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are only examples of devices and methods consistent with some aspects of the present invention.

[0038] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0039] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0040] Figure 1 FIG. 1 is a schematic diagram of the architecture of a system for displaying objects in a vehicle provided by an exemplary embodiment. Figure 1 As shown, the system includes AR glasses 11 and a target vehicle 12.

[0041] As a wearable device, AR glasses 11 can combine virtual content with the real world by superimposing digital information in the user's field of view. This device can be interconnected with the target vehicle 12 and transmit data. During the operation of the system, AR glasses 11 can obtain the interior space model of the target vehicle and the relative position information between the user wearing the AR glasses 11 and the target vehicle, and determine the interior image when the target vehicle is in the visible area of ​​the user's field of view. Finally, the interior image is superimposed and displayed on the corresponding position of the target vehicle through the display screen, so that the user can immediately understand the situation inside the car.

[0042] The target vehicle 12 is equipped with a corresponding on-board monitoring system, which can monitor the situation inside the vehicle through, for example, a high-resolution camera and a sensor. During the operation of the system, the target vehicle 12 can obtain the interior space model based on the on-board monitoring system and transmit it to the paired AR glasses 11, so that the AR glasses 11 can determine and display the corresponding interior image.

[0043] It should be noted that, under the premise that both the AR glasses 11 and the target vehicle 12 are equipped with wireless communication modules, long-distance data transmission between each other can be achieved based on technologies such as Bluetooth, wireless fidelity (Wi-Fi) and other wireless local area networks or 5G networks, and this is not limited in this manual.

[0044] Figure 2 The present invention is a flowchart of a method for displaying an in-vehicle image provided by an exemplary embodiment, and the method is applied to AR glasses; the method may include the following steps:

[0045] Step S202, obtaining the interior space model of the target vehicle and the relative position information between the user and the target vehicle.

[0046] When the user wants to observe the interior image of the target vehicle behind the AR glasses, the user first needs to use the AR glasses to obtain the interior space model of the target vehicle and the relative position information between the user and the target vehicle. Among them, the above-mentioned interior space model can be created by the above-mentioned target vehicle based on its own sensor. At the same time, the way the target vehicle creates the interior space model is affected by the sensor used by the target vehicle, and this specification does not limit this. For example, the target vehicle is equipped with a depth camera and multiple two-dimensional cameras for the interior space. First, feature point matching (Feature Matching) technology such as SIFT (Scale-Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF) can be used. Images from different cameras are aligned and fused into a complete panoramic image. Then, the depth map can be determined based on the depth camera or the disparity map of multiple two-dimensional cameras based on multi-eye stereo vision (Stereo Vision) technology, and the RGB image (color image) of the corresponding camera and the above-mentioned depth map are combined to generate three-dimensional point cloud data (Point Cloud), and the interior space model is created based on the three-dimensional point cloud data. For another example, when the target vehicle is equipped with a laser radar for the interior space, the target vehicle can directly generate three-dimensional point cloud data (PointCloud) based on the laser radar, and create an interior space model based on the three-dimensional point cloud data. The above relative position information may include helping to determine the spatial relationship between the user and the target vehicle. Specifically, it may include but is not limited to the distance, azimuth, pitch angle and coordinates of each other in the same spatial coordinate system. Considering the relativity of the above relative position information, this specification can be configured according to actual needs by the sensor in the target vehicle or AR glasses to generate relative position information. In particular, if the relative position information is generated by the target vehicle, it can be transmitted to the AR glasses together with the above interior space model, thereby avoiding the waste of resources caused by multiple data transmissions; or the interior space model and the relative position information are transmitted to the AR glasses in sequence according to the order of the generation time points, so as to ensure that the AR glasses receive and analyze the corresponding data in advance and improve the overall efficiency of determining the interior image. It is worth mentioning that the premise for obtaining the above interior space model and relative position information can be designed to be that the vehicle is in a stationary state to ensure the safety of the user and the vehicle.

[0047] In the process of the AR glasses acquiring the interior space model of the vehicle, the acquisition strategy of the interior space model can also be adjusted according to actual conditions.

[0048] In one embodiment, the AR glasses may initiate a model acquisition request to the target vehicle, requesting to obtain the above-mentioned in-vehicle space model. However, after receiving the request, the vehicle may perform device identity verification on the AR glasses to ensure that only authorized devices can access vehicle data. Once the verification is successful, the model acquisition request may be returned, otherwise it may be refused to be provided. In this embodiment, the method in which the AR glasses actively initiate a model acquisition request is suitable for scenarios that require real-time or on-demand access to the interior of the vehicle, such as when the owner wants to quickly check whether there are any items left in the car, or monitor the status of children in the car, and this method can save more device resources. In addition, the so-called device identity verification may be verification of the unique device identifier of the AR glasses, or verification of the above-mentioned user identity information when the above-mentioned model acquisition request carries user identity information, which is not limited in this specification.

[0049] In another embodiment, the AR glasses can receive the interior space model pushed by the target vehicle when the target vehicle meets the model push conditions, wherein specific events occurring inside or outside the vehicle can trigger the model push mechanism for the interior space model, such as unlocked doors, detection of moving objects inside the vehicle, etc. When the triggering conditions are met, the target vehicle can automatically send relevant information to the designated AR glasses device without the need for a request from the device. At the same time, the target vehicle can also continuously send the interior space model to the device within a preset sending time to ensure that the user can obtain real-time images inside the vehicle. In this embodiment, the way in which the AR glasses passively receive model acquisition requests is suitable for situations where the interior environment needs to be continuously monitored, or where the owner needs to be notified immediately under specific conditions.

[0050] Step S204, when it is determined that the target vehicle is in the visible area of ​​the user's field of vision, determine the in-vehicle image presented by the in-vehicle space model from the user's perspective according to the relative position information.

[0051] After obtaining the above-mentioned in-vehicle space model and the above-mentioned relative position information, the AR glasses can determine whether the above-mentioned target vehicle is in the user's visible field of view. If so, the in-vehicle image presented by the above-mentioned in-vehicle space model from the user's perspective can be determined based on the above-mentioned relative position information for display in subsequent steps; if not, the content may not be displayed or alternative content may be displayed.

[0052] For the above-mentioned alternative contents, this specification may obtain them through sensors based on the in-vehicle space model or the target vehicle.

[0053] In one embodiment, the AR glasses can obtain at least one monitoring screen obtained by the target vehicle through the vehicle monitoring system. When it is determined that the target vehicle is not in the visible area of ​​the user's field of view, the vehicle interior space model and / or the monitoring screen can be displayed, wherein the vehicle interior space model and / or the monitoring screen can be displayed in the form of a floating panel at any position in the display screen of the AR glasses, and the size, shape, transparency and other parameters can be adjusted according to the user's map operation, thereby providing portable monitoring capabilities for the target vehicle when the user is not paying attention to the target vehicle. The monitoring screen can be an image or video arranged based on time, which is not limited in this specification.

[0054] It should be noted that the premise for determining whether the target vehicle is in the user's visual field is that the user has left the target vehicle. In other words, once the user is in the vehicle, the relative position information between the user and the target vehicle can be regarded as overlapping positions in a broad sense, and the above-mentioned in-vehicle image cannot be determined. Therefore, the method in one embodiment can be applied to the case where the user enters the target vehicle, that is, when the user enters the target vehicle, the above-mentioned in-vehicle space model and / or the above-mentioned monitoring screen can be displayed. Of course, whether the user enters the target vehicle can be determined based on the scene recognition algorithm of the AR glasses or the passenger perception sensor of the target vehicle, and this is not limited in this specification.

[0055] In this manual, the sensors in the AR glasses can be used to accurately determine whether the above-mentioned target vehicle is within the user's visible field of view.

[0056] In one embodiment, the AR glasses can be configured with environmental sensors and posture sensors. In this case, the AR glasses can obtain surrounding environmental information based on the environmental sensors, and the environmental information is used to characterize surrounding objects and their positions. Then, the user's line of sight direction is determined based on the posture sensor, and the visual area of ​​the field of view is determined based on the line of sight direction and the environmental information. Finally, the target vehicle is determined to be in the visual area of ​​the field of view based on the objects and their positions represented by the environmental information.

[0057] On the one hand, for the above-mentioned environmental information, AR glasses can determine it through the corresponding target environmental sensor, which specifically refers to a device that collects environmental spatial features to build an environmental model. The data directly collected from the surrounding environment can be called raw information. Since raw information is usually difficult to intuitively describe the environmental spatial features, it needs to be further processed to obtain the corresponding environmental information. The environmental information can be map information of the user's surrounding environment generated based on Simultaneous Localization And Mapping (SLAM) technology, or it can be object detection and recognition information, distance and obstacle information, three-dimensional modeling information, environmental classification information, and dynamic object information, etc., which can be used to describe the environmental spatial features.

[0058] On the other hand, for the above-mentioned posture sensor, AR glasses can use the device to determine the user's line of sight direction, and determine the above-mentioned visible area of ​​the field of view based on the above-mentioned line of sight direction and the above-mentioned environmental information. Among them, the user's current viewing angle can be determined according to the posture information provided by the posture sensor, and then the user's line of sight direction can be determined, and then based on the position of the object represented by the environmental information and the user's line of sight direction, through three-dimensional modeling, computer vision and other technologies, analyze which objects may block the user's line of sight, and finally calculate the visible area of ​​the field of view in the direction of the user's line of sight. Of course, in the case where the AR glasses are not equipped with a posture sensor, it can also be replaced by other target environmental sensors at the cost of reducing a certain degree of accuracy, such as the deployment direction of the image sensor to determine the user's current viewing angle.

[0059] Once the environmental information and the visual area are acquired, it is possible to try to determine whether the target vehicle is matched among the objects in the visual area in the environmental information, thereby determining whether the target vehicle is in the visual area.

[0060] The above-mentioned in-car image can also be adjusted to a state that conforms to the observation effect from the user's perspective before being officially displayed, so as to avoid the user feeling uncomfortable with the in-car image.

[0061] In one embodiment, the AR glasses can determine the model size and angle of the in-car space model relative to the user based on the relative position information, and project the in-car space model onto a two-dimensional plane to obtain the corresponding in-car image. Specifically, when the relative position information includes the distance, azimuth, and pitch angle between the user and the target vehicle, the size of the in-car space model can be adjusted according to the distance information between the user and the target vehicle, and the overall principle is that the farther the distance, the smaller the model, and the closer the distance, the larger the model. The specific scaling ratio can be calculated in combination with a preset scaling factor; the angle of the in-car space model can be divided into two dimensions: horizontal plane and vertical plane. According to the azimuth information between the user and the target vehicle, the rotation angle of the in-car space model on the horizontal plane can be adjusted, and according to the pitch angle information between the user and the target vehicle, the inclination angle of the in-car space model on the vertical plane can be adjusted. The adjusted in-car space model can be projected onto a two-dimensional plane based on perspective projection. The so-called perspective projection simulates the visual effect of the human eye, making distant objects look smaller. A perspective projection matrix can be constructed to convert the three-dimensional coordinates of the in-car space model into two-dimensional coordinates, thereby obtaining a two-dimensional in-car image. In addition, rendering operations can be performed on the interior image, such as adding appropriate lighting and texture effects, to make the interior image look more realistic.

[0062] Step S206: Overlay and display the in-vehicle image on the corresponding position of the target vehicle in the visual area of ​​the field of view.

[0063] After the in-car image is determined, the in-car image can be displayed to the user through the display screen of the AR glasses. The display position of the in-car image can be determined as the corresponding position of the target vehicle in the visual field. The corresponding position can be the center point of the target vehicle, that is, the center point of the in-car image coincides with the center point of the target vehicle. Figure 1 For example, the interior image in the user's eyes can just cover the interior area of ​​the target vehicle 12, wherein the passenger whose body was originally partially blocked by the door can have his entire body displayed again in the user's AR glasses through the interior image.

[0064] The target vehicle may also correspond to at least one in-vehicle object, such as a child, a pet, a briefcase, etc. The in-vehicle object may be actively defined or modified by the user, and the main purpose of the user viewing the in-vehicle image may be to observe the state of the in-vehicle object. Then, the specification may further highlight the in-vehicle object, so that the user can more intuitively and conveniently determine the state of the in-vehicle object in the target vehicle.

[0065] In one embodiment, when at least one of the above-mentioned in-vehicle objects is blocked by the above-mentioned target vehicle, the blocked portion of the in-vehicle image corresponding to the above-mentioned target vehicle can be subjected to transparent gradient processing, and the in-vehicle object in the in-vehicle image can be subjected to image enhancement processing. The purpose of the above-mentioned transparent gradient processing is to allow the user to clearly understand that the processed portion is the blocked portion of the in-vehicle object, thereby improving the authenticity of the in-vehicle object in the user's senses; the above-mentioned image enhancement processing includes but is not limited to stroking and thickening the in-vehicle object in the in-vehicle image, changing the color, increasing the contrast, adding icons, arrows, adding special effects such as emitting dim light, etc., and the AR glasses can be adjusted according to the pre-configured or real-time voice or gesture behavior of the user.

[0066] When at least one of the above-mentioned in-car objects is in an abnormal state, the AR glasses can identify the above-mentioned abnormal state through the action behavior of the in-car objects in the in-car space model, thereby sending corresponding risk warning information to the user. Among them, the above-mentioned abnormal state can be identified based on abnormal detection algorithms such as posture detection and behavior analysis determined by the machine learning model, and the abnormal detection algorithm can further determine different thresholds and rules for different in-car objects to determine under what circumstances an alarm needs to be triggered. Still taking children and pets as an example, if the child's movement amplitude is detected to exceed the preset range or frequency, such as frequent standing and jumping, then the AR glasses can issue a risk warning message, or when the pet is monitored to be motionless for a long time or struggling, commotion and other behaviors, the system will conduct further analysis to determine whether there is a possibility of injury. Of course, the above-mentioned abnormal state recognition process can also be performed by the target vehicle, and at this time the AR glasses will only be responsible for sending the above-mentioned risk warning information.

[0067] The above risk warning information can remind the user through at least one of the following methods: sound, vibration, or displaying a visual warning signal on the display screen, so as to increase the user's alertness and prompt them to take timely action. For example, when the AR glasses recognize that the child in the target vehicle continues to jump more than 3 times within 10 seconds, the AR glasses can play a clear warning sound through a speaker or other associated playback device, generate vibrations to the body through a vibration motor or an associated portable device such as a smart bracelet, or display an obvious warning icon on the corresponding display screen until the user actively confirms the risk warning information or directly stops the child's current behavior.

[0068] It is understandable that the above-mentioned risk warning information may not cause actual risks in certain scenarios due to misjudgment and other reasons, resulting in users still choosing to ignore the risk warning information after discovering it. AR glasses can record the triggering of the above-mentioned risk warning information and perform learning optimization based on the deep model, ultimately achieving the goal of optimizing the user's experience of using AR glasses while ensuring user safety. For example, if the user ignores the risk warning information of a certain risk object many times, the sending frequency or sensitivity of the risk warning information can be reduced.

[0069] Combine the following Figure 3 The overall process of the method for displaying the in-vehicle image is introduced. As shown in the figure, the method includes the following steps:

[0070] Step S302: pairing the AR glasses with the on-board monitoring system of the target vehicle.

[0071] In one embodiment, the user can select to establish a Bluetooth connection with the vehicle monitoring system of the target vehicle in the operation interface of the AR glasses. For example, the user can find the "Device" or "Settings" option on the operation interface of the AR glasses, select "Add New Device", then search for nearby vehicles and complete the pairing process according to the prompts.

[0072] Step S304, determining whether the target vehicle is in a stationary state.

[0073] In one embodiment, the onboard monitoring system of the target vehicle can check whether the vehicle is stationary, which is to ensure safety and accuracy when acquiring the in-vehicle image. The stationary state of the vehicle can be determined by the speed information detected by the onboard sensor. If the vehicle is in motion, the next step will not be continued, because the dynamic in-vehicle environment may cause unstable or inaccurate images.

[0074] Step S306, obtaining the monitoring image of the target vehicle, the interior space model of the vehicle, and the relative position information between the user and the target vehicle.

[0075] In one embodiment, once the vehicle is stationary, the vehicle monitoring system can start collecting information, including monitoring images obtained through the camera and the in-vehicle space model created by the laser radar. At the same time, the system also needs to collect the relative position information between the user and the vehicle, which can be obtained through the global positioning system (GPS) module built into the AR glasses.

[0076] Step S308, determining whether the target vehicle is within the user's visual field.

[0077] In one embodiment, the vehicle monitoring system described above uses the environment sensor and the posture sensor in the AR glasses to determine whether the target vehicle is within the user's field of view. If not, jump to step S314; if within the field of view, continue to execute step S310.

[0078] Step S310: determining the in-vehicle image presented by the in-vehicle space model from the user's perspective according to the relative position information.

[0079] In one embodiment, if the vehicle is within the user's field of view, the vehicle monitoring system will adjust the size and angle of the in-vehicle space model according to the relative position information between the user and the vehicle to adapt to the user's perspective. At this time, it is assumed that the user is standing about 5 meters away from the vehicle and the user is facing the direction of the vehicle. The relative position information at this time includes the following: 5 meters between the user and the vehicle, azimuth: 0 degrees (directly in front), pitch angle: 0 degrees (horizontal), since the distance between the user and the vehicle is 5 meters, according to the preset scale factor, it can be set that the model size is reduced by 10% for every increase of 1 meter. Therefore, the model size adjustment factor is (1-0.1)×5=0.5(1-0.1)×5=0.5, that is, the model size is 50% of the original size, and since the rotation angle on the horizontal plane is 0 degrees and the tilt angle on the vertical plane is also 0 degrees, there is no need to rotate or tilt. After the three-dimensional model is projected onto a two-dimensional plane using a perspective projection matrix, appropriate lighting and texture effects are added to the obtained in-vehicle image to enhance the realism of the image.

[0080] Step S312, superimposing the in-vehicle image on the target vehicle in the visible area of ​​the field of view based on the center point of the image.

[0081] In one embodiment, the center point of the vehicle can be obtained through the above relative position information. Assuming that the coordinates of the center point of the target vehicle on the display screen of the AR glasses are (320, 240), the adjusted in-vehicle image can be placed at the above coordinates so that the center points of the two overlap to ensure that the image just covers the actual position of the vehicle.

[0082] Step S314, display the monitoring screen.

[0083] In one embodiment, after determining the in-vehicle image, the vehicle-mounted monitoring system may also display a real-time monitoring image acquired from the target vehicle, which may help users who are far away from the vehicle to better understand the situation in the vehicle.

[0084] Figure 4 is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 4At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410, and may also include hardware required for other functions. One or more embodiments of this specification may be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0085] Corresponding to the embodiment of the method for displaying the in-vehicle image described above, this specification also provides an embodiment of a device for displaying the in-vehicle image. Figure 5 The in-vehicle image display device is applied to AR glasses; the device comprises:

[0086] A model and information acquisition unit 502 is used to acquire an interior space model of a target vehicle and relative position information between the user and the target vehicle;

[0087] An in-vehicle image determination unit 504 is used to determine, when it is determined that the target vehicle is in the visible area of ​​the user's field of vision, an in-vehicle image presented by the in-vehicle space model from the user's perspective according to the relative position information;

[0088] The in-vehicle image display unit 506 is used to overlay and display the in-vehicle image on the corresponding position of the target vehicle in the visual area of ​​the field of view.

[0089] Optionally, the in-vehicle image determining unit 504 is specifically configured to:

[0090] Determining the model size and angle of the in-vehicle space model relative to the user according to the relative position information;

[0091] The in-vehicle space model is projected onto a two-dimensional plane to obtain the corresponding in-vehicle image.

[0092] Optionally, the target vehicle corresponds to at least one in-vehicle object; and the device further includes:

[0093] The in-vehicle image processing unit is used to perform transparent gradient processing on the blocked part of the in-vehicle image corresponding to the target vehicle when at least one in-vehicle object is blocked by the target vehicle, and to perform image enhancement processing on the in-vehicle object in the in-vehicle image.

[0094] Optionally, the target vehicle corresponds to at least one in-vehicle object; and the device further includes:

[0095] a risk warning information sending unit, configured to determine whether the at least one in-vehicle object is in an abnormal state according to the in-vehicle space model;

[0096] If the judgment result is yes, the corresponding risk warning information is sent to the user.

[0097] Optionally, the AR glasses are equipped with an environment sensor and a posture sensor; the in-vehicle image determination unit 504 is specifically used to:

[0098] Acquire surrounding environmental information according to the environmental sensor, wherein the environmental information is used to characterize surrounding objects and their positions;

[0099] Determine the user's line of sight direction according to the posture sensor, and determine the visual area of ​​the field of view according to the line of sight direction and the environmental information;

[0100] The target vehicle is determined to be in the visible area of ​​the field of view according to the objects and their positions represented by the environmental information.

[0101] Optionally, the device further comprises:

[0102] A monitoring screen display unit, used to obtain the monitoring screen of the target vehicle;

[0103] When it is determined that the target vehicle is not in the visible area of ​​the user's field of view, or the user enters the target vehicle, the in-vehicle space model and / or the monitoring screen is displayed.

[0104] Optionally, the model and information acquisition unit 502 is specifically used to:

[0105] Initiate a model acquisition request to the target vehicle to obtain the interior space model returned by the target vehicle; or

[0106] When the target vehicle meets the model pushing condition, the vehicle interior space model pushed by the target vehicle is received.

[0107] Optionally, the relative position information is generated by the target vehicle or the AR glasses according to a corresponding sensor.

[0108] Based on the same concept as the above method, this specification also provides an electronic device, including: a processor; a memory for storing processor executable instructions; wherein the processor implements the steps of the method described in any of the above embodiments by running the executable instructions.

[0109] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0110] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

[0111] Although the present invention includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments of the present invention may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claimed as such, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0112] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0113] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for displaying an in-vehicle image, characterized in that: Applied to AR glasses; the method comprises: Acquire the interior space model of the target vehicle and the relative position information between the user and the target vehicle; In the case where it is determined that the target vehicle is in the visible area of ​​the user's field of vision, determining an in-vehicle image presented by the in-vehicle space model from the user's perspective according to the relative position information; The in-vehicle image is superimposed and displayed on the corresponding position of the target vehicle in the visual area of ​​the field of view.

2. The method according to claim 1, characterized in that The determining, according to the relative position information, an in-vehicle image presented by the in-vehicle space model from a user's perspective includes: Determining the model size and angle of the in-vehicle space model relative to the user according to the relative position information; The in-vehicle space model is projected onto a two-dimensional plane to obtain the corresponding in-vehicle image.

3. The method according to claim 1, characterized in that The target vehicle corresponds to at least one in-vehicle object; the method further includes: In the case where the at least one in-vehicle object is blocked by the target vehicle, a transparent gradient process is performed on the blocked portion of the in-vehicle image corresponding to the target vehicle, and an image enhancement process is performed on the in-vehicle object in the in-vehicle image.

4. The method according to claim 1, characterized in that The target vehicle corresponds to at least one in-vehicle object; the method further includes: determining whether the at least one in-vehicle object is in an abnormal state according to the in-vehicle space model; If the judgment result is yes, the corresponding risk warning information is sent to the user.

5. The method according to claim 1, characterized in that: The AR glasses are equipped with an environmental sensor and a posture sensor; the determining that the target vehicle is in a visible area of ​​the user's field of view includes: Acquire surrounding environmental information according to the environmental sensor, wherein the environmental information is used to characterize surrounding objects and their positions; Determine the user's line of sight direction according to the posture sensor, and determine the visual area of ​​the field of view according to the line of sight direction and the environmental information; The target vehicle is determined to be in the visible area of ​​the field of view according to the objects and their positions represented by the environmental information.

6. The method according to claim 1, characterized in that The method further comprises: Get the monitoring picture of the target vehicle; When it is determined that the target vehicle is not in the visible area of ​​the user's field of view, or the user enters the target vehicle, the in-vehicle space model and / or the monitoring screen is displayed.

7. The method according to claim 1, characterized in that The step of obtaining the interior space model of the target vehicle includes: Initiate a model acquisition request to the target vehicle to obtain the interior space model returned by the target vehicle; or When the target vehicle meets the model pushing condition, the vehicle interior space model pushed by the target vehicle is received.

8. The method according to claim 1, characterized in that The relative position information is generated by the target vehicle or the AR glasses according to the corresponding sensors.

9. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Images