Adaptive display method, device and storage medium for vehicle

Abstract

An adaptive display method, device and storage medium for a vehicle are disclosed. An adaptive display method for a vehicle comprises: receiving a first image obtained by a user photographing a vehicle interior in a first vehicle; generating a second image by replacing an image of a vehicle interior portion in the first image with an image of a corresponding vehicle interior portion of a second vehicle, wherein the second vehicle is a vehicle of interest to the user, and the image of the corresponding vehicle interior portion of the second vehicle is generated based on a 3D model of the second vehicle and / or a panoramic image of the interior of the second vehicle; and sending the second image for display to the user.

Description

Technical Field

[0001] The present disclosure relates to an adaptive display method, device and storage medium for a vehicle. Background Art

[0002] Augmented reality technology is a known technology. It is hoped that augmented reality technology can be applied to vehicle promotion scenarios to enhance user experience. Summary of the invention

[0003] An object of the present disclosure is to provide a new adaptive display method and device for a vehicle.

[0004] The present disclosure proposes an adaptive display method for a vehicle, the method comprising: receiving a first image obtained by a user photographing a vehicle interior in a first vehicle; generating a second image by replacing an image of a vehicle interior portion in the first image with an image of a corresponding vehicle interior portion of a second vehicle, wherein the second vehicle is a vehicle of interest to the user, and the image of the corresponding vehicle interior portion of the second vehicle is generated based on a 3D model of the second vehicle and / or a panoramic image of the interior of the second vehicle; and sending the second image for display to the user.

[0005] Other features and advantages of the present disclosure will become apparent from the following description with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure without limitation. In the various figures, like reference numerals are used to represent like items.

[0007] Figure 1 is a block diagram of an exemplary adaptive display device according to some embodiments of the present disclosure.

[0008] Figure 2 is a flow chart illustrating an exemplary adaptive display method according to some embodiments of the present disclosure.

[0009] Figure 3 is a flow chart illustrating exemplary detailed steps of an enhanced image generation process according to some embodiments of the present disclosure.

[0010] Figure 4 A general hardware environment in which the present disclosure may be applied according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0011] In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the described exemplary embodiments. However, it is clear to those skilled in the art that the described embodiments may be practiced without some or all of these specific details. In the described exemplary embodiments, well-known structures or processing steps are not described in detail in order to avoid unnecessarily obscuring the concepts of the present disclosure.

[0012] The blocks in each block diagram shown below can be implemented by hardware, software, firmware or any combination thereof to implement the principles of the present disclosure. It should be understood by those skilled in the art that the blocks described in each block diagram can be combined or divided into sub-blocks to implement the principles of the present disclosure.

[0013] The steps of the method presented in this disclosure are intended to be illustrative. In certain embodiments, the method can be completed with one or more additional steps not described and / or completed without the one or more steps discussed. In addition, the order in which the steps of the method are illustrated and described is not intended to be restrictive.

[0014] Figure 1 is a block diagram of an exemplary adaptive display device 100 according to some embodiments of the present disclosure. Figure 1 As shown, the device 100 may include: a receiving component 110, configured to receive a first image obtained by a user photographing a vehicle interior in a first vehicle; an enhanced image generating component 120, configured to generate a second image by replacing the image of the vehicle interior portion in the first image with the image of the corresponding vehicle interior portion of a second vehicle, wherein the second vehicle is a vehicle of interest to the user, and the image of the corresponding vehicle interior portion of the second vehicle is generated based on a 3D model of the second vehicle and / or a panoramic image of the interior of the second vehicle; a sending component 130, configured to send the second image for display to the user; a changing component 140, configured to change the color and / or material of at least a portion of the vehicle interior portion in the second image, and / or change the content related to the external environment of the vehicle in the second image; and a feedback collecting component 150, configured to receive data on user feedback while displaying the second image to determine user preferences. Although in Figure 1 Not shown, the device 100 may also include a local or remote storage component, which may store 3D models of multiple vehicles including the second vehicle and / or a panoramic image of the interior of the vehicle (such as a 360° panoramic image), and the storage component may also temporarily cache the generated second image. The storage component may also store other data or information according to actual needs.

[0015] Here, the first vehicle may be the vehicle at the user's current location, may be a self-owned vehicle or a vehicle being tested in a vehicle showroom, and the second vehicle may be a vehicle that the user is interested in and wants to buy. The vehicle interior refers to the overall configuration of the vehicle's interior, including the configuration of the vehicle's steering wheel, instrument panel, center console, seats, doors, sunroof, etc. The vehicle interior may refer to the interior environment of the vehicle.

[0016] The adaptive display device 100 according to the present disclosure may be a terminal device, such as a smart terminal device used by a user to perform shooting, or may be a server device, such as a server device maintained by a vehicle manufacturer or dealer. In the present disclosure, the smart terminal device may include: a smart phone, a tablet computer, AR (augmented reality) glasses, MR (mixed reality) glasses, etc.

[0017] The following will further describe in detail Figure 1 The operation of each component is shown.

[0018] Figure 2 is a flow chart illustrating an exemplary adaptive display method 200 according to some embodiments of the present disclosure.

[0019] The method 200 starts from step S210, where the receiving component 110 receives a first image (i.e., an original image) obtained by a user shooting the interior environment of the vehicle in the first vehicle. Here, the first image includes a plurality of images constituting a video stream. In other words, the first image corresponds to a frame constituting the video stream. The following describes a processing method for one frame of an image, however, it should be understood that each frame of an image in a video stream can be processed in a similar manner. The user can use the smart terminal device as described above to obtain the first image.

[0020] The following scenario can be considered: the user sits in the driver's seat of his own vehicle, raises his smartphone and shoots the scene inside the car from left to right, thereby generating a corresponding video stream. In this video stream, for example, images of the vehicle's left rearview mirror, left A-pillar, front windshield, steering wheel, instrument panel, central control, vehicle's right A-pillar, and right door will be included. More specifically, for example, in the first frame of this video stream, images of the vehicle's left rearview mirror, left A-pillar, front windshield, steering wheel, and instrument panel will be included. By the way, in the case where the front windshield is included in the first frame, the first frame may also contain content about the vehicle's external environment, such as grass and trees outside the vehicle.

[0021] The method 200 proceeds to step S220, where the enhanced image generation component 120 generates a second image (i.e., an enhanced image) by replacing the image of the vehicle interior portion (hereinafter referred to as the first interior portion) in the first image with the image of the corresponding vehicle interior portion (hereinafter referred to as the second interior portion) of the second vehicle. Figure 3 Exemplary detailed steps of the second image generation process will be described below.

[0022] At step S222, the enhanced image generation component 120 determines the position and perspective of the user in the first vehicle by analyzing the first image. Here, the position of the user in the first vehicle may include, for example, the driving seat, the co-pilot seat, the left position of the rear row, the middle position of the rear row, and the right position of the rear row. The perspective of the user in the first vehicle may include, for example, the front, rear, left, right, upper, and lower perspectives, the left front, right front, left rear, right rear perspectives, the upper left, upper right, lower left, and lower right perspectives, as well as the upper front, upper rear, lower front, and lower rear perspectives. It should be understood that the position and perspective of the user in the vehicle are not limited to this. The position and perspective of the user in the vehicle can be further subdivided. Machine learning technology can be used to train a model for analysis based on picture data taken from multiple perspectives at multiple positions in similar vehicles. Here, similar vehicles refer to similar models, such as picture data taken in other five-seater vehicles. The trained model can be used to determine the position and perspective of the user in the first vehicle. For example, when the first image includes images of the vehicle's left rearview mirror, left A-pillar, front windshield, steering wheel, and instrument panel, it can be determined that the user is sitting in the driver's seat and shooting toward the left front.

[0023] At step S224, the enhanced image generation component 120 uses the determined position and viewing angle to determine an image of the second interior part. More specifically, the enhanced image generation component 120 first uses the determined position and viewing angle to determine an interesting portion of the 3D model and / or panoramic image that can be seen at the determined position in the second vehicle at the determined viewing angle, and then determines an image of the second interior part to be used to replace the first interior part through feature matching or similarity calculation.

[0024] The following describes a scenario in which the image of the second interior part is determined by feature matching. A first group of feature points may be annotated on the image of the first interior part, and a second group of feature points may be annotated on the part of interest of the 3D model and / or the panoramic image, and further, the degree of match (or proximity) between the feature points of the first group and the feature points of the second group may be calculated. And, for example, the image of the second interior part that has the highest degree of match with the first interior part may be determined as the image to be used for replacement. Here, the feature points may be, for example, feature points representing the edges of key components in the interior part. Key components refer to components with high recognition, such as steering wheels, rearview mirrors, car doors, and the like. It should be understood that the features considered in the feature matching process are not limited to feature points, but other features such as line features, surface features, and the like may also be used.

[0025] Alternatively, the image of the second interior part may be determined by similarity calculation. Similarity calculation (or correlation calculation) may be performed on the image of the first interior part and the portion of interest of the 3D model and / or the panoramic image. And, for example, the image of the second interior part having the highest similarity to the first interior part may be determined as the image to be used for replacement.

[0026] It should be understood that the first image is usually a 2D image. In order to complete the feature matching or similarity calculation between the first image and the part of interest of the 3D model, the part of interest of the 3D model can be transformed into a 2D image. Alternatively, in the case where the first image contains depth information, the first image can be transformed into a 3D body and then feature matching or similarity calculation with the part of interest of the 3D model is performed.

[0027] In addition, the panoramic image is usually a 2D image. Therefore, feature matching or similarity calculation can be performed directly between the first image and the panoramic image. Alternatively, in the case where both the first image and the panoramic image contain depth information, feature matching or similarity calculation of the two can be performed after the interested parts of the first image and the panoramic image are respectively transformed into 3D bodies.

[0028] At step S226, the enhanced image generation component 120 replaces the image of the first interior part with the image of the second interior part determined in step S224, thereby generating a second image. The component 120 may remove the image of the first interior part from the first image, and then fill the blank portion of the first image with the image of the second interior part, thereby generating the second image. Alternatively, the component 120 may directly superimpose the image of the second interior part onto the image of the first interior part in the first image, thereby generating the second image.

[0029] It should be understood that in order to make the second image look natural and realistic, various known image processing can be performed. For example, before performing the replacement, component 120 can perform scaling, rotation, cropping, etc. on the image of the second interior part so that the image of the second interior part has substantially the same shape and size as the image of the first interior part. Here, "substantially the same" means that the shape difference (such as the ratio of non-overlapping edges) or size difference between the two does not exceed 5% compared to the image of the first interior part. For another example, after performing the replacement, component 120 can perform lighting adjustment, edge smoothing, etc. on the image of the second interior part.

[0030] At step S226, if the first image contains content about the vehicle's external environment as mentioned above, then the content is retained in the second image. That is, while replacing the image of the first interior part with the image of the second interior part, the environmental content is retained unchanged. In this way, a more realistic experience can be brought to the user.

[0031] Alternatively, if the first image contains content about the external environment of the vehicle, the display effect of this content can be adjusted based on the color tone of the second interior part. For example, when the second interior part has a warm color tone (orange, yellow, etc.), the brightness, chroma, saturation, etc. of this content can be adjusted to achieve a warm color display effect. Alternatively, when the second interior part has a cool color tone (black, cyan, etc.), the brightness, chroma, saturation, etc. of this content can be adjusted to achieve a cool color display effect. In some cases, various filters can be applied to the environmental content to match it with the color tone of the second interior part.

[0032] In addition, if the first image contains content about an object other than the vehicle interior in the vehicle, the content about the object is retained in the second image. For example, when a user takes a photo of a vehicle seat, there is a toy on the seat. When the seat image in the first image is replaced with the seat image of the second vehicle, the image of the toy can be retained unchanged. This can be achieved by identifying the object, cutting out the object, and superimposing the cut object in situ on the second image. Again, this can give the user a more realistic experience.

[0033] Next, the method 200 proceeds to step S230, where the sending component 130 sends the second image generated by the enhanced image generating component 120 to the user for display. In the case where the user uses a smart terminal device to obtain the first image, the component 130 can send the second image to the smart terminal device for display. It is easy to understand that in the case where the first image represents a video stream, the second image also represents a video stream, which is a video stream that has undergone content replacement processing as described above. It should be understood that the content replacement processing can be real-time, so the processed video stream can be displayed to the user in real time while the user is shooting. It should be understood that in the case of using AR glasses or MR glasses to display the second image, a 3D display of the second interior component can be provided.

[0034] Next, the method 200 proceeds to step S240, where the change component 140 changes the color and / or material of at least a portion of the vehicle interior part in the second image; and / or changes the content related to the vehicle external environment in the second image. The change operation can be performed in response to a user request. Alternatively, the change operation can be performed automatically. For example, while displaying the second image, a graphical user interface can be superimposed on the second image for the user to initiate a user request. For example, a plurality of touch buttons representing colors and / or materials can be superimposed on the second image for the user to click. In this case, the user can first click to select the interior part of interest (such as a vehicle seat) in the second image, and then click to select the touch button representing the color / material of interest, thereby changing the color / material of the interior part of interest (such as a vehicle seat) in the second image. In the case where the user requests to change the configuration of the vehicle interior, multiple configuration schemes of the vehicle interior that may be provided by the second vehicle can be displayed to the user, and these multiple configuration schemes differ in the color and / or material of the vehicle interior. These change operations can be implemented by using a pre-stored 3D model of the vehicle and multiple configuration schemes of the vehicle interior. In addition, in the case where the user requests to change the environmental content, a second image in which the environmental content is changed can be displayed to the user. For example, the grass and trees outside the first vehicle can be changed to a seaside scene. This change operation can be achieved by using multiple environmental images stored in association with the 3D model or panoramic image of the vehicle. By changing the interior configuration and / or environmental content, a richer experience can be brought to the user.

[0035] It should be understood that in step S240 , the modification component 140 may also change the second vehicle to be displayed in response to a user request.

[0036] Next, the method 200 proceeds to step S250, where the feedback collection component 150 receives data about user feedback while displaying the second image. User feedback may include, for example, the length of the user's viewing time, the user's degree of concentration, and the like. Further, the component 150 may determine the user's preference based on the user feedback data. It is easy to understand that the second vehicle or interior configuration that the user has watched for a longer time and has a higher degree of concentration will be the vehicle or configuration preferred by the user. Based on the determined user preferences, more accurate recommended content can be provided to the user.

[0037] Reference above Figure 2 and Figure 3 An exemplary adaptive display method 200 is introduced. It should be understood that method 200 can be performed separately by a smart terminal device or a server device. For example, method 200 can be performed offline by a user's smart phone or a tablet computer in a vehicle showroom. Alternatively, method 200 can be performed in collaboration by a smart terminal device and a server device. In this case, the smart terminal device and the server device respectively perform a portion of method 200. It should also be understood that the smart terminal device that shoots the first vehicle interior and the smart terminal device that displays the second image can be the same device or different devices. For example, shooting can be performed with a smart phone, while displaying can be performed using a projection device.

[0038] The disclosed adaptive display method and device for vehicles can enable users to experience the immersive feeling of sitting in the vehicle of interest in a simple and vivid way. In addition, users can also experience the realistic different interior configurations of the vehicle, which enhances the fun of users' car selection.

[0039] Hardware Implementation

[0040] Figure 4 A general hardware environment 400 is shown in which the present disclosure may be applied according to an exemplary embodiment of the present disclosure.

[0041] refer to Figure 4 , a computing device 400 will now be described as an example of a hardware device applicable to various aspects of the present disclosure. The computing device 400 may be any machine configured to perform processing and / or computing, and may be, but is not limited to, a workstation, a server, a desktop computer, a laptop computer, a tablet computer, a personal digital assistant, a smart phone, a portable camera, or any combination thereof. The above-mentioned apparatus 100 may be implemented in whole or at least in part by the computing device 400 or a similar device or system.

[0042] The computing device 400 may include an element that can be connected to or communicate with the bus 402 via one or more interfaces. For example, the computing device 400 may include a bus 402, one or more processors 404, one or more input devices 406, and one or more output devices 408. The one or more processors 404 may be any type of processor, and may include, but are not limited to, one or more general-purpose processors and / or one or more special-purpose processors (such as special-purpose processing chips). The input device 406 may be any type of device that can input information to the computing device, and may include, but are not limited to, a mouse, a keyboard, a touch screen, a microphone, and / or a remote controller. The output device 408 may be any type of device that can present information, and may include, but are not limited to, a display, a speaker, a video / audio output terminal, and / or a printer. The computing device 400 may also include or be connected to a non-transient storage device 410, which may be any storage device that is non-transient and can implement a data repository, and may include but is not limited to a disk drive, an optical storage device, a solid-state storage device, a floppy disk, a flexible disk, a hard disk, a magnetic tape or any other magnetic medium, a compact disk or any other optical medium, a ROM (read-only memory), a RAM (random access memory), a cache memory and / or any other memory chip or cartridge, and / or any other medium from which a computer can read data, instructions and / or code. The non-transient storage device 410 may be detachable from the interface. The non-transient storage device 410 may have data / instructions / code for implementing the above methods and steps. The computing device 400 may also include a communication device 412. The communication device 412 may be any type of device or system capable of communicating with an external device and / or with a network, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication equipment and / or a communication device such as Bluetooth. TM Chipsets for devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication facilities, etc.

[0043] The bus 402 may include, but is not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0044] Computing device 400 may also include working memory 414, which may be any type of working memory that can store instructions and / or data useful for operation of processor 404, and may include, but is not limited to, random access memory and / or read-only memory devices.

[0045] The software elements may be located in the working memory 414, including but not limited to an operating system 416, one or more applications 418, a driver and / or other data and code. Instructions for executing the above methods and steps may be included in one or more applications 418, and the components of the above device 100 may be implemented by the processor 404 reading and executing the instructions of one or more applications 418. More specifically, the enhanced image generation component 120 may be implemented, for example, by the processor 404 when executing an application 418 having an instruction to execute step S220 (or steps S222, S224, and S226). The change component 140 may be implemented, for example, by the processor 404 when executing an application 418 having an instruction to execute step S240. The feedback collection component 150 may be implemented, for example, by the processor 404 when executing an application 418 having an instruction to execute step S250. Similarly, the receiving component 110 and the sending component 130 may be implemented, for example, by the processor 404 when executing an application 418 having an instruction to execute steps S210 and S230, respectively. The executable code or source code of the instructions of the software elements may be stored in a non-transitory computer-readable storage medium (such as the storage device(s) 410 described above) and may be read into the working memory 414 where it may be compiled and / or installed. The executable code or source code of the instructions of the software elements may also be downloaded from a remote location.

[0046] From the above embodiments, it will be clear to those skilled in the art that the present disclosure can be implemented by software and necessary hardware, or can be implemented by hardware, firmware, etc. Based on this understanding, the embodiments of the present disclosure can be partially implemented in the form of software. Computer software can be stored in a computer-readable storage medium, such as a floppy disk, a hard disk, an optical disk, or a flash memory. Computer software includes a series of instructions that enable a computer (such as a personal computer, a service station, or a network terminal) to run a method according to each embodiment of the present disclosure or a part thereof.

[0047] The disclosure having thus been described, it will be apparent that the same may be varied in many ways. Such variations are not to be regarded as a departure from the spirit and scope of the disclosure, and all such modifications as would be obvious to one skilled in the art are intended to be included within the scope of the following claims.

Claims

1. An adaptive display method for a vehicle, comprising: Receiving a first image obtained by a user photographing a vehicle interior in a first vehicle; generating a second image by replacing the image of the vehicle interior portion in the first image with an image of the corresponding vehicle interior portion of a second vehicle, wherein the second vehicle is a vehicle of interest to the user, and the image of the corresponding vehicle interior portion of the second vehicle is generated based on a 3D model of the second vehicle and / or a panoramic image of the interior of the second vehicle; as well as sending the second image for display to the user, The generating of the second image comprises: Determine the position and viewing angle of the user in the first vehicle by analyzing the first image using machine learning technology, wherein the position of the user in the first vehicle includes one of the following: a driver's seat, a front passenger seat, a rear left position, a rear middle position, and a rear right position; and An image of a corresponding vehicle interior portion of a second vehicle is determined using the determined in-vehicle position and viewing angle.

2. The method according to claim 1, wherein: Based on the degree of matching between features of key components of the vehicle interior portion in the first image and features of key components of the corresponding vehicle interior portion of the second vehicle, an image of the corresponding vehicle interior portion of the second vehicle is determined.

3. The method according to claim 1, wherein: Based on the similarity between the image of the vehicle interior portion in the first image and the image of the corresponding vehicle interior portion of the second vehicle, the image of the corresponding vehicle interior portion of the second vehicle is determined.

4. The method according to claim 1, wherein: In the case where the first image includes content related to the vehicle's external environment, the display effect of the content related to the vehicle's external environment is adjusted based on the color tone of the corresponding vehicle interior part of the second vehicle.

5. The method according to claim 1, wherein: In the case where the first image contains content about an object other than the vehicle interior inside the vehicle, the content about the object is retained in the generated second image.

6. The method according to claim 1, further comprising: Changing the color and / or material of at least a portion of the vehicle interior in the second image; and / or The content of the second image related to the environment outside the vehicle is changed.

7. The method according to claim 1, wherein: The first image includes a plurality of images constituting a video stream.

8. An adaptive display device for a vehicle, comprising: at least one processor; as well as At least one storage device stores instructions, which, when executed by the at least one processor, cause the at least one processor to perform the method according to any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium, characterized in that: Instructions are stored which, when executed by a processor, cause the method according to any one of claims 1 to 7 to be performed.

10. A computer program product, characterized in that Contains instructions which, when executed by a processor, cause the method according to any one of claims 1 to 7 to be performed.

Images