A vehicle-based interactive display system

Abstract

The application discloses a kind of interactive display systems based on carrier, comprising: carrier is provided with camera for real-time acquisition in visual range Environment image;Controller is used to control the movement of carrier in local environment;Computing unit is used to identify whether the environment image collected by camera contains obstacle in real time, and when it is determined that the environment image contains obstacle, the corresponding virtual content information is obtained;Display device is used to receive the environment image collected by camera and the generated virtual content information, and real-time display environment image and superimposed virtual content.The interactive display system of the application helps to improve the diversity of virtual display technology in human-computer interaction.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a vehicle-based interactive display system. Background Technology

[0002] With the development of technology and the increasing prevalence of machine intelligence and information intelligence, virtual display technology has entered people's daily lives. For example, in augmented reality technology, a common presentation method involves using a smart terminal's camera to capture and display images of the real environment, and then overlaying virtual objects onto these images. Currently, virtual display technology lacks diverse applications, and how to further enhance its versatility in human-computer interaction has become a key research direction for engineers. Summary of the Invention

[0003] This application proposes a vehicle-based interactive display system, comprising: a vehicle equipped with a camera for real-time acquisition of environmental images within its visual range; a controller for controlling the movement of the vehicle within a local environment; a computing unit for real-time identification of whether the environmental images acquired by the camera contain obstacles, and for obtaining corresponding virtual content information when it is determined that the environmental images contain obstacles; and a display device for receiving the environmental images acquired by the camera and the generated virtual content information, and for real-time display of the environmental images and the superimposed virtual content.

[0004] Furthermore, the camera is mounted on the vehicle via a rotating platform, which can rotate horizontally relative to the vehicle. The controller includes a camera rotation control module for controlling the horizontal rotation angle of the camera relative to the vehicle.

[0005] Furthermore, the system also includes a scanning unit, which drives the camera to rotate horizontally one revolution relative to the vehicle each time it is turned on, in order to obtain information about the vehicle's surrounding environment.

[0006] Furthermore, when determining that an obstacle is present, the computing unit is also used to calculate and obtain the position data of the obstacle in the environmental image in real time, and generate virtual content information containing position information based on the position data; the display device receives the virtual content information containing the position information, and overlays and displays the virtual content at the position corresponding to the position information in the environmental image.

[0007] Furthermore, the computing unit is also used to compare the acquired location information with a preset area of ​​the environmental image, and when the location data is within the preset area, generate obstacle avoidance prompt information; the display device receives the obstacle avoidance prompt information and displays it.

[0008] Furthermore, the calculation unit is also used to calculate the area ratio of the obstacle in the environmental image, and based on the area ratio, calculate the display ratio of the corresponding virtual content; the display device displays the corresponding virtual content according to the display ratio.

[0009] Furthermore, when determining that an obstacle is present, the computing unit is also used to calculate and determine the identity feature data of the obstacle, and based on the identity feature data, generate virtual content information corresponding to the identity data of the obstacle.

[0010] Furthermore, the system also includes a server, and the computing unit also sends the generated virtual content information to the server for display devices in other locations to display the virtual content corresponding to the local obstacles.

[0011] Furthermore, the system also includes a server that receives another virtual content information, the other virtual content including another virtual content generated based on obstacles in the remote environment captured by a camera on a remote vehicle; the display device is also used to overlay and display the other virtual content.

[0012] Furthermore, the vehicle also includes a first vehicle and a second vehicle. The first vehicle is controlled by a first controller and moves in the local environment, and the second vehicle is controlled by a second controller and moves in the local environment. The computing unit is used to determine in real time whether the environmental image captured by the camera of the first vehicle contains obstacles, and when it is determined that there are obstacles, it generates corresponding first virtual content information. The computing unit is also used to obtain the position information of the first and second vehicles in the local environment, and generate virtual position content information of the second vehicle relative to the first vehicle. The display device receives the virtual position content and displays the environmental image and the superimposed first virtual content, as well as the content of the virtual position of the second vehicle relative to the first vehicle.

[0013] The interactive display system of this application utilizes the ease of controlling the movement of a vehicle. Through an operation controller, users can view images captured by a camera on the vehicle during its movement on a display device in real time. Furthermore, a computing unit identifies in real time whether there are obstacles in the captured images. When an obstacle is identified, the system obtains virtual content information corresponding to that obstacle. This allows the display device to overlay the corresponding virtual content onto the images captured by the camera, thereby enhancing the diversity of virtual display technology in human-computer interaction. Attached Figure Description

[0014] Figure 1A schematic diagram of the framework of an interactive display system provided in an embodiment of this application is shown.

[0015] Figure 2A-2B A schematic diagram of virtual content placement provided in one embodiment of this application is shown.

[0016] Figure 2C A schematic diagram of virtual content placement provided in another embodiment of this application is shown.

[0017] Figure 3 A schematic diagram of the framework of an interactive display system provided in another embodiment of this application is shown. Detailed Implementation

[0018] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0019] In the description of this application, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0020] In the description of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "installed," "equipped with," "connected," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0021] With the development of virtual display technology, related applications have gradually entered people's daily lives. Utilizing the convenience of vehicle movement control, research on interactive technologies combining vehicles with virtual display technology has become one of the inventors' research directions. After in-depth research, the inventors have proposed the architecture of the vehicle-based interactive display system of this application. The interactive display system provided in this application will be described in detail below through specific embodiments.

[0022] Please see Figure 1This illustration shows the architecture of an interactive display system 100 provided in an embodiment of this application. The interactive display system 100 includes a carrier 10, a controller 20, and a display device 30. The carrier 10 is equipped with a camera 11, which, upon activation, can capture environmental images within its field of vision in real time. The controller 20 is communicatively connected to the carrier 10 and is equipped with control components for controlling the movement of the carrier 10 in the real-world environment. The display device 30 has a display unit and is communicatively connected to the carrier 10. The environmental images captured in real time by the camera 11 on the carrier 10 can be transmitted to the display device 30, allowing the display unit of the display device 30 to display the images captured by the camera 11 in real time. Users can operate the controller 20 to view the images captured in real time by the camera 11 on the carrier 10 during the movement of the carrier 10.

[0023] The controller 20 and the vehicle 10 can be connected via radio, Bluetooth, or wireless network to control the vehicle 10. The display device 30 and the camera 11 of the vehicle 10 can be connected via Bluetooth, 4G / 5G, Wi-Fi, etc., to receive and display image data captured by the camera 11. Of course, the communication method between the controller 20, the vehicle 10, and the display device 30 is not limited in this embodiment and can also be other unmentioned communication methods.

[0024] The display device 30 can be in the form of a smart mobile terminal, a display screen, a projection device, etc. The specific form is not limited in this application, as long as it can receive image data captured by the camera 11 and display it to the user.

[0025] In some embodiments, the camera 11 is mounted on the vehicle 10 via a rotating platform (not shown). The rotating platform 11 can rotate horizontally relative to the vehicle 10. The controller 20 can be equipped with a camera rotation control module, allowing the user to control the horizontal rotation angle of the camera 11 relative to the vehicle 20, thereby viewing images of the vehicle's surroundings that the user wishes to see. In one embodiment, the camera 11 is also mounted on the rotating platform via a pitch platform (not shown). The pitch platform can tilt relative to the rotating platform. The controller 20 also includes a camera pitch control module for controlling the pitch angle of the camera 11 relative to the vehicle 10. The user can adjust the viewing angle of the image captured by the camera 11 according to their needs via the controller 20.

[0026] In some implementations, vehicle 10 has the function of Simultaneous Localization and Mapping (SLAM). During the movement of vehicle 10 in the real environment, it can build a map and structure in the real environment and locate the position and orientation of vehicle 10 in the real environment.

[0027] In this embodiment, the interactive display system 100 further includes a computing unit 40. The computing unit 40 can perform recognition processing on the environmental image captured by the camera 11 on the vehicle 10 to determine whether the environmental image contains obstacles. When the computing unit 40 determines that the captured environmental image contains obstacles, the computing unit 40 obtains virtual content information corresponding to the identified obstacles based on the identified obstacles and sends the virtual content information to the display device 30. Upon receiving the virtual content information, the display device 30 can overlay the virtual content corresponding to the virtual content information onto the real-time displayed environmental image. Here, obstacles refer to actual objects in the real environment that have a potential impact on the movement of the vehicle 10 during its movement. Users can control the vehicle 10 through the controller 20 to avoid obstacles. Virtual content information refers to data information such as images or models that can enable the display device 30 to display corresponding virtual content. Virtual content information can be pre-stored on a designated server or stored on a public server in the cloud. When the computing unit 40 identifies an obstacle, it obtains the virtual content information corresponding to the obstacle from the server. The computing unit 40 can use artificial intelligence algorithms to identify whether there are obstacles in the image. The specific method of identifying obstacles is not limited in this application.

[0028] The interactive display system of this application utilizes the ease of controlling the movement of the vehicle 10. Through the operation controller 20, the user can view in real time the images captured by the camera 11 on the vehicle 10 during the movement of the vehicle 10 on the display device 30. Furthermore, the computing unit 40 identifies in real time whether there are obstacles in the captured images. When an obstacle is determined to exist, the system obtains the virtual content information corresponding to the obstacle. This allows the display device 30 to overlay the corresponding virtual content on top of the images captured by the camera 11, thereby improving the diversity of virtual display technology in human-computer interaction.

[0029] The computing unit 40 may include any suitable type of general-purpose or special-purpose microprocessor, digital signal processor, or microcontroller. The computing unit 40 may be configured to receive data and / or signals from various components of the system via, for example, a network. The computing unit 40 may also process the data and / or signals to determine one or more operating conditions in the system. For example, the computing unit 40 may generate image data corresponding to pre-stored virtual content information and send it to a display device for display; it may also receive image data captured by a camera via a wired or wireless network, analyze the received image data, identify obstacle information in the image data, and obtain the corresponding virtual content information.

[0030] The computing unit 40 may be a processor integrated on the carrier 10 or on the display device 30. In some embodiments, the computing unit 40 may also be integrated on a local or cloud server, meaning that information processing tasks such as identifying whether the environmental image contains obstacles and obtaining the corresponding virtual content information when obstacles are determined to be present are all performed on the server.

[0031] In some implementations, the interactive display system 100 also has a scanning function. When the scanning function is activated, each time the camera 11 of the vehicle 10 is turned on, the system 100 drives the camera 11 to rotate horizontally one full circle relative to the body of the vehicle 10 through the scanning unit to obtain the surrounding environmental information. This makes it easier for the computing unit 40 to identify the obstacle information around the vehicle 10 and obtain the corresponding virtual content information in advance. This allows the vehicle 10 to quickly call up the corresponding virtual content to display on the display device 30 no matter which direction it moves, thereby improving the response time of the computing unit 40 and saving computing power.

[0032] In some embodiments, the position of the virtual content displayed by the display device 30 in the environmental image is related to the position of the actual obstacle in the environmental image. When the calculation unit 40 determines that the environmental image contains an obstacle, it further calculates and obtains the position data of the obstacle in the environmental image in real time, and generates virtual content information containing position information based on the position data; when the display device 30 receives the virtual content information containing position information, it overlays and displays the corresponding virtual content at the position in the environmental image corresponding to the position information.

[0033] In one embodiment, the position data of the obstacle in the environmental image can be the same as the position information in the virtual content information. That is, the position point of the virtual content superimposed on the environmental image is the same as the position point of the obstacle in the environmental image. The virtual content information includes information about the placement point of the virtual content, and the superimposed display of the virtual content on the environmental image is based on the alignment of the placement point of the virtual content with the superimposed display position point. The position point of the obstacle in the environmental image can be the geometric center of the obstacle's image. The specific method for calculating the geometric center is not limited. Of course, the position point of the obstacle in the environmental image can also be other possible feature point positions, which will not be described in this embodiment. Figure 2A-2B As shown, in the environmental image 200a, the calculation unit 40 calculates the geometric center point 22a of the obstacle 21a image, which is the superimposed display point in the virtual content information. At this time, the position information in the virtual content information can instruct the display device 30 to superimpose the virtual content 23a with the geometric center point 22a as the placement point of the virtual content.

[0034] In another embodiment, the overlay display position of the virtual content in the environmental image and the position position of the obstacle in the environmental image may have a preset relationship, which can be a direction and distance value. For example, such as Figure 2C As shown, obstacle 21b is in the environmental image 200b. The calculation unit 40 calculates the geometric center point 22b of the obstacle 21b image. The preset relationship includes a distance deviation value d and a direction deviation of n o'clock. At this time, the position information in the virtual content information can instruct the display device 30 to display the virtual content 23b at the placement point 24b at a distance d from the geometric center point 22b in the n o'clock direction. It should be noted that when the placement point is calculated to be outside the environmental image according to the preset relationship and the geometric center point of the obstacle image, the calculation unit 40 can further perform mapping processing to map the placement point into the environmental image. For example, the edge of the environmental image adjacent to the placement point can be used as the axis of symmetry to map the placement point into the environmental image to obtain an updated placement point.

[0035] In some implementations, in addition to the correlation between the position of the virtual content overlaid on the environmental image and the position of the obstacle in the environmental image, the size of the virtual content displayed by the display device 30 can also be related to the size of the obstacle in the environmental image. The calculation unit 40 can calculate the area ratio of an obstacle's image in the environmental image, and based on this area ratio value, calculate the corresponding display ratio value of the virtual content. The display device 30 then displays the corresponding virtual content according to this display ratio value. For example, when the area ratio of an obstacle in the environmental image is found to be 20%, the corresponding display ratio value is calculated to be 2. The display device 30 can then enlarge the corresponding virtual content by 2 times its original model size and display it at the corresponding position.

[0036] In some embodiments, the display device 30 can also display obstacle avoidance prompts. The computing unit 40 can compare the acquired location information with a preset area of ​​the environmental image. When the location information indicates that the virtual content is placed within the preset area, the computing unit 40 can generate corresponding obstacle avoidance prompts and send them to the display device 30 for display. The method of displaying the obstacle avoidance prompts is not limited in this application; it can be a method such as highlighting the outline of the displayed virtual content in red, or overlaying a prompt information box onto the environmental image.

[0037] In some embodiments, when the computing unit 40 determines that the environmental image contains information about an obstacle, it can further calculate and determine the identity feature data of the obstacle, and based on the identity feature data, obtain virtual content information corresponding to the identity data of the obstacle. The virtual content information corresponding to the identity feature data can be pre-stored. For example, if the computing unit 40 identifies an obstacle as a sphere in the environmental image, and the system pre-stores animation information of a virtual cat corresponding to the sphere, the computing unit 40 can directly retrieve the animation information of the virtual cat and send it to the display device 30 for display. After receiving the animation information of the virtual cat, the display device 30 displays the animation of the virtual cat at the position corresponding to the sphere in the environmental image. In one embodiment, the virtual content information corresponding to the identity feature data can be randomly obtained. For example, if the computing unit 40 identifies an obstacle as a sphere in the environmental image, the computing unit 40 can intelligently and randomly obtain it from a database or server storing virtual content information, such as obtaining a virtual cat, virtual dog, or virtual ball-like object (basketball, football, etc.) associated with the sphere. This application does not limit the technical architecture of randomly obtaining virtual content.

[0038] In some embodiments, the computing unit 40 is further configured to identify and determine the identity features of obstacles to obtain identity feature data corresponding to the obstacles; based on the identity feature data corresponding to the obstacles, the computing unit 40 obtains the corresponding virtual content information, which includes feature data of the virtual content; the computing unit 40 can obtain the mapping relationship between the identity feature data corresponding to the obstacles and the feature data of the virtual content, and send the mapping relationship to the memory for storage, so that when the computing unit determines that the obstacle is re-included in the environmental image, it can directly obtain the virtual content corresponding to the obstacle based on the mapping relationship. In one embodiment, when the system restarts or is restarted, the mapping relationship is deleted or overwritten.

[0039] In some embodiments, while generating virtual content information containing location information, the computing unit 40 can also identify the identity features of obstacles to obtain corresponding identity feature data; based on the identity feature data corresponding to the obstacles, the computing unit 40 obtains the corresponding virtual content information, which includes feature data of the virtual content; the computing unit 40 can calculate the correlation between the identity feature data corresponding to the obstacles and the feature data in the virtual content information, thereby determining the placement relationship of the virtual content relative to the obstacles in the environmental image, so that the display device 30 displays the corresponding virtual content according to the placement relationship.

[0040] The placement relationship can refer to the masking relationship between the displayed virtual content and obstacles in the environmental image. For example, if the virtual content is a cat and the obstacle is a ball, the computing unit can determine the overlapping part of the virtual content and the obstacle in the environmental image, i.e., the relevant part, based on the position information of the virtual content, and mask the overlapping part of the virtual content and the obstacle to give the user the visual perception that the virtual cat is behind the obstacle. When the virtual content is a soccer ball and the obstacle is a circular object, the computing unit generates virtual soccer ball information superimposed on the obstacle, i.e., the entire virtual soccer ball is superimposed on the obstacle. It should be noted that this application does not limit the specific method of determining whether the virtual content of the overlapping part is masked by the obstacle or superimposed on the obstacle. Of course, this placement relationship is not limited to the masking relationship between the virtual content and the obstacle; for example, it can also be other placement relationships such as distance and scale, which will not be described in detail in this application.

[0041] In some embodiments, when the computing unit 40 determines that the environmental image contains information about obstacles, it can further identify and determine the shape of the obstacles and obtain corresponding shape data. Based on the shape data, the computing unit 40 can obtain virtual content information corresponding to the shape data. For example, when the computing unit 40 identifies an obstacle as circular in the environmental image, it can obtain circular or elliptical virtual content from a database or server storing virtual content information. This application does not limit how the specific virtual content information is related to the shape data.

[0042] In some embodiments, when the computing unit 40 determines that the environmental image contains obstacles, it can further identify and determine the color of the obstacles and obtain the corresponding color data. Based on this color information, the computing unit can obtain virtual content information corresponding to the color data. For example, when the computing unit 40 identifies an obstacle as blue in the environmental image, it can obtain virtual content similar to blue from a database or server storing virtual content information. This application does not limit how the specific virtual content information is related to the color data.

[0043] The interactive display system described in this application can also be applied to multi-person interactive scenarios.

[0044] The local interactive display system 100 communicates with the remote interactive display system through a server. The computing unit 40 can also send the generated virtual content information to the server. The server sends the generated virtual content information to the display device in the remote interactive display system. The remote display device can receive the virtual content and display it. That is, the remote display device can display images of the remote environment, virtual content from the remote location, and virtual content corresponding to the virtual content information sent locally.

[0045] The local interactive display system 100 can also receive virtual content information from other locations through the server. The virtual content refers to the virtual content generated based on obstacles in the environment captured by the camera on the vehicle in another location. The display device 30 can further overlay and display the virtual content.

[0046] like Figure 3As shown, in some embodiments, the vehicles in the interactive display system 300 include a first vehicle 10a and a second vehicle 10b. The first vehicle 10a is controlled by a first controller 20a and moves in the local environment, while the second vehicle 10b is controlled by a second controller 20b and moves in the local environment. The computing unit 40 can determine in real time whether the environmental image captured by the camera of the first vehicle 10a contains obstacles, and when it is determined that obstacles are contained, it obtains the corresponding first virtual content information. The computing unit 40 is also used to obtain the position information of the first vehicle 10a and the second vehicle 10b in the local environment, and generate virtual position content information of the second vehicle 10b relative to the first vehicle 10a. The display device 30 is used to display the image captured by the camera of the first vehicle 10a. The display device 30 receives the virtual position content information and, while displaying the environmental image and the superimposed first virtual content, also displays the content of the virtual position of the second vehicle 10b relative to the first vehicle 10a. The location information of the first vehicle 10a and the second vehicle 10b in the local environment can be obtained through a SLAM algorithm or through GPS positioning; this application does not impose any specific limitations. Furthermore, the form in which the display device 30 displays the virtual position of the second vehicle 10b relative to the first vehicle 10a is also not limited in this application. For example, it can overlay a comparison of the two vehicles' positions on a map in the lower right corner of an environmental image, where the map can be generated in real-time using a SLAM algorithm.

[0047] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. In the absence of conflict, the above embodiments can be combined with each other, and the features of each embodiment can also be combined with each other, and are not limited to the embodiments.

Claims

1. A vehicle-based interactive display system, characterized by, The system includes: A vehicle equipped with a camera for real-time acquisition of environmental images within its visual range; A controller for controlling the movement of the vehicle in the local environment; The computing unit is configured to identify in real time whether an obstacle is present in the environmental image captured by the camera, and when it is determined that an obstacle is present in the environmental image, to calculate and obtain the position data of the obstacle in the environmental image and to calculate and determine the identity feature data of the obstacle; based on the identity feature data, to obtain virtual content information corresponding to the obstacle identity feature data, wherein the virtual content information includes feature data of the virtual content; to calculate the correlation between the obstacle identity feature data and the feature data of the virtual content; to determine the placement relationship of the virtual content relative to the obstacle in the environmental image according to the correlation and the position data of the obstacle in the environmental image, wherein the placement relationship includes the occlusion relationship between the virtual content and the obstacle in the environmental image; and to generate virtual content information containing the placement relationship. The display device is used to receive environmental images captured by the camera and generated virtual content information containing the placement relationship, display the environmental images in real time, and overlay the virtual content in the environmental images at the positions corresponding to the placement relationship.

2. The system of claim 1, wherein: The camera is mounted on the vehicle via a rotating platform, which can rotate horizontally relative to the vehicle. The controller includes a camera rotation control module for controlling the horizontal rotation angle of the camera relative to the vehicle.

3. The system of claim 2, wherein: The system also includes a scanning unit, which drives the camera to rotate horizontally one revolution relative to the vehicle each time it is turned on, in order to obtain information about the vehicle's surrounding environment.

4. The system of claim 1, wherein: The computing unit is also used to compare the acquired location data with a preset area of ​​the environmental image, and generate obstacle avoidance prompt information when the location data is within the preset area; The display device receives the obstacle avoidance prompt information and displays it.

5. The system of claim 1, wherein: The calculation unit is also used to calculate the area ratio of the obstacle in the environmental image, and based on the area ratio, calculate the display ratio of the corresponding virtual content. The display device displays the corresponding virtual content according to the display ratio value.

6. The system as described in claim 1, characterized in that: The system also includes a server, and the computing unit also sends the generated virtual content information to the server for display devices in other locations to display the virtual content corresponding to the local obstacles.

7. The system as described in claim 1, characterized in that: The system also includes a server that receives another virtual content information, which includes another virtual content generated based on obstacles in a remote environment captured by a camera on a remote vehicle. The display device is also used to overlay the other virtual content.

8. The system as described in claim 1, characterized in that: The vehicle also includes a first vehicle and a second vehicle, the first vehicle being controlled by a first controller and moving in the local environment, and the second vehicle being controlled by a second controller and moving in the local environment; The computing unit is used to determine in real time whether the environmental image captured by the camera of the first vehicle contains obstacles, and when it is determined that there are obstacles, it generates corresponding first virtual content information; The computing unit is also used to obtain the location information of the first and second vehicles in the local environment, and generate virtual location content information of the second vehicle relative to the first vehicle; The display device receives the virtual location content information and, while displaying the environmental image and the superimposed first virtual content, also displays the virtual location of the second vehicle relative to the first vehicle.

Images