Assisted driving method and apparatus, image collection method and apparatus, device, medium, and product

Abstract

An assisted driving method and apparatus, an image collection method and apparatus, a device, a medium, and a product, relating to the field of assisted driving. The method comprises: establishing a communication connection with a vehicular device, wherein the vehicular device comprises at least one camera facing a blind spot of a vehicle, and the blind spot refers to an outer area that cannot be observed by a wearer due to obstruction of an obstructing object (210); acquiring assistance data by means of the communication connection, wherein the assistance data is obtained on the basis of an environment image collected by the at least one camera (220); and on the basis of the assistance data, displaying a glasses image having see-through content, wherein the see-through content is an environment image of the blind spot presented after making the obstructing object transparent in the image collected by electronic glasses (230). In the method provided by the present application, when wearing electronic glasses, a wearer can see through obstructing objects in all directions, achieving complete removal of blind spots, enhancing driving safety.

Description

Method for assisting driving, method for collecting picture, device, equipment, medium and product

[0001] The present application claims priority to the Chinese patent application No. 202410657852.1, filed on May 24, 2024, and entitled "Method for assisting driving, method for collecting picture, device, equipment, medium and product", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of assisting driving, in particular to a method for assisting driving, a method for collecting picture, a device, an equipment, a medium and a product. BACKGROUND

[0003] With the evolution of electronic products and the upgrade of computer software, the assisting driving technology of vehicles has also developed, for example, radar sensing, intelligent parking, high-speed camera, intelligent vehicle light and other assisting driving technologies have appeared. However, in the actual driving process, the vehicle blind area is still the biggest trouble for the driver.

[0004] In the related art, there are solutions for some vehicle blind areas. These solutions often eliminate some vehicle blind areas by adding a display screen to the vehicle, for example, for the A-pillar blind area, the related art uses a display screen embedded in the A-pillar to display the picture collected by the camera outside the A-pillar in real time; for the rearview mirror blind area, two cameras are used to replace the rearview mirrors on both sides of the vehicle, and a display screen is added on both sides of the vehicle to display the picture collected by the corresponding camera.

[0005] The above method can eliminate some vehicle blind areas by adding a display screen, however, any vehicle blind area may cause serious traffic accidents when driving, and it is desirable to design a method that can completely eliminate vehicle blind areas. SUMMARY

[0006] The present application provides a method for assisting driving, a method for collecting picture, a device, an equipment, a medium and a product, and the technical solution is as follows:

[0007] According to an aspect of the present application, a method for assisting driving is provided, which is executed by electronic glasses worn by a wearer inside a vehicle, and the method comprises:

[0008] establishing a communication connection with a vehicle device, the vehicle device comprising at least one camera facing a blind area of the vehicle, the blind area referring to an external area that the wearer cannot observe due to an obstruction;

[0009] obtaining assistance data through the communication connection, the assistance data being obtained based on an environment picture collected by the at least one camera;

[0010] display a glasses picture with see-through content based on the auxiliary data, the see-through content being an environmental picture of the blind area presented in a picture captured by the electronic glasses after performing see-through display on the occlusion.

[0011] According to an aspect of the present application, a picture capturing method is provided, the method being performed by a vehicle device, the vehicle device having a first communication connection with at least one camera facing a blind area of a vehicle, the blind area being an external area that a wearer cannot observe due to an occlusion, the method comprising:

[0012] establishing a second communication connection with electronic glasses worn by the wearer inside the vehicle;

[0013] determining auxiliary data based on an environmental picture captured by the at least one camera, the environmental picture being obtained based on the first communication connection, the auxiliary data being used to cause the electronic glasses to display a glasses picture with see-through content, the see-through content being an environmental picture of the blind area presented in a picture captured by the electronic glasses after performing see-through display on the occlusion;

[0014] sending the auxiliary data to the electronic glasses through the second communication connection.

[0015] According to an aspect of the present application, an auxiliary driving device is provided, the device being performed by electronic glasses worn by a wearer inside a vehicle, the device comprising:

[0016] a first communication module configured to establish a communication connection with a vehicle device, the vehicle device comprising at least one camera facing a blind area of the vehicle, the blind area being an external area that the wearer cannot observe due to an occlusion;

[0017] a first obtaining module configured to obtain auxiliary data through the communication connection, the auxiliary data being obtained based on an environmental picture captured by the at least one camera;

[0018] a first display module configured to display a glasses picture with see-through content based on the auxiliary data, the see-through content being an environmental picture of the blind area presented in a picture captured by the electronic glasses after performing see-through display on the occlusion.

[0019] According to an aspect of the present application, a picture capturing device is provided, the device being performed by a vehicle device, the device having a first communication connection with at least one camera facing a blind area of a vehicle, the blind area being an external area that a wearer cannot observe due to an occlusion, the device comprising:

[0020] a second communication module, configured to establish a second communication connection with the electronic glasses, the electronic glasses being worn by the wearer inside the vehicle;

[0021] a first determination module, configured to determine auxiliary data based on an environment picture collected by the at least one camera, the environment picture being obtained based on the first communication connection, and the auxiliary data being used to cause the electronic glasses to display a glasses picture with perspective content, the perspective content being an environment picture of a blind area presented after performing perspective display on the occlusion in a picture collected by the electronic glasses;

[0022] a second sending module, configured to send the auxiliary data to the electronic glasses through the second communication connection.

[0023] According to an aspect of the present application, an electronic glasses is provided, comprising a processor and a memory, the memory storing at least one program; the processor is configured to execute the at least one program in the memory to implement the above-mentioned auxiliary driving method.

[0024] According to an aspect of the present application, a vehicle device is provided, comprising a processor and a memory, the memory storing at least one program; the processor is configured to execute the at least one program in the memory to implement the above-mentioned picture collection method.

[0025] According to an aspect of the present application, a computer readable storage medium is provided, the computer readable storage medium storing executable instructions, the executable instructions being loaded and executed by a processor to implement the above-mentioned auxiliary driving method and / or picture collection method.

[0026] According to an aspect of the present application, a computer program product is provided, the computer program product comprising computer instructions, the computer instructions being stored in a computer readable storage medium, and a processor reading and executing the computer instructions from the computer readable storage medium to implement the above-mentioned auxiliary driving method and / or picture collection method.

[0027] The technical scheme provided by the present application has at least the following beneficial effects:

[0028] The electronic glasses obtain auxiliary data determined based on an environment picture collected by at least one camera based on the communication connection with the carrier device, and display a glasses picture with perspective content according to the auxiliary data, so that the wearer of the electronic glasses can directly see the environment picture of the blind area observed through the obstruction, that is, the wearer can observe the blind area that cannot be observed due to the obstruction, and the traffic accidents caused by the blind area in the related art can be avoided as much as possible, thereby improving the safety during driving the vehicle. And the specific definition of the perspective content is shown, that is, the obstruction (that is, the obstruction in front of the wearer after wearing the electronic glasses) displayed in the picture collected by the electronic glasses is displayed in a perspective manner, thereby presenting the environment picture of the blind area blocked by the obstruction, realizing the perspective of the obstruction, directly presenting the environment picture of the blind area, and enabling the wearer to drive the vehicle according to the glasses picture with perspective content displayed in the electronic glasses, thereby improving the driving safety and reducing the incidence of traffic accidents caused by the blind area as much as possible. BRIEF DESCRIPTION OF DRAWINGS

[0029] FIG. 1 shows a schematic diagram of a vehicle blind area in the related art;

[0030] FIG. 2 shows a schematic diagram of a car structure in the related art;

[0031] FIG. 3 shows a schematic diagram of a vehicle blind area of a large car in the related art;

[0032] FIG. 4 shows a schematic diagram of a vehicle blind area of a large car in the related art;

[0033] FIG. 5 shows a schematic diagram of a passing blind area in the related art;

[0034] FIG. 6 shows a schematic diagram of an inner wheel difference blind area in the related art;

[0035] FIG. 7 shows an architecture diagram of a computer system according to an example embodiment of the present application;

[0036] FIG. 8 shows a flowchart of an auxiliary driving method according to an example embodiment of the present application;

[0037] FIG. 9 shows a schematic diagram of an auxiliary driving method according to an example embodiment of the present application;

[0038] FIG. 10 shows a schematic diagram of a visual range according to an example embodiment of the present application;

[0039] FIG. 11 shows a schematic diagram of an auxiliary driving method according to an example embodiment of the present application;

[0040] FIG. 12 shows a flowchart of an auxiliary driving method according to an example embodiment of the present application;

[0041] FIG. 13 shows a flowchart of an assisted driving method according to an example embodiment of the present application;

[0042] FIG. 14 shows a schematic diagram of zoomed content according to an example embodiment of the present application;

[0043] FIG. 15 shows a schematic diagram of a rearview mirror content and a rearview mirror extended content according to an example embodiment of the present application;

[0044] FIG. 16 shows a flowchart of an assisted driving method according to an example embodiment of the present application;

[0045] FIG. 17 shows a schematic diagram of a rearview mirror content and a rearview mirror extended content according to an example embodiment of the present application;

[0046] FIG. 18 shows a flowchart of an assisted driving method according to an example embodiment of the present application;

[0047] FIG. 19 shows a schematic diagram of a glasses view with a virtual warning element according to an example embodiment of the present application;

[0048] FIG. 20 shows a flowchart of an assisted driving method according to an example embodiment of the present application;

[0049] FIG. 21 shows a schematic diagram of a preserved view content according to an example embodiment of the present application;

[0050] FIG. 22 shows a flowchart of a view acquisition method according to an example embodiment of the present application;

[0051] FIG. 23 shows a flowchart of an assisted driving method and a view acquisition method according to an example embodiment of the present application;

[0052] FIG. 24 shows a schematic diagram of a visual range of a human eye according to an example embodiment of the present application;

[0053] FIG. 25 shows a schematic diagram of a hierarchy of a glasses view according to an example embodiment of the present application;

[0054] FIG. 26 shows a flowchart of an assisted driving method and a view acquisition method according to an example embodiment of the present application;

[0055] FIG. 27 shows a flowchart of an assisted driving method and a view acquisition method according to an example embodiment of the present application;

[0056] FIG. 28 shows a block diagram of an assisted driving device according to an example embodiment of the present application;

[0057] FIG. 29 shows a structural block diagram of a picture acquisition device according to an example embodiment of the present application;

[0058] FIG. 30 shows a structural schematic diagram of an electronic glasses according to an example embodiment of the present application;

[0059] FIG. 31 shows a structural schematic diagram of a vehicle device according to an example embodiment of the present application. DETAILED DESCRIPTION

[0060] It should be noted that the user information (including but not limited to user device information, user personal information, vehicle driving information, image information collected by the camera, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the biometric identification operation, image acquisition operation, human-computer interaction identification operation and other information involved in the present application are obtained under full authorization. It should be understood that although the terms first, second, etc. may be adopted in the present disclosure to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, the first parameter can also be referred to as the second parameter, and similarly, the second parameter can also be referred to as the first parameter without departing from the scope of the present disclosure. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon determining" or "in response to determining".

[0061] Firstly, the related terms involved in the present application are introduced:

[0062] Vehicle blind area: refers to the area that cannot be directly observed by the driver when sitting on the driving seat due to the obstruction of the vehicle body or road barrier. Next, the classification of vehicle blind area is introduced in detail.

[0063] (1) Front blind area: the area in front of the vehicle that cannot be observed from the driving seat. For a family car, it is usually the area that cannot be seen from the front of the engine cover. As shown in the front blind area 10 in FIG. 1. The size of the front blind area is usually affected by the height of the vehicle body, the height of the seat, the length of the vehicle head (the length of the engine cover), and the driver's body shape.

[0064] (2) Rear blind area: the area behind the vehicle that cannot be observed through the rearview mirror in the driving seat. In addition to the range difference caused by the position of the rearview mirror and the curvature of the mirror surface, the rear blind area is also affected by the obstruction of the rear windshield. The general shape of the rear blind area is shown in the rear blind area 11 in FIG. 1. In related technologies, a reversing radar is usually used to eliminate the rear blind area when reversing.

[0065] (3) Blind area of rearview mirror: the area that cannot be observed by the outside rearview mirror on the driver's seat. The outside rearview mirror includes the left rearview mirror and the right rearview mirror. As shown in the left side rearview mirror blind area 12 and the right side rearview mirror 13 in FIG. 1.

[0066] (4) Blind area of shelter: the area that cannot be observed on the driver's seat due to the shelter of the body pillar. The body pillar is a component for connecting the vehicle body and the roof. According to the body structure, as shown in FIG. 2, the body pillar usually includes the A pillar 20 between the front windshield and the front door, the B pillar 21 between the front door and the rear door, and the C pillar 22 between the rear windshield and the rear door. Here, a common four-seat car is taken as an example, and for a six-seat car or an eight-seat car, several more groups of body pillars can be additionally provided. The positions and names of the body pillars provided by the present application are not limited. The shelter blind area caused by the ABC pillars is shown in the left side A pillar blind area 14, the right side A pillar blind area 15, the left side B pillar blind area 16, the right side B pillar blind area 17, the left side C pillar blind area 18, and the right side C pillar blind area 19 in FIG. 1. The shelter blind area can also be referred to as the pillar blind area, the AB pillar blind area, and the like.

[0067] The above vehicle blind area is taken as an example of the car that is most contacted by ordinary people in daily life. However, due to the differences in the body structure and the size of the vehicle body, the vehicle blind area of large vehicles such as trucks, buses, and semitrailers is different from that of the car. Here, the vehicle blind area of the large vehicle is introduced separately.

[0068] (5) Front blind area: the area that cannot be observed on the driver's seat in front of the vehicle. The area can also be referred to as the vehicle head blind area, which is similar to the vehicle head blind area of the car introduced above, and will not be described here again. The range of the vehicle head blind area is shown in the C area in FIG. 3. When there is a pedestrian or a road barrier in the vehicle head blind area, the driver is almost unable to observe the pedestrian or the road barrier in the vehicle head blind area, as shown in the (1) part in FIG. 4. The size of the vehicle head blind area also depends on the height of the corresponding obstacle.

[0069] (6) Left side blind area: the area that cannot be observed on the driver's seat on the left side of the driver's seat. The range of the left side blind area is shown in the B area in FIG. 3. Since the driver's seat of the large vehicle is usually higher than that of the small vehicle, the corresponding left side blind area and right side blind area are also larger, as shown in the (2) part in FIG. 4.

[0070] (7) Right side blind area: the area that cannot be observed on the driver's seat on the right side of the driver's seat. The right side blind area is shown in the A area in FIG. 3.

[0071] (8) Occlusion blind area: the area that cannot be observed due to the occlusion of the vertical beam of the vehicle body at the driving position. The occlusion blind area of a large vehicle is similar to that of a small vehicle, but for a truck or a semi-trailer, there is usually only an A-pillar blind area. The occlusion blind area is shown as E1 area (i.e., left A-pillar blind area) and E2 area (i.e., right A-pillar blind area) in FIG. 3.

[0072] (9) Rearview mirror blind area: the area that cannot be observed through the outside rearview mirror at the driving position. The rearview mirror blind area is shown as D1 area (i.e., left rearview mirror blind area) and D2 area (i.e., right rearview mirror blind area) in FIG. 3.

[0073] (10) Rear blind area: the area that cannot be observed through the rearview mirror in the vehicle at the driving position. The rear blind area is shown as D3 area in FIG. 3.

[0074] It should be noted that the blind area range shown in the above FIG. 3 is only illustrative, and in an actual driving scenario, the area that cannot be observed by the driver can be greater than or less than the shown blind area range, which is not limited in the embodiments of the present application.

[0075] In addition, during vehicle driving, the following vehicle blind areas can also be formed due to the vehicle body structure or the driving road conditions.

[0076] (11) Overtaking blind area: the area that cannot be observed due to the occlusion of the preceding vehicle during overtaking. As shown in FIG. 5, there are four vehicles driving on the road, which are vehicle 31, vehicle 32, vehicle 33 and vehicle 34. The driving directions of vehicle 31, vehicle 32 and vehicle 33 are the same, and the driving direction of vehicle 34 is opposite to that of vehicle 31, vehicle 32 and vehicle 33. The line-of-sight range of the driver of vehicle 31 facing forward during driving is line-of-sight range 35. When vehicle 31 needs to overtake vehicle 32, the driver of vehicle 31 cannot observe vehicle 33 and vehicle 34 due to the occlusion of vehicle 32, i.e., vehicle 33 and vehicle 34 are located in the overtaking blind area of vehicle 31.

[0077] (12) Inner wheel difference blind area: an area where the driver cannot correctly determine the moving position of the vehicle due to the existence of the inner wheel difference when the vehicle is turning. The inner wheel difference refers to the difference between the turning radius of the front inner wheel and the turning radius of the rear inner wheel when the vehicle is turning. The inner wheel difference blind area is usually caused by the fact that the driver has difficulty in estimating the inner wheel difference with the naked eye or estimating the turning path of the rear inner wheel. As shown in FIG. 6, when the vehicle 36 is turning, the corresponding right front wheel trajectory is trajectory 37, the left front wheel trajectory is trajectory 38, and the right rear wheel trajectory is trajectory 39; the inner wheel difference blind area 40 is formed between the right front wheel trajectory 37 and the right rear wheel trajectory 39, and the driver of the vehicle 36 usually has difficulty in observing the pedestrians or vehicles in the inner wheel difference blind area 40 or realizing the existence of the inner wheel difference blind area 40 (i.e., it is difficult to determine whether the right rear wheel will cause damage to the pedestrians or vehicles in the inner wheel difference blind area 40 during right turning).

[0078] (13) Mountain road blind area: an area on a mountain road that cannot be observed due to the obstruction of roadside rocks. The mountain road blind area usually occurs on a mountain road at a turning section, and the driver cannot observe the road conditions on the other side of the rocks due to the obstruction of the rocks.

[0079] (14) Light blind area: an area that cannot be observed due to light changes. For example, when exiting a tunnel, the driver's eyes may be suddenly darkened from a relatively dark light environment to a relatively bright light environment, or when entering a tunnel, the driver's eyes may be suddenly darkened from a relatively bright light environment to a relatively dark light environment, or the driver's eyes may be temporarily darkened due to various strong light stimuli (such as the high beam of an oncoming vehicle) during night driving.

[0080] Three-dimensional reconstruction: a method of modeling an object to obtain a virtual three-dimensional model of the object. The virtual three-dimensional model is a computer-readable model, which can be understood as a mathematical model that can be represented and processed by a computer. Three-dimensional reconstruction is the basis for using a computer to analyze objects in the real world. In the field of computer vision, three-dimensional reconstruction specifically refers to the process of reconstructing original three-dimensional information from a single view or multiple views. A single view refers to a photograph taken by a camera at a fixed angle for a to-be-reconstructed object; multiple views refer to photographs taken by multiple cameras at different angles for a to-be-reconstructed object. Generally speaking, the reconstruction effect of the multiple view method is better, but the time required for reconstruction of the multiple view method is longer than that of the single view method. Three-dimensional reconstruction technology has been widely applied in image entertainment, virtual fitting, smart home, cultural relic reconstruction, AR (Augmented Reality) tourism, autonomous driving, and large-scene three-dimensional reconstruction.

[0081] With the development of three-dimensional reconstruction technology, in addition to studying how to extract three-dimensional information from two-dimensional information, three-dimensional reconstruction methods have gradually shifted to extracting two-dimensional information from the three-dimensional information obtained by reconstruction, that is, new view generation. In general, it is to perform three-dimensional reconstruction on an object based on images from partial views to obtain a virtual three-dimensional model of the object, i.e., a virtual object; and to construct a virtual camera to collect images of the virtual object from the desired new view to obtain images under the new view.

[0082] Three-dimensional engine: a software framework or core component that has been written to create and render three-dimensional models. The software framework provides designers of three-dimensional models with various tools for creating and rendering three-dimensional models, with the purpose of enabling designers to quickly produce three-dimensional models without understanding how to obtain three-dimensional models from computer languages. A three-dimensional engine includes the following systems: a rendering engine (i.e., a "renderer", including a two-dimensional image engine and a three-dimensional image engine), a physics engine, a collision detection system, sound effects, a script engine, computer animation, artificial intelligence, a network engine, and scene management.

[0083] Rendering engine: in the field of image technology, a rendering engine refers to rendering a three-dimensional model of a virtual object that has been modeled into a two-dimensional image, so that the three-dimensional model still maintains the stereoscopic effect in the two-dimensional image. Usually, after the model data of the three-dimensional model that has been modeled is imported into the rendering engine, the rendering engine drives the rendering pipeline in the GPU (Graphics Processing Unit, image processor) to perform rendering, thereby visualizing the object indicated by the three-dimensional model on the display screen of the game device.

[0084] Unity: one of the most widely used three-dimensional engines. It provides powerful tools and functions that enable developers to create realistic three-dimensional environments and interactive experiences. Here are some of the main applications of Unity in scene virtualization:

[0085] (1) Virtual Reality (VR) and Augmented Reality: Unity supports the creation of immersive VR and AR experiences, allowing users to enter virtual worlds through headsets or smartphones and other devices. In these applications, Unity's 3D rendering capabilities and interactive design tools enable developers to design a variety of virtual scenes, such as games, educational simulations, training scenarios, etc.

[0086] (2) Digital Twin: Digital Twin refers to creating a virtual copy of a physical object or system to simulate, analyze, and optimize in a virtual environment. Unity can be used to build digital twin models, such as buildings, factories, cities, etc., which can be used for planning, monitoring, and maintenance.

[0087] (3) Architecture Visualization: In the field of architecture and urban planning, Unity can be used to create three-dimensional visualizations of architectural projects. This not only helps designers and clients better understand the design intent but can also be used for marketing and presentations.

[0088] (4) Game Development: Unity is one of the preferred platforms for game developers, offering a rich asset library, physics engine, and animation tools that allow developers to quickly build and iterate on game scenes.

[0089] (5) Education and Training: Unity can be used to create educational software and training simulations, such as medical simulations, flight simulators, etc. These virtual scenarios provide a safe environment for users to learn and practice without actual risk.

[0090] (6) Industrial Design and Simulation: In the industrial field, Unity can be used to simulate production lines, mechanical operations, etc., helping engineers test and optimize designs during the design phase.

[0091] Image Segmentation: The process of dividing an image into multiple segments. Depending on the granularity of its segmentation or the granularity of the segments, it can be divided into semantic segmentation, instance segmentation, and part segmentation.

[0092] (1) Semantic Segmentation: Semantic segmentation is a pixel-level image segmentation that assigns a class to each pixel in a given image, resulting in a semantic segmentation map. In a semantic segmentation map, the image is divided into several non-intersecting regions, each composed of pixels of the same class, with pixels within the same region exhibiting consistency or similarity and pixels between different regions exhibiting significant differences. For example, separating a target object from the background in an image containing the target object is a semantic segmentation of the target object. The goal of semantic segmentation is to understand the content of an image at the pixel level and assign an object class to each pixel. For example, an image containing people, vehicles, and buildings can be segmented into multiple regions, each representing a specific object.

[0093] (2) Instance Segmentation: Dividing the image into regions corresponding to specific instances. Instance segmentation corresponds to target regions such as a specific car or a specific building.

[0094] (3) Part Segmentation: Dividing the image into regions corresponding to parts of instances. Part segmentation corresponds to target regions such as the doors, body, and wheels of a car.

[0095] Object Detection: The process of automatically locating and identifying specific objects in an image or video.

[0096] Target recognition: The process of identifying a specific target in a series of images or video frames.

[0097] Image composition: Cutting the foreground of one picture and pasting it onto another background picture to get a composite image.

[0098] Transform matrix: A mathematical tool that represents the position, orientation, and size changes of a graphical object in two-dimensional or three-dimensional space. In computer graphics, Transform matrix, Position, Rotation, and Scale are closely related. Transform matrix is composed of these three factors.

[0099] Position: Represents the position of an object in 3D space. Position is a 3D vector containing x, y, z components.

[0100] Rotation: Represents the rotation of an object in 3D space. Rotation is a quaternion containing four components: w, x, y, z. Quaternion can represent rotation in 3D space, which can avoid gimbal lock problem and is smoother in interpolation.

[0101] Scale: Represents the size of an object in 3D space. Scale is a 3D vector containing x, y, z components.

[0102] Transform matrix is composed of these three factors. It is a 4 × 4 matrix that can represent translation, rotation, scaling and other transformations. The structure of Transform matrix is as follows: Transform = Position × Rotation × Scale.

[0103] Electronic glasses: Glasses with independent operating system, controlled by voice or action to realize different functions. Electronic glasses are similar to smart phones that can install different function applications, and electronic glasses usually also have Bluetooth connection, wireless network connection and other communication connection methods to support smart glasses and other electronic devices communication. Electronic glasses can also be called smart glasses.

[0104] VR: also known as virtual reality or virtual reality technology, is a computer simulation system that can create and experience virtual environments. VR technology includes computer, electronic information, simulation technology, its basic implementation is to use computer technology as the main, using and integrating three-dimensional graphics technology, multimedia technology, simulation technology, display technology, servo technology and other high-tech latest development achievements, with the help of computer and other equipment to produce a realistic three-dimensional visual, tactile, olfactory and other sensory experience of virtual environment, so as to combine virtual and reality, make people in virtual environment produce a sense of being there.

[0105] AR: a kind of virtual information and real world cleverly integrated technology, widely used in multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, sensing and other technical means, after simulation of computer generated text, image, three-dimensional model, music, video and other virtual information, applied to the real world, two kinds of information complement each other, so as to realize the "enhancement" of the real world.

[0106] MR(Mixed Reality, mixed reality): MR technology is the further development of VR technology, MR technology through the virtual scene in the real scene information, in the real world, virtual world and user between the interactive feedback of information loop, in order to enhance the user experience of real sense.

[0107] Transportation: refers to all kinds of man-made devices for human transportation. Including cars, ships, aircraft and other widely used transportation tools, also including low altitude manned aircraft, unmanned manned aircraft and other still in development of transportation tools, as well as submarines, rockets, manned satellites and other unusual transportation tools.

[0108] Communication connection: the connection state of information transmission and exchange between multiple communication devices through a certain way(wired or wireless). According to the signal transmission mode, it can be roughly divided into two categories: wireless communication connection and wired communication connection.

[0109] (1) Wireless communication connection: No need for physical line connection, through electromagnetic wave in space to realize the communication connection of information transmission. Wireless communication connection technology is various, each has its own characteristics, such as WiFi (Wireless Fidelity), Bluetooth, mobile communication network, satellite communication, V2X (Vehicle to X, car to other equipment, X can represent anything) and so on. WiFi is a technology that allows electronic devices to connect to a wireless local area network (WLAN), widely used in home, office and other scenarios, providing high-speed Internet access. Bluetooth is a short-range wireless communication technology used to connect and transmit data between devices such as mobile phones, headphones, computers, etc. Mobile communication network includes 2G, 3G, 4G, 5G, etc., through base station and mobile device (such as mobile phone, tablet computer) communication, realize voice call and data transmission. Satellite communication refers to using artificial satellites as relay stations to forward radio signals to realize communication between different places on earth. V2X is a communication technology that enables vehicles to exchange data with various entities in the surrounding environment, V2X can include: Vehicle to Vehicle (V2V) communication, Vehicle to Infrastructure (V2I) communication, Vehicle to Pedestrian (V2P) communication or Vehicle to Network (V2N) communication.

[0110] (2) Wired communication connection: communication connection that realizes information transmission through physical lines. Wired communication connection technology is usually more stable and faster in transmission speed, but is limited by the physical length and layout of the line. Wired communication connection technologies include Ethernet connection, optical fiber communication, serial port connection, USB (Universal Serial Bus) connection, CAN (Controller Area Network) bus, etc. Ethernet connection is a local area network communication technology based on IEEE (Institute of Electrical and Electronics Engineers) 802.3 standard, widely used in office, factory and other scenes, providing high-speed, stable data transmission. Optical fiber communication is the use of light waves to transmit information in optical fibers, with the advantages of fast transmission speed, large capacity and strong anti-interference ability. Serial port connection is a commonly used serial communication interface standard, used to connect computers and external devices (such as printers, modems, etc.). USB connection is a communication interface widely used between computers and peripherals, with the characteristics of plug and play, high-speed transmission, etc. CAN bus is a serial communication network that effectively supports distributed control systems, commonly used in the automotive field, responsible for communication between internal components of the vehicle.

[0111] Human-computer interaction method: the method of information exchange and operation control between human and computer or other intelligent devices through certain ways. There are many corresponding interaction methods, including: mouse and keyboard input, touch screen input, gesture interaction, eye movement interaction, voice interaction, brain-computer interface, etc. Among them, mouse and keyboard input and touch screen input are widely used input methods, but the adaptability of these two interaction methods to electronic glasses is low, which will not be described here. The remaining several interaction methods are suitable for electronic glasses, and the application of these several interaction methods in electronic glasses will be briefly introduced.

[0112] (1) Gesture interaction refers to the interaction between the user and the electronic glasses through hand movements, such as gesture recognition technology that can recognize the user's hand gestures such as waving, pinching, rotating, etc., and convert them into corresponding instructions to change the content of the picture displayed by the electronic glasses.

[0113] (2) Eye movement interaction refers to realizing human-computer interaction by collecting the user's eye movement, such as the user can select or control the electronic glasses to perform corresponding operations by staring at a specific area after wearing electronic glasses.

[0114] (3) Voice interaction refers to the interaction between the user and the electronic glasses through speaking, and the electronic glasses understand the user's intention through voice recognition technology and make corresponding responses (such as displaying corresponding picture content). Voice interaction is also a widely used way at present.

[0115] (4) Brain-Computer / Machine Interface (BCI / BMI) refers to the realization of information exchange and operation control between a person and electronic glasses through direct reading and decoding of electrical signals of the brain. Brain-Computer / Machine Interface is a kind of human-computer interaction mode that has not been widely used but has great potential.

[0116] FIG. 7 shows an architecture diagram of a computer system provided by an example embodiment of the present application. The computer system includes the electronic glasses 110 and the carrier device 120.

[0117] The electronic glasses 110, which can also be referred to as smart glasses, wearable devices, portable devices, etc., can be VR devices, AR devices, or MR devices, etc. The electronic glasses 110 can be divided into two categories: one is electronic glasses with a transparent display screen (which can be referred to as penetrating electronic glasses), and the other is electronic glasses with an opaque display screen (which can be referred to as non-penetrating electronic glasses). The embodiments of the present application do not limit this. The electronic glasses 110 are used to display a glasses picture with see-through content based on auxiliary data; or in other words, the electronic glasses 110 are used to enable a wearer to observe a glasses picture with see-through content based on auxiliary data.

[0118] The carrier device 120 at least includes at least one camera facing the blind area of the vehicle 130. The at least one camera in the carrier device 120 is used to collect an environment picture, and the auxiliary data is related to the environment picture collected by the at least one camera.

[0119] Optionally, the carrier device 120 further includes a control terminal of the vehicle 130, which can be an electronic device such as a mobile phone, a tablet computer, a vehicle terminal (car machine), a wearable device, a PC (Personal Computer), an unmanned order terminal, a computer installed with a ship main engine control system, etc. The vehicle 130 can be the above-mentioned car (including large cars and small cars, etc.), ship, airplane, submarine, low-altitude manned aircraft, unmanned manned aircraft, etc. The embodiments of the present application do not limit this, and the control terminal is used to establish a communication connection with the at least one camera and the electronic glasses 110.

[0120] For example, the at least one camera 1 is used to collect an environment picture 2, which is processed by the electronic glasses 110 or the control terminal in the carrier device 120 to obtain the auxiliary data 3. Before the electronic glasses 110 obtains the auxiliary data 3, the electronic glasses 110 can display a glasses picture 4 without perspective content, in which there is a blind area caused by the blocking of the vehicle body 6, which refers to the external area of the vehicle that the wearer cannot observe due to the blocking of the blocking object (vehicle body); after the electronic glasses 110 obtains the auxiliary data 3, the glasses picture 5 with perspective content can be displayed based on the auxiliary data 3, and the perspective content is as shown in the area 7, that is, the perspective display processing is performed on the vehicle body 6 to display the entire vehicle body 6 as semi-transparent, so that the wearer can observe the objects that are originally blocked by the vehicle body 6, so that the wearer can observe the ball 8 originally located in the blind area. The glasses picture with perspective content enables the wearer to timely notice the sudden situation in the blind area.

[0121] FIG. 8 shows a flowchart of an auxiliary driving method according to an example embodiment of the present application. The method is performed by the electronic glasses worn by the wearer inside the vehicle, which can be the electronic glasses in the computer system described above. The method includes at least part of steps 210, 220 and 230:

[0122] Step 210: Establish a communication connection with a carrier device, the carrier device including at least one camera facing a blind area of the vehicle, the blind area referring to an external area that the wearer cannot observe due to the blocking of the blocking object.

[0123] The carrier device includes at least one camera, and the carrier device is configured to collect an environment picture based on the at least one camera. Specifically, the carrier device includes at least one camera facing a blind area of the vehicle, the blind area referring to an external area of the vehicle that the wearer cannot observe due to the blocking of the blocking object; wherein the wearer is located inside the vehicle, and the blocking object is at least one of a roadblock, a component of the vehicle, a passenger and an article located inside the vehicle. It can be understood that the blind area refers to an external area of the vehicle that the wearer located inside the vehicle cannot observe due to the blocking of the blocking object. The external area of the vehicle refers to the outside of the entire vehicle. For example, the vehicle is a car, and the external area of the vehicle refers to the area outside the car frame; the vehicle is a ship, and the external area of the vehicle refers to the area outside the entire ship frame; the vehicle is an airplane, and the external area of the vehicle refers to the area outside the entire airplane frame.

[0124] Generally, the wearer is located at the driving seat, and the "vehicle blind area" mentioned above refers to an area outside the vehicle that cannot be directly observed by the wearer at the driving seat due to the obstruction of the vehicle body (i.e., a component of the vehicle) or a roadblock. However, the embodiments of the present application do not limit the position of the wearer inside the vehicle, i.e., for the automobile driving scenario, the wearer can also be a passenger located at the co-driver seat or in the back row. That is, the blind area in the embodiments of the present application is for any wearer located inside the vehicle, not only for the driver (i.e., the wearer at the driving seat); for example, for automobile driving, when the wearer is located at the co-driver seat, the blind area refers to the direction of the line of sight of the wearer. In general, when different wearers (height, proportion of upper body and lower body, etc.) are located at the same position or the same wearer is located at different positions, the areas that they cannot observe may have certain differences, i.e., their blind areas (position, range, etc.) have certain differences.

[0125] The at least one camera is a camera facing the blind area of the vehicle, i.e., the at least one camera is used to capture the environmental picture of the blind area of the vehicle; it can also be said that the at least one camera is used to capture the environmental picture of the blind area corresponding to the wearer. In fact, the environmental picture captured by the at least one camera should be an environmental picture including the blind area of the vehicle or the blind area corresponding to the wearer, i.e., the range of the picture that can be captured by the at least one camera is greater than the range of the blind area.

[0126] In some embodiments, the at least one camera corresponds to at least one blind area. For example, one camera corresponds to one blind area, or one camera corresponds to multiple blind areas, multiple cameras correspond to one blind area, or multiple cameras correspond to multiple blind areas. For example, if discontinuous (or disjoint blind areas) are regarded as a single blind area, one camera corresponds to one blind area, or one camera corresponds to multiple blind areas, or multiple cameras correspond to one blind area. Taking the above-mentioned vehicle blind area as an example, the camera located at the front of the vehicle can capture the environmental picture of the front blind area and the obstruction blind area for any wearer; the camera located on the left side mirror and the camera located on the right side mirror can capture the environmental picture of the side mirror blind area; the camera located at the rear of the vehicle can capture the environmental picture of the tail blind area, etc.

[0127] Optionally, the at least one camera includes at least one of a camera configured when the vehicle is manufactured, a camera added to the vehicle by a user, and a camera located in a driving environment. For example, when the at least one camera is located on the vehicle, the at least one camera can be located at a position near a blind area of the vehicle, or located at any position of the vehicle and facing the blind area of the vehicle. For example, the vehicle is a car, and in order to eliminate the above-mentioned blind area of the vehicle, the camera can be installed at the front of the car, on the left and right rearview mirrors, on the AB column, on the vehicle base, etc.; for the blind area of the vehicle during driving, the camera can be a camera installed on a road facility (such as a street lamp, a telegraph pole, etc.), another car, which is not limited in the embodiments of the present application.

[0128] The vehicle includes a car, a ship, an airplane, and a driving environment is an environment in which the vehicle travels, for example, for a car as a vehicle, a corresponding driving environment is usually a road; for a ship as a vehicle, a corresponding driving environment is usually a waterway; for an airplane, a manned aircraft, and the like as a vehicle, a corresponding driving environment is usually an airway.

[0129] The electronic glasses can establish a communication connection with the vehicle device. Optionally, the communication connection established by the electronic glasses with the vehicle device includes at least one of a wireless communication connection and a wired communication connection; and a connection mode of the communication connection established by the electronic glasses with the vehicle device includes at least one of a direct connection and an indirect connection, the direct connection means that the communication connection between the electronic glasses and the vehicle device does not pass through other communication devices, and the indirect connection means that the communication connection between the electronic glasses and the vehicle device passes through the relay of other communication devices.

[0130] Taking a car as an example, the carrier device includes at least one camera facing the blind area of the vehicle, the communication connection between the at least one camera and the electronic glasses is a wireless communication connection, and the connection mode is direct connection; or, the carrier device includes at least one camera facing the blind area of the vehicle, the communication connection between the at least one camera and the vehicle terminal is a wired communication connection, the communication connection between the vehicle terminal and the electronic glasses is a wireless communication connection, at this time, the communication connection established between the electronic glasses and the carrier device includes a wireless communication connection and a wired communication connection, the connection mode is indirect connection, and the vehicle terminal acts as a relay device; or, the at least one camera is installed on the car, the communication connection between the at least one camera and the vehicle terminal is a wired communication connection, such as a CAN bus, a USB connection, etc., and the communication connection between the vehicle terminal and the electronic glasses is a wireless communication connection; or, part of the cameras are installed on the car, the connection between the part of the cameras and the vehicle terminal is a wired communication connection, and the other part of the cameras are installed on road facilities or other cars, the communication connection between the other part of the cameras and the vehicle terminal is a wireless connection, such as a Bluetooth connection, a V2X connection, etc. The embodiments of the present application do not limit this.

[0131] Step 220: acquiring auxiliary data through the communication connection, the auxiliary data being obtained based on the environment picture collected by the at least one camera.

[0132] The auxiliary data is data used to assist the electronic glasses to display the environment picture with the perspective content. The auxiliary data is obtained based on the environment picture collected by the at least one camera. It can be understood that the auxiliary data is data obtained by encapsulating the environment picture collected by the at least one camera; or, the auxiliary data is data obtained by digitizing the environment picture collected by the at least one camera; or, the auxiliary data is obtained based on the processing of the environment picture collected by the at least one camera, for example, the auxiliary data is data obtained by digitizing the glasses picture with the perspective content, or the auxiliary data is data obtained by encapsulating the environment picture of the blind area, etc.

[0133] For example, the auxiliary data acquired by the electronic glasses at the current moment is determined based on the environment picture collected by the at least one camera at the current moment; or, the auxiliary data acquired by the electronic glasses at the current moment is determined based on the environment picture collected by the at least one camera in the past n seconds, n being a positive integer. Optionally, the value of n is determined based on at least one of the driving speed of the vehicle, the road condition, and the driving scene. For example, in the scenario of high-speed driving (e.g., the driving speed of the vehicle is higher than 100 kilometers per hour), the value of n is small, such as 0.01; in the scenario of low-speed driving (e.g., the driving speed of the vehicle is lower than 10 kilometers per hour), the value of n is large, such as 5; in the scenario of not driving, the value of n can be larger, such as 10 or even 60; in the scenario of complex road conditions (e.g., traffic jam, continuous sharp turns, etc.), the value of n is small; in the scenario of simple road conditions, the value of n is large; in the scenario of frequent accidents (e.g., starting, reversing, etc.), the value of n is small. It can be understood that the smaller the value of n is, the higher the timeliness of the auxiliary data is.

[0134] Optionally, the environment picture collected by the at least one camera is at least one of a picture, a video, and a video frame.

[0135] Step 230: displaying a glasses picture with perspective content based on the auxiliary data, the perspective content being an environment picture of a blind area presented after the blind area is displayed in a perspective manner in the picture collected by the electronic glasses.

[0136] The environment picture of the blind area presented in the perspective content is an environment picture corresponding to all or part of the blind area of the vehicle. For example, the environment picture of the blind area presented in the perspective content is a combined picture or a fused picture of the environment picture corresponding to all or part of the blind area of the vehicle, or an environment picture corresponding to each of the blind area. The part of the blind area refers to a blind area selected by the wearer to be displayed in a perspective manner, or a blind area set by the developer to be displayed in a perspective manner, or a blind area existing in the picture collected by the electronic glasses.

[0137] The environment picture of the blind area is determined based on the environment picture collected by the at least one camera. Optionally, the environment picture of the blind area (or the perspective content) is determined directly based on the environment picture collected by the at least one camera, i.e., the environment picture of the blind area includes part of the environment picture collected by the at least one camera; or, the environment picture of the blind area includes the environment picture collected by part of the cameras; or, the environment picture of the blind area includes part of the environment picture collected by part of the cameras. Or, the environment picture of the blind area is determined after processing the environment picture collected by the at least one camera.

[0138] The eyeglass picture without see-through content is shown in part (1) of FIG. 9, the occlusion 90 causes occlusion to the road, and the wearer cannot observe the road occluded by the occlusion 90, i.e., the picture shown in part (1) of FIG. 9 has a blind area for the wearer; the eyeglass picture with see-through content is shown in part (2) of FIG. 9, the occlusion 90 is completely see-through, and the wearer has a full view of the current driving scene, thereby eliminating the blind area caused by the occlusion 90. In the corresponding scene of FIG. 9, the occlusion 90 refers to the parts of the car and the passengers, i.e., the parts of the car are the entire car body, and the passengers refer to the part of the body of the driver, i.e., the arm of the wearer.

[0139] The eyeglass picture refers to the picture observed by the wearer of the electronic glasses after wearing the electronic glasses. Alternatively, the eyeglass picture refers to the picture displayed on the display screen of the electronic glasses. Alternatively, the picture content contained in the eyeglass picture is set by the user; or, the picture content contained in the eyeglass picture is set by the developer.

[0140] The visual range of the eyeglass picture is the same as or different from the visual range of the direct-view picture, the direct-view picture refers to the picture content observed without wearing the electronic glasses. Alternatively, the visual range of the eyeglass picture is greater than the visual range of the direct-view picture, or the visual range of the eyeglass picture is equal to the visual range of the direct-view picture, or the visual range of the eyeglass picture is less than the visual range of the direct-view picture, which is set according to actual technical needs. For example, as shown in FIG. 10, there are a total of 5 trees in a row, and the direct-view picture observed by the observer 41 without wearing the electronic glasses is picture 42; the eyeglass picture observed by the observer 41 with wearing the electronic glasses can be picture 43 with a visual range greater than the direct-view picture, can be picture 42 with a visual range equal to the direct-view picture, or can be picture 44 with a visual range less than the direct-view picture. It should be noted that the visual range shown in FIG. 10 includes changes in the length and width of the picture, but in fact, the influencing factors of the visual range include at least one of the length, the width, and the picture center. That is, the difference between the visual range of the eyeglass picture and the direct-view picture is reflected in the difference in the length, the width, and / or the picture center.

[0141] In summary, in the method provided by the embodiments of the present application, the electronic glasses obtain auxiliary data determined based on the environment picture collected by the at least one camera based on the communication connection with the carrier device, and display the glasses picture with the perspective content based on the auxiliary data, so that the wearer of the electronic glasses can directly observe the environment picture of the blind area observed through the obstruction, that is, the wearer can observe the blind area that cannot be observed due to the obstruction, thereby improving the safety during driving the vehicle. Moreover, the basic processing manner of the perspective content is shown, that is, the obstruction (the obstruction appearing in front of the wearer after wearing the electronic glasses) displayed in the picture collected by the electronic glasses is displayed in a perspective manner, thereby presenting the environment picture of the blind area blocked by the obstruction, so that the wearer can drive the vehicle according to the glasses picture with the perspective content displayed in the electronic glasses, thereby improving the driving safety and reducing the incidence of traffic accidents caused by the blind area as much as possible. Further, the auxiliary data is obtained based on the environment picture collected by the at least one camera, and the at least one camera is the camera facing the blind area of the vehicle, that is, the manner of eliminating the blind area is shown, that is, the glasses picture with the perspective content (or the perspective content) is indirectly obtained based on the environment picture collected by the at least one camera facing the blind area of the vehicle. In addition, the blind area of the vehicle that can be eliminated by the embodiments of the present application not only includes the blind area caused by the components of the vehicle, but also can eliminate the artificial blind area caused by the passengers, the articles in the vehicle and the like, thereby comprehensively considering the blind area factor during driving the vehicle and realizing the omnidirectional blind area elimination.

[0142] In some embodiments, the perspective content further displays the semi-transparent whole or partial obstruction. That is, when the perspective content is displayed in the obstruction, the obstruction is not fully transparent, but has a certain transparency. In addition, the environment picture of the blind area displayed by the perspective content of the obstruction can also be semi-transparent, that is, the environment picture of the whole or partial blind area displayed in the perspective content is semi-transparent.

[0143] The electronic glasses can be divided into two categories, one is the electronic glasses with a transparent display screen (which can be referred to as a penetrating electronic glasses), and the other is the electronic glasses with an opaque display screen (which can be referred to as a non-penetrating electronic glasses). For the penetrating electronic glasses, the glasses picture with the perspective content means that the perspective content is displayed on the basis of the real scene directly observed by the wearer, that is, the semi-transparent environment picture of the whole or partial blind area is displayed in the position of the obstruction. For the non-penetrating electronic glasses, all the picture content in the glasses picture with the perspective content is processed picture content, therefore, the perspective content can display at least one of the semi-transparent whole obstruction, the semi-transparent partial obstruction, the semi-transparent whole blind area environment picture and the semi-transparent partial blind area environment picture.

[0144] Optionally, the semi-transparent refers to having transparency, but the value of the transparency is not limited. The value of the transparency ranges from greater than 0 to less than 100, and the specific value can be set according to actual technical needs, wherein 0 represents completely opaque, and 100 represents completely transparent. In some embodiments, the transparency corresponding to different occlusions in the see-through content can be the same or different. For example, there are occlusions a1 to a3, the transparency of the occlusion a1 displayed in the see-through content is 25%, the transparency of the occlusion a2 is 50%, and the transparency of the occlusion a3 is 75%. From the display effect, the transparency of the occlusion a1 < the transparency of the occlusion a2 < the transparency of the occlusion a3. The semi-transparent can also refer to the opacity. For example, there are occlusions a1 to a3, the opacity of the occlusion a1 displayed in the see-through content is 25%, the opacity of the occlusion a2 is 50%, and the opacity of the occlusion a3 is 75%. From the display effect, the transparency of the occlusion a1 > the transparency of the occlusion a2 > the transparency of the occlusion a3.

[0145] For example, the see-through content displays all semi-transparent occlusions. As shown in FIG. 9, part (1) of FIG. 9 shows a glasses picture without see-through content, i.e., a picture with a blind area, parts (3) and (4) of FIG. 9 show glasses pictures with see-through content, the see-through content includes all semi-transparent occlusions, and the transparency of the occlusions shown in part (3) of FIG. 9 is higher than that of the occlusions shown in part (4) of FIG. 9.

[0146] Optionally, in addition to the vehicle device including at least one camera, the driving environment includes at least one environmental camera, and a communication connection is established between the at least one environmental camera and the electronic glasses. At this time, the occlusion further includes a roadblock outside the vehicle, and the see-through content further displays a semi-transparent roadblock. For example, as shown in FIG. 11, the vehicle is a car, and when the car is driving on a mountain road with a sharp turn, as shown in part (1) of FIG. 11, part of the body of the object car 70 can be observed, and another part of the body of the object car 70 is occluded by the mountain 71. After wearing the electronic glasses, the glasses picture with see-through content displayed is as shown in part (2) of FIG. 11, the occlusion of the mountain 71 is displayed as a semi-transparent mountain 72 in part (2) of FIG. 11, and the environmental picture of the blind area occluded by the mountain includes the entire body of the object car 70 and the mountain road occluded by the mountain.

[0147] Optionally, the environmental picture of all blind areas displayed in the see-through content is semi-transparent. As shown in part (3) of FIG. 11, relative to part (1) of FIG. 11, the occlusion of the mountain 71 is not transparent, and the object car 70 and the road in the environmental picture of the blind area are transparent.

[0148] In summary, the method provided by the embodiments of the present application displays the semi-transparent occlusion or blind area environment picture in the see-through content, rather than the fully transparent occlusion or blind area picture. On the one hand, the wearer can observe the environment picture of the blind area; on the other hand, the wearer will not make a wrong judgment on the actual driving condition of the vehicle. If the occlusion is fully transparent, as shown in the (2) part of FIG. 9, the wearer may have a blurred feeling about the fact that the vehicle is being driven, which may easily lead to an accident.

[0149] In some embodiments, after wearing the electronic glasses, the glasses picture with see-through content is displayed based on the auxiliary data. However, on the one hand, for a wearer who is not used to the see-through content, the normal display of the see-through content may cause trouble to the driving, such as a wearer who is used to driving a normal car. The glasses picture with see-through content may make him feel that he is driving an open-top car, which may make the wearer difficult to adapt to the driving process of wearing the electronic glasses. Therefore, the glasses picture with see-through content can be displayed in the manner shown in FIG. 12. The see-through content is displayed only when the see-through condition is met. The step 230 can be implemented as step 231.

[0150] Step 231: Display the glasses picture with see-through content based on the auxiliary data when the see-through condition is met. The see-through condition is a condition for triggering the display of the see-through content.

[0151] That is, the electronic glasses detect that the see-through condition is met, and then display the glasses picture with see-through content based on the auxiliary data. When the see-through condition is not met, the glasses picture without see-through content is displayed based on the auxiliary data; or, when the see-through condition is not met, the glasses picture is displayed based on the auxiliary data, and there is no see-through content in the glasses picture.

[0152] Optionally, the see-through condition includes at least one of the following: the wearer has a demand to observe the see-through content; the blind area of the vehicle has a dangerous situation; the traffic scene needs the wearer to observe the blind area of the vehicle; and the auxiliary driving mode is started. The auxiliary driving mode can also be referred to as a see-through driving mode or a see-through mode.

[0153] The judgment of the perspective condition is performed by the electronic glasses or the vehicle. For example, the electronic glasses are configured to collect sensing information of a wearer, and detect whether the wearer has a demand for observing the perspective content based on the sensing information, the sensing information including at least one of eye movement information, gesture information, voice information, and brain wave information. For another example, the electronic glasses are configured to obtain a human-computer interaction instruction issued by the wearer, and detect whether the wearer has a demand for observing the perspective content based on the human-computer interaction instruction, the instruction issued by the wearer including at least one of an eye movement instruction, a voice instruction, and a gesture instruction. Alternatively, the vehicle is configured to identify whether there is a dangerous situation in a blind area of the vehicle, and send an identification result to the electronic glasses. The auxiliary data includes the identification result of the vehicle, and the electronic glasses judge whether the blind area of the vehicle has a dangerous situation based on the identification result of the vehicle. Alternatively, the auxiliary data includes an environmental picture of the blind area, and the electronic glasses judge whether the blind area of the vehicle has a dangerous situation based on the environmental picture of the blind area. Alternatively, the vehicle is configured to identify whether a traffic scene needs the wearer to observe the blind area of the vehicle, and send an identification result to the electronic glasses. The auxiliary data includes the identification result of the vehicle, and the electronic glasses judge whether the traffic scene needs the wearer to observe the blind area of the vehicle based on the identification result of the vehicle. Alternatively, the electronic glasses are configured to detect whether an auxiliary driving mode is enabled.

[0154] The wearer has a demand for observing the perspective content, which can also be referred to as an observation demand of the wearer, an observation habit of the wearer, a perspective habit of the wearer, a perspective preference of the wearer, or an observation preference of the wearer, and the like. The observation habit (which can also be referred to as a perspective habit, a perspective preference, or an observation preference) refers to a driving scene in which the frequency of the wearer's demand for displaying the perspective content reaches a preference threshold. For example, the wearer demands to display the perspective content every time the wearer passes a certain road section while wearing the electronic glasses, and demands not to display the perspective content when leaving the road section. This can be recorded as the observation habit of the wearer, and the electronic glasses automatically display the perspective content every time the wearer passes the road section in the subsequent wearing of the electronic glasses.

[0155] Optionally, the observation habit stored by the electronic glasses is not distinguished according to the identity of the wearer. Alternatively, the electronic glasses store the observation habit corresponding to each wearer, and identify the identity of the wearer after the wearer wears and enables the electronic glasses, and load the observation habit corresponding to the wearer. The identity of the wearer can be identified by using a biological recognition technology such as iris, face, voiceprint, and the like, which is not limited in the embodiments of the present application.

[0156] The dangerous situation is a traffic situation that affects the continuous driving of the vehicle, and can be an obstacle that suddenly appears in the blind area of the vehicle, an obstacle located in the blind area of the vehicle at the start, an obstacle on the predicted moving route, etc. The predicted moving route refers to a moving route predicted according to at least one of the recorded driving habits and the current driving speed, the current deflection angle of the steering wheel.

[0157] The traffic scene needs the wearer to observe the blind area of the vehicle, that is, the current traffic scene is a traffic scene with high perspective demand, or the current traffic scene is a traffic scene with frequent accidents. The traffic scene refers to the scene in which the vehicle is located, and is related to the driving scene of the vehicle, the road surface condition, etc. For example, the traffic scene is a road near a school; or the traffic scene is a mountain road with many sharp turns; or the traffic scene is a narrow road; or the traffic scene is an up-and-down slope, etc.

[0158] Optionally, the perspective condition is set by the wearer; or the perspective condition is fixed; or the perspective condition is intelligently set by the electronic glasses.

[0159] In summary, the method provided by the embodiments of the present application sets a trigger condition, that is, a perspective condition, for the display of the perspective content, so that after the wearer wears the electronic glasses, the perspective content will only be displayed when the perspective condition is met, and if the perspective condition is not met, the perspective content will not be displayed; in the case of not affecting the normal driving of the wearer, the perspective content is displayed only when the perspective condition is met, which can avoid the discomfort caused by the normalized perspective content to the wearer.

[0160] In addition, the setting of the perspective condition considers two aspects. On the one hand, the needs of the user are considered, and the perspective content is displayed only when the wearer expects to observe the perspective content; and the observation habits of the wearer can be recorded, so that the wearer does not need to actively provide indication information to instruct the electronic glasses to display the perspective content, and the electronic glasses can directly display the perspective content according to the observation habits, improving the human-computer interaction efficiency and providing a better experience for the wearer; and the wearer can be more immersed in driving, avoiding the distraction of the wearer in driving to instruct the electronic glasses to display the perspective content; on the other hand, the scene that needs to be perspective in the actual driving scene is also considered, that is, when there is a dangerous situation in the blind area, the perspective content needs to be displayed to the wearer to remind the wearer to pay attention to the dangerous situation.

[0161] Next, the perspective condition will be described in more detail.

[0162] 1. The wearer has a demand to observe the perspective content

[0163] The perspective condition (or the wearer has the need to observe the perspective content) includes at least one of the following:

[0164] · the line of sight of the wearer is towards the occlusion;

[0165] · the head of the wearer is towards the occlusion;

[0166] · the occlusion appears in the glasses view;

[0167] · the wearer requests a human-machine interaction indication to see through the occlusion.

[0168] Next, the above several perspective conditions will be introduced one by one.

[0169] 1.1 The line of sight of the wearer is towards the occlusion

[0170] When the line of sight of the wearer is towards the occlusion, the wearer can be provided with an environment view that eliminates the blind area caused by the occlusion in the direction of the line of sight of the wearer.

[0171] The line of sight of the wearer towards the occlusion includes at least one of the following situations: the fixation target of the wearer is the occlusion; the fixation point of the wearer is on the occlusion; the eye of the wearer is towards the occlusion.

[0172] The electronic glasses perform recognition on the line of sight of the wearer, which can use a visual estimation method to determine the fixation target or fixation point or eye direction of the wearer, and in the case of determining the fixation point or eye direction of the wearer, the occlusion indicated by the fixation point or the occlusion in the eye direction also needs to be determined through target detection; the line of sight recognition or visual estimation method needs to use the eye view collected by the camera located inside the electronic glasses. For example, the vehicle is a car, the wearer sits in the driver's seat, and the occlusion refers to the passenger sitting in the front passenger seat. When the eye direction of the wearer is towards the passenger in the front passenger seat, an environment view that displays the blind area of the passenger (i.e., an environment view that displays the vehicle exterior occluded by the passenger through the passenger) is displayed; or, the vehicle is a car, and when the wearer looks at the left front of the car, it is detected that the line of sight target of the wearer is the left A-pillar located in the left front, and an environment view that displays the blind area of the left A-pillar (i.e., an environment view that displays the vehicle exterior occluded by the left A-pillar through the left A-pillar) is displayed.

[0173] In summary, the method provided by the embodiments of the present application can represent the observation demand of the wearer to a great extent through the direction of the line of sight of the wearer. Whenever the line of sight of the wearer is towards the occlusion, it is considered that the wearer wants to see through the occlusion, thereby eliminating the blind area caused by the occlusion, that is, the environment view occluded by the occlusion is expected to be seen, which can improve the human-machine interaction efficiency and the safety during driving.

[0174] 1.2 The head of the wearer is towards the occlusion

[0175] For the head of the wearer is oriented towards the obstruction, the electronic glasses can perform head pose estimation on the wearer, such as detecting the head pose of the wearer by using physical sensors (such as a gyroscope); or the vehicle can perform head pose estimation on the wearer, such as there is a camera oriented towards the wearer inside the vehicle, and head pose estimation is performed based on the image of the wearer collected by the camera. For example, the vehicle is a car, and the obstruction refers to the A-pillar (including the left A-pillar and the right A-pillar) of the car. When the head of the wearer is oriented towards the left side, the environment picture of the blind area of the left A-pillar is displayed.

[0176] In summary, the method provided by the embodiments of the present application can detect the head orientation of the wearer relative to the obstruction, confirm the direction that the wearer expects to see through according to the head orientation of the wearer, and provide the wearer with the expected perspective content while avoiding the line of sight of the wearer from leaving the road surface, thereby reducing the risk of traffic accidents caused by the line of sight of the wearer leaving the road surface and improving the safety during driving.

[0177] 1.3 Obstruction appearing in the glasses picture

[0178] When the obstruction appears in the glasses picture, the blind area caused by the obstruction in the line of sight of the wearer is eliminated. The obstruction refers to at least one of the components of the vehicle, the occupant, and the objects inside the vehicle. The components of the vehicle refer to various accessories that make up the vehicle, such as the body, steering wheel, etc. of the car, and the operating table, the body, etc. of the ship. The occupant refers to the person located on or inside the vehicle. The objects inside the vehicle refer to the objects installed or placed inside the vehicle after the vehicle is manufactured, such as the decorations on the center console, the decorations hanging on the inside rearview mirror, and the decorations placed in front of the rear windshield glass, etc.

[0179] In summary, the method provided by the embodiments of the present application eliminates the blind area caused by all occlusions in the glasses picture. The method processes all occlusions in the glasses picture or the picture content observed by the wearer uniformly, on the one hand, without the need for other identification means to segment different occlusions in the picture content, thereby improving the processing efficiency and reducing the processing difficulty; on the other hand, the uniformly processed glasses picture can reduce the sense of fragmentation caused by the picture content to the wearer to a certain extent. For example, for a car, the entire frame is an occlusion, if only part of the frame is transparent, a car with a missing or semi-transparent left A-pillar may appear, which conflicts with the car in the common sense of the wearer, and may cause a strong sense of fragmentation to the wearer, which not only affects the user experience, but also may cause a traffic accident due to the confusion of the wearer. If the entire car body is processed, i.e., a transparent or semi-transparent car body is displayed, the entire glasses picture has a relatively harmonious color tone, which can reduce the sense of fragmentation.

[0180] 1.4 Human-computer interaction indication of the wearer requiring to see through the occlusion

[0181] The wearer can use different human-computer interaction methods to issue the human-computer interaction indication. The human-computer interaction method includes at least one of voice interaction, gesture interaction, eye movement interaction, and brain-computer interface. In addition, due to the special scene of wearing electronic glasses in a vehicle, the human-computer interaction method can also include the interaction of the wearer with the vehicle.

[0182] In the case of including multiple occlusions or multiple occlusion types, the human-computer interaction indication can be used to indicate the occlusion or the occlusion type, the occlusion type being used to indicate the type of the occlusion, such as the occlusion type including the components of the vehicle, the passengers, and the articles in the vehicle. For example, the occlusion type includes type 1, type 2, and type 3; the occlusions include occlusion a1 to occlusion a6, wherein occlusion a1, occlusion a2, and occlusion a3 belong to type 1, occlusion a4 and occlusion a5 belong to type 2, and occlusion a6 belongs to type 3; the wearer can indicate to see through occlusion a1, or can indicate to see through the occlusions of type 1 (i.e., occlusion a1, occlusion a2, and occlusion a3).

[0183] The human-computer interaction indication is a human-computer interaction instruction indicating the electronic glasses to display the see-through content, and the human-computer interaction instruction is an instruction using the human-computer interaction method.

[0184] The human-computer interaction indication of the wearer's request for the see-through occlusion includes at least one of the following cases: a voice see-through instruction of the wearer for the occlusion; a gesture selection instruction of the wearer for the occlusion; a directional operation of the wearer for the vehicle, the directional operation being in a corresponding relationship with the occlusion; an eye movement selection instruction of the wearer for the occlusion; and an electroencephalogram instruction of the wearer for the occlusion.

[0185] The voice see-through instruction is, for example, a voice "see through the entire vehicle body", "see through the A-pillar of the vehicle", "I want to see the picture in front of the vehicle", and the like. The gesture selection instruction is, for example, that the wearer touches or points to a certain part of the vehicle, or that the wearer uses a finger to encircle a certain occlusion. The eye movement selection instruction is, for example, that the wearer gazes at the occlusion for more than 1 second. The electroencephalogram instruction is, for example, the thought of the wearer for the see-through occlusion. The directional operation of the wearer for the vehicle is, for example, that the wearer turns the steering wheel to the left, that the wearer turns on the left turn signal, or that the wearer starts the vehicle.

[0186] Optionally, the human-computer interaction indication is multi-modal, that is, multiple human-computer interaction indications described above are used. For example, in combination with the voice see-through instruction and the eye movement selection instruction, the wearer gazes at a certain occlusion and says "see through". Or, in combination with the voice see-through instruction and the gesture selection instruction, the wearer's left hand touches a certain occlusion and says "see through the occlusion pointed by the left hand".

[0187] In summary, the method provided by the embodiments of the present application uses diversified human-computer interaction indications to confirm the demand of the wearer for see-through content, and improves the interaction efficiency and user experience by taking the demand of the wearer as the basis. Meanwhile, the electronic glasses are usually equipped with various sensors based on human-computer interaction. Under the premise that the electronic glasses are equipped with human-computer interaction sensors, the human-computer interaction indication of the wearer's request for the see-through occlusion is detected, and the occlusion is see-through based on the intention of the wearer. This can not only avoid the see-through content that is not accustomed to the normal state, but also meet the occasional see-through demand of the wearer.

[0188] 2. The blind area of the vehicle exists warning information

[0189] The see-through condition (or the warning information existing in the blind area of the vehicle) includes at least one of the following: the warning information existing in a warning area corresponding to the occlusion, the warning area being an area in the range of the side of the vehicle; and the warning information existing in a predicted moving route of the occlusion, the predicted moving route being a route to be moved by the vehicle within a time threshold or a distance threshold.

[0190] The warning area is an area in a peripheral range of the vehicle. Optionally, the peripheral range refers to a circular range with the center of the vehicle as the center, and the warning area corresponding to the occlusion is a blind area caused by the occlusion in the peripheral range. For example, as shown in FIG. 1, the warning area is an area in a circular range with the center of the vehicle as the center, that is, the warning area is a circle 9, and the occlusion is a vehicle head, and the warning area corresponding to the occlusion is a vehicle head blind area 10.

[0191] The predicted moving route is a route in which the vehicle will move within a time threshold or a distance threshold. For example, the predicted moving route is a route in which the vehicle will move within 5 seconds; or, the predicted moving route is a route in which the vehicle will move within 100 meters.

[0192] The electronic glasses or the vehicle will predict a route in which the vehicle will move within a time threshold or a distance threshold. The electronic glasses or the vehicle will predict based on at least one of a driving habit and a current driving speed, a current deflection angle of a steering wheel, and a road surface condition.

[0193] The warning information includes at least one of an obstacle and an abnormal situation. The obstacle is usually immovable and fixed, such as a telegraph pole, a wall, a mountain, a green belt, etc. The abnormal situation is an unusual or sudden traffic situation, such as a suddenly appearing pedestrian, a suddenly turning vehicle, a child in the vehicle head blind area at the start, etc.

[0194] That is, the warning information exists in the warning area corresponding to the occlusion, including at least one of the following situations: an obstacle appears in the warning area corresponding to the occlusion; an abnormal situation appears in the warning area corresponding to the occlusion. The warning information exists in the predicted moving route corresponding to the occlusion, including at least one of the following situations: an obstacle exists in the predicted moving route corresponding to the occlusion; an abnormal situation exists in the predicted moving route corresponding to the occlusion.

[0195] In summary, the method provided by the embodiments of the present application displays the see-through content corresponding to the warning information to the wearer when the electronic glasses or the vehicle detects that the warning information exists, and does not display the see-through content when there is no warning information. Compared with the current daily driving, the change is not big, and the wearer can more easily adapt to the see-through content displayed in the electronic glasses. At the same time, the see-through content is displayed only when the warning information appears, which fully considers the traffic accidents that may be caused by the warning information, displays the see-through content to remind the wearer that the warning information appears in the blind area, so that the wearer can more intuitively pay attention to the warning information in the environment picture of the blind area, thereby making a response decision, and improving the safety during driving.

[0196] 3. The traffic scene needs the wearer to observe the blind area of the vehicle

[0197] Optionally, the current traffic scene in which the wearer is located is identified by the electronic glasses or the vehicle.

[0198] The traffic scene requires the wearer to observe the blind area of the vehicle, that is, the current traffic scene is a high perspective requirement traffic scene, or the current traffic scene is a traffic scene with frequent accidents. The traffic scene refers to the scene in which the vehicle is located, which is related to the driving scene of the vehicle, the road conditions, etc. For example, the traffic scene is a road near a school; or the traffic scene is a mountain road with many sharp turns; or the traffic scene is a narrow road; or the traffic scene is an up-and-down slope, etc.

[0199] In summary, the method provided by the embodiments of the present application judges the traffic scene by the electronic glasses or the vehicle, and displays the perspective content in the traffic scene requiring the wearer to observe the blind area. The wearer does not need to indicate or judge, on the one hand, the judgment by the electronic glasses or the vehicle can release the brain power of the wearer, so that the wearer can focus more on the driving process, or the wearer can do what he wants to do; on the other hand, for the wearer with insufficient driving experience, it is often difficult for them to judge when to observe the blind area, and the assistance of the electronic glasses or the vehicle can make the wearer with insufficient driving experience also have the awareness of paying attention to the blind area, which can not only exercise the driving ability, but also improve the safety of driving.

[0200] In addition to the above perspective content, in order to further improve the safety and human-computer interaction requirements during vehicle driving, the electronic glasses also provide other types of picture content. Such as zoom content, rearview mirror content, rearview mirror extension content, virtual warning elements and reserved picture content, etc. Next, these picture contents and the display scene (triggering condition) of the picture contents will be introduced one by one.

[0201] 4.1 Zoom content

[0202] In some embodiments, as shown in FIG. 13, the method further includes:

[0203] Step 310: In response to the zoom trigger operation for the first area picture in the glasses picture, display the glasses picture with zoom content.

[0204] The first area picture is part of the picture content in the glasses picture, and the zoom content refers to the picture content after zooming the first area picture.

[0205] Optionally, the first area picture is part of the picture content in the glasses picture; or the first area picture is the picture content of the first area in the glasses picture, and the first area is a certain area in the glasses picture.

[0206] The zoom trigger operation on the first region picture in the glasses picture is used to change the display size of the first region picture. The zoom content refers to the picture content after zooming the first region picture. The zooming includes zooming out or zooming in. For example, the glasses picture is shown in part (1) of FIG. 14, and the first region picture is picture 45. After performing the zoom trigger operation on the first region picture, the glasses picture with zoom content is shown in part (2) of FIG. 14, and the zoom content is picture 46, which is zoomed in relative to the first region picture 45.

[0207] Optionally, in response to the zoom trigger operation on the zoom content in the glasses picture, the display size of the zoom content is changed. As shown in part (3) of FIG. 14, the zoom content 47 is zoomed in (or said to be zoomed in continuously on the first region picture), and the zoom degree of the zoom content 47 relative to the first region picture is larger than that of the zoom content 46 shown in part (2) of FIG. 14. Alternatively, the zoom content can be zoomed out. As shown in part (4) of FIG. 14, the zoom content 48 is zoomed out relative to the first region picture, and the zoom degree of the zoom content 48 relative to the first region picture is smaller than that of the zoom content 46 shown in part (2) of FIG. 14.

[0208] Optionally, the zoom trigger operation on the first region picture in the glasses picture can be triggered by a human-computer interaction instruction, such as that the wearer selects the first region picture in the glasses picture by a gesture selection instruction and says “zoom in the selected region” by a voice instruction; or the wearer indicates to select the first region picture and zoom in the first region picture to obtain the zoom content by an electroencephalogram instruction. The form of the corresponding human-computer interaction instruction is similar to “1.4. Wearer’s human-computer interaction instruction for requesting to see through the occlusion”, which is not described herein again.

[0209] Optionally, when zooming the first region picture, the entire first region picture is zoomed; or a picture element in the first region picture is zoomed. That is, the zoom content refers to the picture content after zooming the entire first region picture; or the zoom content refers to the picture content after zooming a picture element in the first region picture. As shown in FIG. 14, parts (2) to (4) of FIG. 14 are zooming the entire picture content; and part (5) of FIG. 14 is zooming only the picture element 49 “puppy” in the first region picture.

[0210] In summary, the method provided by the embodiments of the present application can scale the first area picture in the glasses picture, so that the wearer can better observe the first area picture in the glasses picture. For example, when the wearer is preparing to start, a child is found in front, but it is difficult to clearly observe due to the long distance, so the area where the child is located can be selected as the first area picture, and the first area picture is zoomed in to clearly observe. Or, when driving a fighter, a black dot suspected to be an enemy is found in the distance, and the first area picture where the black dot is located can be selected and zoomed in to better observe the enemy situation.

[0211] 4.2 Rearview mirror content

[0212] The rearview mirror content is a partial picture of the external environment in the non-forward view angle of the wearer. The rearview mirror content is displayed through at least one of an optical rearview mirror, an electronic rearview mirror, and a virtual rearview mirror.

[0213] The optical rearview mirror is a rearview mirror based on optical imaging. The optical rearview mirror is the most widely used rearview mirror in the field of automobiles at present. The electronic rearview mirror is a rearview mirror that displays the external environment in the non-forward view angle through an electronic screen. The imaging of the electronic rearview mirror usually also needs a camera in communication connection with the electronic rearview mirror to be responsible for collecting the external environment picture. The virtual rearview mirror is a rearview mirror that displays the external environment in the non-forward view angle in the second area of the glasses picture by the electronic glasses. That is, the virtual rearview mirror is invisible without wearing the electronic glasses.

[0214] Optionally, the virtual rearview mirror is displayed in the glasses picture in a normal state. Or, the virtual rearview mirror is displayed in the glasses picture based on a rearview mirror display condition. The virtual rearview mirror is located in the second area of the glasses picture. The rearview mirror display condition includes at least one of the following: the line of sight of the wearer is directed to a trigger area; the trigger area appears in the glasses picture; a human-computer interaction instruction that the wearer requires to display the virtual rearview mirror; warning information appears in the rearview mirror content; and a traffic scene requires the wearer to observe the rearview mirror content. The trigger area is an area in the vehicle, that is, the trigger area is a real existing area. For example, the vehicle is a car, and the car does not have an interior rearview mirror, and the trigger area is the position of the original rearview mirror. The rearview mirror display condition is similar to the perspective condition, and will not be described here.

[0215] Optionally, the position and size of the virtual rearview mirror in the glasses picture are adjustable, that is, the position and size of the second area are adjustable.

[0216] The blind area in the rearview mirror content refers to an external region that is not observed by the wearer due to an occlusion in the rearview mirror content. That is, the actual position of the occlusion in the rearview mirror content is not necessarily in front of the wearer, but is most likely in the rear or side or rear side of the wearer. For example, as shown in part (1) of FIG. 15, the occlusion displayed in the rearview mirror 80 is the rear side of the vehicle body 81.

[0217] Optionally, in the case where there is an occlusion in the rearview mirror content, the occlusion is not see-through. Or, in the case where there is an occlusion in the rearview mirror content, the occlusion is see-through to display an environmental picture of the blind area. As shown in FIG. 15, when the vehicle is a car and the occlusion is the vehicle body, in the case where there is an occlusion in the rearview mirror content, the occlusion is not see-through, as shown in part (1) of FIG. 15, for the vehicle body 81 displayed in the rearview mirror 80, there is no see-through display of the vehicle body. In the case where there is an occlusion in the rearview mirror content, the occlusion is see-through to display an environmental picture of the blind area, as shown in part (2) of FIG. 15, for the vehicle body 81 displayed in the rearview mirror 80, the vehicle body 81 is see-through, and the external environment 82 that is not in the forward viewing angle and is occluded by the vehicle body is displayed. That is, the rearview mirror content includes see-through content or does not include see-through content.

[0218] Optionally, whether the rearview mirror content includes see-through content is set by the wearer or by the developer.

[0219] In summary, the method provided by the embodiments of the present application shows the classification of the rearview mirror content displayed in the glasses picture, and for different types of rearview mirrors, the external environment of the vehicle in the non-forward viewing angle (or the external environment behind the vehicle, the side rear of the vehicle, and the external environment on both sides) can be displayed in the glasses picture displayed by the electronic glasses, thereby expanding the range of the external environment that can be observed in the glasses picture. In addition, the method of assisting driving by wearing electronic glasses can adapt to various vehicle models, whether it is an optical electronic rearview mirror or an electronic rearview mirror, or even in the case where there is no rearview mirror, a virtual rearview mirror can be added to observe the external environment in the non-forward viewing angle in the case of wearing electronic glasses, thereby achieving the effect of eliminating blind areas in all directions.

[0220] 4.3 Rearview mirror extension content

[0221] In some embodiments, the glasses picture displays the rearview mirror content, as shown in FIG. 16, and the method further includes:

[0222] Step 410: In response to the triggering operation of the picture expansion mode of the rearview mirror content, a glasses picture with rearview mirror extension content is displayed, and the rearview mirror extension content is the picture content after the extension of the partial picture in the rearview mirror content.

[0223] Optionally, the rearview mirror extension content is a picture content extended from a partial picture in the rearview mirror content; or, the rearview mirror extension content is a picture content extended from the rearview mirror content. The rearview mirror extension content and the rearview mirror content are picture contents in the same perspective, and the picture center of the rearview mirror extension content is consistent with the picture center of the rearview mirror content. The rearview mirror extension content is shown in FIG. 15. When the rearview mirror extension content is not displayed, the wearer observes the picture in the glasses related to the rearview mirror as shown in FIG. 15, part (1), the rearview mirror content is displayed in the rearview mirror 80, and the front picture of the vehicle is displayed outside the rearview mirror 80, which shows the guardrail and a tree 83 outside the guardrail; after triggering the picture expansion mode, as shown in FIG. 15, part (3), the front picture of the vehicle is not displayed outside the rearview mirror 80, but the external area in the same perspective as the rearview mirror content but in a non-forward perspective is displayed, which only shows the guardrail 84. It can be understood that the rearview mirror content is a picture content obtained based on the first mirror reflection, the rearview mirror extension content is a picture content obtained based on the second mirror reflection, the positions, angles and mirror types of the first mirror and the second mirror are the same, and the size of the first mirror is smaller than that of the second mirror; or, the rearview mirror content is obtained based on the first camera, and the rearview mirror extension content is obtained based on the second camera, the positions and angles of the first camera and the second camera are the same, and the zoom of the first camera is smaller than that of the second camera.

[0224] Optionally, the rearview mirror extension content does not include the rearview mirror content; or, the rearview mirror extension content includes the rearview mirror content. In the case that the rearview mirror extension content does not include the rearview mirror content, as shown in FIG. 15, part (3), the rearview mirror extension content is the picture content in FIG. 15, part (3) except the rearview mirror 80, which shows the external area outside the rearview mirror range, and the external area shows a guardrail 84; or, in the case that the rearview mirror extension content includes the rearview mirror content, as shown in FIG. 15, part (3), the rearview mirror content is the content displayed in the mirror area in the rearview mirror 80, and the rearview mirror extension content is all the content shown in FIG. 15, part (3). That is, the rearview mirror extension content is the picture content displayed outside the rearview mirror content after matching with the rearview mirror content; or, the rearview mirror extension content is the picture content overlaid on the rearview mirror content after matching with the rearview mirror content.

[0225] Optionally, in the case that the rearview mirror content includes see-through content, the rearview mirror extended content also includes see-through content, the rearview mirror content is as shown in part (2) of FIG. 15, and the rearview mirror extended content is as shown in part (4) of FIG. 15; or, in the case that the rearview mirror content includes see-through content, the rearview mirror extended content includes see-through content, the rearview mirror content is as shown in part (2) of FIG. 15, and the rearview mirror extended content is as shown in part (3) of FIG. 15; or, in the case that the rearview mirror content does not include see-through content, the rearview mirror extended content also does not include see-through content, the rearview mirror content is as shown in part (1) of FIG. 15, and the rearview mirror extended content is as shown in part (3) of FIG. 15; or, in the case that the rearview mirror content does not include see-through content, the rearview mirror extended content includes see-through content, the rearview mirror content is as shown in part (1) of FIG. 15, and the rearview mirror extended content is as shown in part (4) of FIG. 15.

[0226] For example, the rearview mirror content 86 and the rearview mirror extended content 87 are as shown in FIG. 17, the rearview mirror extended content 87 shown in part (2) of FIG. 17 is an extension of the rearview mirror content 86 shown in part (1) of FIG. 17, i.e., the picture range of the rearview mirror extended content 87 is larger than the picture range of the rearview mirror content 86, and the picture range of the rearview mirror extended content 87 is generally much smaller than the picture range 85 of the glasses picture shown in part (1) of FIG. 17, especially for the inner rearview mirror, the rearview mirror extended content 87 does not cause excessive occlusion to the external environment picture of the forward view angle displayed in the glasses picture.

[0227] Optionally, the triggering operation of the picture expansion mode includes at least one of the following: the wearer's eyeball is oriented towards the rearview mirror content; the wearer's head is oriented towards the rearview mirror content; there is warning information in the warning area corresponding to the rearview mirror extended content; there is warning information in the predicted moving route corresponding to the rearview mirror extended content; the traffic scene requires the wearer to observe the rearview mirror extended content; and the human-computer interaction indication of the wearer requiring to turn on the picture expansion mode. The "there is warning information in the warning area corresponding to the rearview mirror extended content; there is warning information in the predicted moving route corresponding to the rearview mirror extended content" is similar to the above-mentioned "2. There is warning information in the blind area of the vehicle", the "the traffic scene requires the wearer to observe the rearview mirror extended content" is similar to the above-mentioned "3. The traffic scene requires the wearer to observe the blind area of the vehicle", and the "the wearer's eyeball is oriented towards the rearview mirror content; the wearer's head is oriented towards the rearview mirror content; the human-computer interaction indication of the wearer requiring to turn on the picture expansion mode" is similar to the above-mentioned "1. The wearer has the demand of observing see-through content", which will not be repeated here.

[0228] In some embodiments, the display range of the rearview mirror extension content can be adjusted; the display range of the rearview mirror extension content is determined in response to a zoom operation on the rearview mirror extension content. The zoom operation includes a zoom-out operation or a zoom-in operation. The zoom operation on the rearview mirror extension content can be triggered by a human-computer interaction instruction, such as the wearer selecting the corresponding rearview mirror extension content by gesture selection instruction, and saying "enlarge the display range of the extension content" by voice instruction; or the wearer indicates the rearview mirror extension content by electroencephalogram instruction and indicates to zoom out the display range of the rearview mirror extension content. The corresponding form of the human-computer interaction instruction is similar to "1.4. The human-computer interaction instruction of the wearer's request for the perspective barrier", which will not be described here. Alternatively, the minimum display range of the rearview mirror extension content is the display range of the corresponding rearview mirror content, and the maximum display range is the display range of the glasses screen.

[0229] In summary, the method provided by the embodiments of the present application can display the rearview mirror extension content, solve the problem of limited reflection range of the optical rearview mirror, enable the wearer to observe more pictures in the rearview mirror blind area, and avoid the electronic rearview mirror, which displays a large range of external environment in a small screen. Instead, the rearview mirror extension content is displayed outside the original rearview mirror, and the rearview mirror content or the rearview mirror extension content is not zoomed out, avoiding misjudgment of the human eye due to different zoom ratios, and improving the safety of driving.

[0230] In addition, the setting of the triggering condition considers two aspects. On the one hand, the user's needs are considered, and the rearview mirror extension content is displayed only when the wearer expects to observe the rearview mirror extension content. The observation habits of the wearer can be recorded, and the electronic glasses can display the perspective content according to the observation habits without the wearer actively providing instruction information to indicate the electronic glasses to display the rearview mirror extension content, thereby improving the human-computer interaction efficiency and providing a better experience for the wearer. The wearer can be more immersed in driving, and the wearer does not need to be distracted to indicate the electronic glasses to display the rearview mirror extension content while driving. On the other hand, the scene in which perspective is needed in actual driving scenarios is considered, that is, when there is a dangerous situation in the blind area corresponding to the rearview mirror extension, the wearer needs to be reminded of the dangerous situation by displaying the rearview mirror extension content.

[0231] In addition, zooming in or out the display range of the rearview mirror extension content enables the wearer to freely adjust according to the driving scene, which can improve the experience of the wearer on the one hand, and on the other hand, for some scenes that need to be carefully observed from non-forward view angles, such as reversing and lane changing, the wearer can adjust the display range of the rearview mirror extension content, so that the wearer can observe a larger range of pictures from non-forward view angles, indirectly improving the safety of driving.

[0232] 4.4 Virtual alert element

[0233] In some embodiments, when a dangerous situation is detected, the wearer of the electronic glasses is alerted, as shown in FIG. 18, the method further comprises:

[0234] Step 510: in response to the alert trigger operation, a virtual alert element is displayed in the glasses picture, the virtual alert element is used to indicate the picture content that the wearer needs to pay attention to.

[0235] The virtual alert element is a picture element with an alerting meaning, which is used to indicate the picture content that the wearer needs to pay attention to. The picture content that the wearer needs to pay attention to includes alerting information, i.e. obstacles and / or abnormal situations. The picture content can be the picture content in the blind area (or the environmental picture of the blind area), or the picture content in the non-blind area (i.e. the environmental picture not blocked by the occlusion).

[0236] The virtual alert element is at least one of an icon, a figure, and an animation. As shown in FIG. 19, the virtual alert element displayed in the glasses picture includes an icon 61 and a figure 62. The virtual alert element can be highlighted, such as having a highlight, a border, a highlight border, etc.; the virtual alert element can also be dynamic, such as a virtual alert element that flashes constantly, an alert element whose color changes over time, etc.

[0237] Optionally, the display effect of the virtual alert element is related to the distance between the vehicle and the alerting information, the speed of the vehicle, and the estimated collision time of the vehicle and the alerting information. The estimated collision time is the time when the vehicle and the alerting information are estimated to collide. For example, the virtual alert element is an icon, the closer the distance between the vehicle and the alerting information, the larger the display effect of the icon; or, the virtual alert element is an icon, the color of the icon is green in the initial state, and as the estimated collision time between the vehicle and the alerting information becomes shorter, the color of the icon gradually changes from green to orange, and finally to red; or, the virtual alert element has a highlight border, the faster the speed of the vehicle, the higher the brightness of the highlight border; or, the virtual alert element is an icon, the distance between the vehicle and the alerting information or the estimated collision time of the vehicle and the alerting information is displayed on the icon, and as the distance between the vehicle and the alerting information or the estimated collision time of the vehicle and the alerting information changes, the number on the icon will also change accordingly.

[0238] Optionally, in response to the warning triggering operation, a virtual obstacle avoidance element is displayed on the glasses picture, and the virtual obstacle avoidance element is used to instruct the wearer to avoid the warning information. The virtual obstacle avoidance element is used to indicate at least one of an obstacle avoidance route and an obstacle avoidance scheme. For example, when a pedestrian is detected to be crossing the pedestrian crossing, a stop icon is displayed on the glasses picture; or, when a large stone is detected in the middle of the road, a driving route that bypasses the large stone is displayed, and the like.

[0239] In addition to providing the virtual warning element or the virtual obstacle avoidance element in the picture, a corresponding voice can be played to remind the wearer.

[0240] Optionally, in response to the warning triggering operation, a warning voice is played; and / or, in response to the warning triggering operation, an obstacle avoidance voice is played. For example, in the scenario shown in FIG. 19, a warning voice “there is a child passing by in front of the left, please pay attention to the distance” is played; or, an obstacle avoidance voice “there is a child passing by in front of the left, please slow down” is played. Or, when driving on a narrow road, a voice prompt is given according to the distance from the roadside obstacle, such as playing a warning voice “it is safe to pass”, “the left side mirror is too close to the left side obstacle, only 20 cm, please pay attention to the distance”.

[0241] The virtual warning element and / or the warning voice are used to indicate at least one of a warning information position, a warning information type, and a distance from the warning information. The virtual obstacle avoidance element and / or the obstacle avoidance voice are used to indicate at least one of an obstacle avoidance route and an obstacle avoidance scheme.

[0242] In summary, the method provided by the embodiments of the present application can display a virtual warning element to prompt the wearer to pay attention to the warning information after detecting a warning triggering operation, so as to enable the wearer who is used to normal display of see-through content to pay attention to the warning information. The method can also prompt the wearer who has poor driving habits, such as the wearer who is prone to distraction, to pay attention to the warning information. In the case of unmanned driving, the method can display the warning information through the electronic glasses, so as to enable the wearer who is doing other things to pay attention to the warning information and take over the driving in time. Thus, the safety of driving is improved.

[0243] 4.5 Retaining picture content

[0244] In some embodiments, the see-through content includes see-through content corresponding to a plurality of occluders, as shown in FIG. 20, the method further includes:

[0245] Step 610: In response to a retaining operation for the partial occluder, a glasses picture with retaining picture content is displayed, and the retaining picture content refers to picture content corresponding to the partial occluder and having a higher opacity than the see-through content.

[0246] Optionally, the retaining operation for the partial occlusion can also be said to be a human-computer interaction instruction for the wearer to request retaining the partial occlusion; that is, in response to the retaining operation for the partial occlusion, the glasses picture with the retaining picture content is displayed; or, in response to the human-computer interaction instruction for the wearer to request retaining the partial occlusion, the glasses picture with the retaining picture content is displayed. The human-computer interaction instruction for the wearer to request retaining the partial occlusion is similar to the “1.4 human-computer interaction instruction for the wearer to request see-through occlusion” described above, and will not be described here again.

[0247] The retaining picture content refers to the picture content with higher opacity than the see-through content; for example, the opacity of the occlusion in the see-through content is 0%, and the opacity of the occlusion in the retaining picture content is 100%; or, the opacity of the occlusion in the see-through content is 25%, and the opacity of the occlusion in the retaining picture content is 75%; or, the retaining picture content refers to the picture content with lower transparency than the see-through content; for example, the transparency of the occlusion in the see-through content is 100%, and the transparency of the occlusion in the retaining picture content is 0%; or, the transparency of the occlusion in the see-through content is 75%, and the transparency of the occlusion in the retaining picture content is 25%.

[0248] When there are multiple occlusions in the glasses picture, the wearer can set the see-through condition (or the retaining condition) of each occlusion. For example, as shown in FIG. 21, the wearer selects four occlusions by fingers, which are the steering wheel, the human hand, and two instrument panels, and there are three regions in total, as shown in part (1) of FIG. 21. After selecting the occlusions to be retained, the electronic glasses automatically identify the occlusions selected by the wearer and display the glasses picture with the retaining picture content for the wearer to confirm, as shown in part (2) of FIG. 21. After the wearer confirms that the glasses picture with the retaining picture content is the desired glasses picture, the retaining operation is ended, and the glasses picture with the retaining picture content is displayed normally, as shown in part (3) of FIG. 21.

[0249] Optionally, the display of the reserved picture content is cancelled in the glasses picture when a reserved cancellation condition is met. The reserved cancellation condition is a condition for cancelling the display of the reserved picture content. The reserved cancellation condition includes at least one of the following: there is warning information in the warning area corresponding to the reserved picture content; there is warning information in the predicted moving route corresponding to the reserved picture content; and the human-computer interaction instruction of the wearer for cancelling the reserved picture content. The "there is warning information in the warning area corresponding to the reserved picture content; there is warning information in the predicted moving route corresponding to the reserved picture content" is similar to the "2. There is warning information in the blind area of the vehicle" described above, and the "human-computer interaction instruction of the wearer for cancelling the reserved picture content" is similar to the "1.4. Human-computer interaction instruction of the wearer for requiring the see-through obstruction" described above, and will not be described here. For example, as shown in the (4) part of FIG. 21, during driving, a small dog appears in the warning area (blind area caused by the obstruction corresponding to the reserved picture content) corresponding to the reserved picture content, the reserved effect (or the opacity effect) of the part of the reserved picture content is cancelled, and the transparency thereof is changed to be consistent with the transparency of the transparent content. Optionally, a virtual warning element such as a warning icon and a circle selection figure is displayed on the small dog.

[0250] In summary, the method provided by the embodiments of the present application improves the autonomy of the wearer in controlling the glasses picture by allowing the wearer to define whether the obstruction needs to be see-through or reserved. For some wearers, if all components of the vehicle are see-through, the rapidly changing ground guide lines, road signs and the like when the vehicle is driving at high speed may distract them during driving, which may lead to traffic accidents. At this time, they can reserve the base and the components on both sides and do not see through them to improve the safety of driving.

[0251] FIG. 22 shows a flowchart of a picture acquisition method provided by an example embodiment of the present application. The method is performed by a carrier device, which can be the carrier device in the computer system described above. The method includes at least part of steps 710, 720 and 730:

[0252] The vehicle device has a first communication connection with at least one camera facing a blind area of the vehicle, the blind area refers to an external area that the wearer cannot observe due to an obstruction, and the obstruction is at least one of a roadblock, a component of the vehicle, an occupant, and an article located in the vehicle. For example, the vehicle is a car, and in order to eliminate the blind area of the vehicle shown above, the camera can be installed on the front of the car, on the left and right rearview mirrors, on the AB column, on the vehicle base, etc.; for the blind area of the vehicle during driving, the camera can be a camera installed on road facilities (such as street lamps, telegraph poles, etc.), other cars, and the like, which are not limited by the embodiments of the present application. For the camera installed on the vehicle, the first communication connection between the vehicle device and the camera is usually a wired communication connection, such as a CAN bus, a USB connection, etc.; for the camera installed on the road facility or other vehicle, the first communication connection between the vehicle device and the camera is usually a wireless communication connection, such as a Bluetooth connection, a V2X connection, etc. It should be noted that the communication connection between the vehicle device and the camera installed on the vehicle can also be a wireless communication connection, which is not limited by the embodiments of the present application.

[0253] Step 710: Establishing a second communication connection with the electronic glasses, the electronic glasses being worn by the wearer located inside the vehicle.

[0254] Optionally, the second communication connection established between the vehicle device and the electronic glasses includes at least one of a wireless communication connection and a wired communication connection; the connection mode of the second communication connection established between the vehicle device and the electronic glasses includes at least one of direct connection and indirect connection, the direct connection means that the communication connection between the vehicle device and the electronic glasses does not pass through other communication devices, and the indirect connection means that the communication connection between the vehicle device and the electronic glasses passes through the relay of other communication devices.

[0255] Optionally, the first communication connection established between the vehicle device and the at least one camera includes at least one of a wireless communication connection and a wired communication connection; the connection mode of the first communication connection established between the vehicle device and the at least one camera includes at least one of direct connection and indirect connection, the direct connection means that the communication connection between the vehicle device and the at least one camera does not pass through other communication devices, and the indirect connection means that the communication connection between the vehicle device and the at least one camera passes through the relay of other communication devices.

[0256] The communication connection modes adopted by the first communication connection and the second communication connection can be the same or different, which is not limited by the embodiments of the present application.

[0257] Step 720: Determining the auxiliary data based on the environment picture collected by the at least one camera.

[0258] The environment picture is obtained based on the first communication connection, and the auxiliary data is used to make the electronic glasses display a glasses picture with see-through content. The see-through content is an environment picture of a blind area presented after performing see-through display on an occlusion in a picture collected by the electronic glasses.

[0259] The auxiliary data is data used to assist the electronic glasses to display an environment picture with see-through content. The auxiliary data is obtained based on an environment picture collected by at least one camera. It can be understood that the auxiliary data is data obtained by encapsulating the environment picture collected by the at least one camera; or the auxiliary data is data obtained by digitizing the environment picture collected by the at least one camera; or the auxiliary data is obtained based on the environment picture collected by the at least one camera, for example, the auxiliary data is data obtained by digitizing the glasses picture with see-through content, or the auxiliary data is data obtained by encapsulating the environment picture of the blind area, and the like.

[0260] Optionally, the auxiliary data is determined based on an environment picture collected by the at least one camera at a current time; or the auxiliary data is determined based on an environment picture collected by the at least one camera in the past n seconds, where n is a positive integer.

[0261] Optionally, the value of n is determined based on at least one of a driving speed of the vehicle, a road condition, and a driving scene. For example, in a high-speed driving scene (for example, the driving speed of the vehicle is greater than 100 kilometers per hour), the value of n is small, for example, 0.01; in a low-speed driving scene (for example, the driving speed of the vehicle is less than 10 kilometers per hour), the value of n is large, for example, 5; in a non-driving scene, the value of n can be larger, for example, 10 or even 60; in a complex road condition (for example, traffic jam, continuous sharp turns, and the like), the value of n is small; in a simple road condition, the value of n is large; in a driving scene where accidents occur frequently (for example, starting and reversing), the value of n is small. It can be understood that the smaller the value of n is, the higher the timeliness of the auxiliary data is.

[0262] Optionally, the environment picture collected by the at least one camera is at least one of a picture, a video, and a video frame.

[0263] Step 730: sending the auxiliary data to the electronic glasses through the second communication connection.

[0264] The environment picture of the blind area is determined based on the environment picture collected by the at least one camera. Optionally, the environment picture of the blind area (or the see-through content) is directly determined based on the environment picture collected by the at least one camera, that is, the environment picture of the blind area includes part of the environment picture collected by the at least one camera; or the environment picture of the blind area includes the environment picture collected by part of the cameras; or the environment picture of the blind area includes part of the environment picture collected by part of the cameras. Or, the environment picture of the blind area is determined after processing the environment picture collected by the at least one camera.

[0265] In summary, the method provided by the embodiments of the present application enables the carrier device to establish a communication connection with the electronic glasses and transmit auxiliary data for displaying the see-through content to the electronic glasses. The electronic glasses can successfully display the see-through content, and meanwhile, the electronic glasses can be responsible for part of the preparation work of the see-through content, thereby reducing the power consumption of the electronic glasses and improving the endurance of the electronic glasses.

[0266] In some embodiments, the auxiliary data is used to enable the electronic glasses to display the glasses picture with the see-through content when the see-through condition is met. The see-through condition is as described above in “1. The wearer has the demand to observe the see-through content” “2. The blind area of the vehicle exists warning information” “3. The traffic scene needs the wearer to observe the blind area of the vehicle”, which will not be repeated here. In addition, the trigger condition for the display of the see-through content is set, i.e., the see-through condition, so that after the wearer wears the electronic glasses, the see-through content will only be displayed when the see-through condition is met, and if the see-through condition is not met, the see-through content will not be displayed; the discomfort caused by the normal see-through content to the wearer can be avoided, and the see-through content is displayed only when the see-through condition is met without affecting the normal driving of the wearer.

[0267] According to the different solutions, the content corresponding to the auxiliary data is different. The auxiliary data includes at least one of at least one external environment picture, at least one environment picture collected by the camera, and the glasses picture with the see-through content. Different auxiliary data corresponds to different synthesis methods of the driver picture, and the synthesis of the glasses picture with the see-through content needs the assistance of the external environment picture. The following three synthesis methods of the glasses picture will be shown in the embodiments of the present application:

[0268] Method one: the vehicle-mounted terminal synthesizes the external environment picture, and the electronic glasses synthesize the glasses picture with the see-through content (the auxiliary data at least includes at least one external environment picture).

[0269] Method two: the electronic glasses synthesize the external environment picture and the glasses picture with the see-through content (the auxiliary data at least includes at least one environment picture collected by the camera).

[0270] Method three: the vehicle-mounted terminal synthesizes the external environment picture and the glasses picture with the see-through content (the auxiliary data at least includes the glasses picture with the see-through content).

[0271] The following will introduce the above three methods respectively, but the order of the introduction of the three methods is not limited to the advantages and disadvantages of the three methods.

[0272] Method one: the vehicle-mounted terminal synthesizes the external environment picture, and the electronic glasses synthesize the glasses picture with the see-through content (the auxiliary data at least includes at least one external environment picture).

[0273] FIG. 23 shows a flowchart of an overall process of the method of assisting driving according to an example embodiment of the present application. The method is performed by the electronic glasses, and the method further includes:

[0274] In some embodiments, the auxiliary data includes at least one external environment picture. Step 220 can be implemented as step 810.

[0275] In some embodiments, step 810 and step 820 can be exchanged in order or performed simultaneously.

[0276] Step 810: Obtain at least one external environment picture through a communication connection.

[0277] Step 820: Obtain a picture collected by the electronic glasses.

[0278] The electronic glasses include a camera having a same direction view angle as the wearer, and the camera is used to collect a picture in the same direction as the line of sight of the wearer. It should be noted that the electronic glasses can include one or more cameras, and the embodiments of the present application do not limit this. In the case where the electronic glasses include only one camera, only the picture collected by the camera is obtained; in the case where the electronic glasses include multiple cameras, the pictures collected by the multiple cameras are obtained. Alternatively, the picture collected by the electronic glasses refers to the picture directly observed by the wearer through the lenses of the electronic glasses.

[0279] For example, the electronic glasses include two cameras, and the two cameras are respectively used to collect the picture content observed by the left eye of the wearer and the picture content observed by the right eye of the wearer. The camera used to collect the picture content observed by the left eye can be referred to as a left-eye camera, and the camera used to collect the picture content observed by the right eye can be referred to as a right-eye camera. Based on the picture collected by the left-eye camera (left-eye picture), the glasses picture displayed on the lens (or screen) corresponding to the left eye in the electronic glasses is determined; based on the picture collected by the right-eye camera (right-eye picture), the glasses picture displayed on the lens (or screen) corresponding to the right eye in the electronic glasses is determined. Alternatively, the picture collected by the left-eye camera (left-eye picture) and the picture collected by the right-eye camera (right-eye picture) are sampled to obtain the picture collected by the electronic glasses (or referred to as a binocular picture), and based on the picture collected by the electronic glasses (or referred to as a binocular picture), the glasses picture is determined.

[0280] For example, the electronic glasses include a camera which is located at the center of the electronic glasses or the face of the wearer. The image (or binocular image) collected by the camera (or the electronic glasses) is sampled to obtain a left eye image and a right eye image. The left eye image is used to determine the glasses image displayed by the lens (or screen) corresponding to the left eye in the electronic glasses. The right eye image is used to determine the glasses image displayed by the lens (or screen) corresponding to the right eye in the electronic glasses. Alternatively, the glasses image is determined based on the image collected by the camera (or the electronic glasses).

[0281] In the above-mentioned "the image (left eye image) collected by the left eye camera and the image (right eye image) collected by the right eye camera are sampled to obtain the image (or binocular image) collected by the electronic glasses" or "the image (or binocular image) collected by the camera (or the electronic glasses) is sampled to obtain a left eye image and a right eye image", the viewing angle range observed by the human eye is involved. In general, as shown in FIG. 24, the vertical viewing angle range of the human eye is considered to be 150°, the horizontal viewing angle range of a single eye is 156°, i.e. the left eye viewing angle range is 156°, the right eye viewing angle range is also 156°, and the maximum viewing angle range of the two eyes is 188°. Therefore, for the case of "the image (left eye image) collected by the left eye camera and the image (right eye image) collected by the right eye camera are sampled to obtain the image (or binocular image) collected by the electronic glasses", feature matching can be performed on the left eye image and the right eye image to find the image overlapping part (e.g. the diagonal area in FIG. 24) in the left eye image and the right eye image, and the image overlapping part is cropped or superimposed to finally obtain the binocular image which meets the viewing angle range of the two eyes. For "the image (or binocular image) collected by the camera (or the electronic glasses) is sampled to obtain a left eye image and a right eye image", the left eye image which meets the left eye viewing angle range is directly sampled from left to right according to the viewing angle range of the left and right eyes, and the right eye image which meets the right eye viewing angle range is directly sampled from right to left.

[0282] In fact, due to the physical limitations of the camera lens and the like, the camera may capture a picture range greater than or less than the visual angle range observed by the human eye, that is, the picture range of the eyeglass picture displayed by the electronic eyeglasses may be greater than or less than the direct view picture observed by the human eye. As shown in FIG. 10, there are five trees in a row, and the direct view picture observed by the observer 41 without wearing electronic eyeglasses is picture 42; the eyeglass picture observed by the observer 41 with wearing electronic eyeglasses can be picture 43 with a picture range greater than the direct view picture, can be picture 42, or can be picture 44 with a picture range smaller than the direct view picture. It should be noted that the picture range shown in FIG. 10 includes the changes in the length and width of the picture, but in fact, the influencing factors of the picture range include at least one of the length, the width, and the picture center. That is, the difference in the picture range between the eyeglass picture and the direct view picture is reflected in the difference in the length, the width, and / or the picture center. That is, in the case of "sampling the picture (or called binocular picture) captured by the electronic eyeglasses from the picture (left eye picture) captured by the left eye camera and the picture (right eye picture) captured by the right eye camera" or "sampling the left eye picture and the right eye picture from the picture (or called binocular picture) captured by the camera (or called electronic eyeglasses)", the visual angle range used for sampling may not be the visual angle range corresponding to the human eye, but the visual angle range designed by the developer of the electronic eyeglasses, which is greater than or smaller than the visual angle range of the human eye. The embodiments of the present application do not limit this.

[0283] Step 830: based on the at least one external environment picture, performing a see-through display on the occlusion in the picture captured by the electronic eyeglasses to obtain an eyeglass picture with see-through content.

[0284] The external environment picture refers to the environment picture outside the vehicle, and the at least one external environment picture is used to indicate at least one of the external environment of the forward visual angle and the external environment of the non-forward visual angle. Optionally, the external environment picture includes the environment picture of the blind area.

[0285] Optionally, in the case that the collection view angle of the external environment picture is consistent with the view angle of the picture collected by the electronic glasses, it can be understood as the case that the camera parameter of the camera collecting the external environment picture is consistent with the camera parameter of the camera in the electronic glasses; the feature points in the picture collected by the electronic glasses and at least one external environment picture are matched to determine the perspective mode of the picture collected by the electronic glasses and at least one external environment picture; based on the perspective mode and at least one external environment picture, the occlusion in the picture collected by the electronic glasses is performed perspective display to obtain the glasses picture with perspective content. The perspective mode includes at least one of the perspective display of the obstacle in the picture collected by the electronic glasses, the perspective display of the picture collected by the electronic glasses, the perspective display of the obstacle in the external environment picture, the perspective display of the environment picture corresponding to the obstacle in the external environment picture (i.e. the environment picture of the blind area), and the perspective display of the external environment picture.

[0286] Optionally, in the case that the collection view angle of the external environment picture is inconsistent with the view angle of the picture collected by the electronic glasses, the superposition position of the external environment picture is determined; based on the superposition position of the external environment picture, the occlusion in the picture collected by the electronic glasses is performed perspective display to obtain the glasses picture with perspective content.

[0287] The picture range of the external environment picture is greater than, equal to, or less than the picture range of the picture collected by the electronic glasses. In the case that the picture range of the external environment picture is greater than the picture range of the picture collected by the electronic glasses, the position of the picture collected by the electronic glasses in the external environment picture is determined, the external environment picture is cropped to obtain the external environment picture with the picture range equal to that of the picture collected by the electronic glasses; based on at least one cropped external environment picture, the occlusion in the picture collected by the electronic glasses is performed perspective display to obtain the glasses picture with perspective content; or, in the case that the picture range of the external environment picture is greater than the picture range of the picture collected by the electronic glasses, based on at least one external environment picture, the occlusion in the picture collected by the electronic glasses is performed perspective display to obtain the glasses picture with perspective content, and the picture range of the glasses picture is equal to the picture range of the external environment picture. In the case that the picture range of the external environment picture is equal to the picture range of the picture collected by the electronic glasses, based on at least one external environment picture, the occlusion in the picture collected by the electronic glasses is performed perspective display to obtain the glasses picture with perspective content. In the case that the picture range of the external environment picture is less than the picture range of the picture collected by the electronic glasses, the position of the external environment picture in the picture collected by the electronic glasses is determined, and based on at least one external environment picture and its position in the picture collected by the electronic glasses, the occlusion in the picture collected by the electronic glasses is performed perspective display to obtain the glasses picture with perspective content.

[0288] In some embodiments, based on the at least one external environment picture, performing perspective display on the occlusion in the picture collected by the electronic glasses to obtain a glasses picture with perspective content, including: performing perspective display on the part of the area corresponding to the occlusion in the picture collected by the electronic glasses to obtain a processed picture; according to the position of the occlusion, superimposing the at least one external environment picture and the processed picture to obtain a glasses picture with perspective content. Perspective display refers to reducing the opacity of all or part of the area in the picture collected by the electronic glasses; or, perspective display refers to increasing the transparency of all or part of the area in the picture collected by the electronic glasses. Wherein, the part of the area refers to the area corresponding to the occlusion that needs to be perspective.

[0289] Optionally, when superimposing the at least one external environment picture and the processed picture, the layer corresponding to the at least one external environment picture is lower than the layer corresponding to the picture collected by the electronic glasses; or, the layer corresponding to the at least one external environment picture is higher than the layer corresponding to the processed picture; or, the layer corresponding to part of the external environment picture is lower than the layer corresponding to the processed picture, and the layer corresponding to part of the external environment picture is higher than the layer corresponding to the processed picture. For example, as shown in FIG. 25, the layer corresponding to the above-mentioned perspective content is lower than the layer corresponding to the processed picture; the layer corresponding to the above-mentioned rearview mirror content is higher than the layer corresponding to the processed picture, the layer corresponding to the rearview mirror extension content is higher than or equal to the layer corresponding to the processed picture; the layer corresponding to the above-mentioned virtual warning element is higher than the layer corresponding to the rearview mirror content and the rearview mirror extension content. In combination with FIG. 25, the layers from low to high are: the layer corresponding to the external environment picture of the forward view angle (the environment picture of the blind area, the perspective content), the layer corresponding to the processed picture, the layer corresponding to the external environment picture of the non-forward view angle (the rearview mirror content, the rearview mirror extension content), and the layer corresponding to the virtual warning element.

[0290] Wherein, in the case that the first layer is lower than the second layer, the picture content of the first layer will be occluded by the picture content of the second layer, if the picture content of the second layer is provided with transparency, the picture content of the first layer can be displayed by perspective display of the picture content of the second layer; the higher the transparency of the picture content in the second layer, the clearer the picture content of the first layer displayed after the final superposition, the lower the transparency of the picture content in the second layer, the more blurred the picture content of the first layer displayed after the final superposition. It should be further pointed out that the description of the first and the second is only for the differentiation of the same type of data, and does not limit the order.

[0291] In summary, the method provided by the embodiments of the present application shows how to obtain the eyeglass picture with the perspective content in the case that the auxiliary data includes at least one external environment picture, superimposes the external environment picture and the picture collected by the electronic eyeglasses, and can make the environment picture of the blind area included in the external environment picture appear in the eyeglass picture, and also can make the environment picture inside the vehicle in the picture collected by the electronic eyeglasses appear in the eyeglass picture, which not only ensures the target of eliminating the blind area, but also enables the wearer to better adapt to the eyeglass picture with the perspective content.

[0292] Further, the eyeglass picture is obtained by picture superposition instead of picture fusion, without the need to excessively align and calibrate the partial picture content, so as to reduce the difficulty of obtaining the eyeglass picture and enable the electronic eyeglasses with poor performance to also implement the auxiliary driving method.

[0293] In addition, the picture collected by the electronic eyeglasses is processed instead of processing the external environment or other, because the picture is collected by the electronic eyeglasses itself, without the need to obtain the picture through multiple layers of communication connection, and the picture has low acquisition delay and good real-time performance. The real-time performance of the eyeglass picture can be ensured, and the driving risk caused by the display delay can be reduced.

[0294] FIG. 23 shows the overall flowchart of the picture collection method provided by an example embodiment of the present application. The method is executed by the vehicle device, and the method further includes:

[0295] Step 910: Obtain the environment picture collected by at least one camera through the communication connection.

[0296] In some embodiments, the step 910 is executed before the above-mentioned step 720.

[0297] Optionally, the environment picture collected by the at least one camera is collected by the at least one camera at the same time; or, the environment picture collected by the at least one camera is collected by the at least one camera at different times.

[0298] Optionally, the at least one camera includes the camera facing the blind area of the vehicle and the camera for assisting in reconstruction. The camera for assisting in reconstruction is similar to the camera facing the blind area of the vehicle, and is used to collect the environment picture. The collected environment picture will be used to construct the virtual environment. The camera for assisting in reconstruction has a different direction from the camera facing the blind area of the vehicle, and is used to supplement the environment picture that is not collected by the camera facing the blind area of the vehicle.

[0299] Step 920: Construct the virtual environment based on the environment picture collected by the at least one camera.

[0300] Optionally, the at least one environment picture captured by the camera is spliced to construct a virtual environment. That is, feature points in the at least one environment picture captured by the camera are determined; the feature points in each environment picture are matched to determine a mapping relationship of each environment picture; and the at least one environment picture is fused based on the mapping relationship of each environment picture to obtain a virtual environment, which is a panoramic view obtained by fusing the at least one environment picture.

[0301] In some embodiments, the at least one environment picture captured by the camera is input into a three-dimensional reconstruction model to obtain a virtual environment, the three-dimensional reconstruction model being a machine learning model for generating a virtual three-dimensional model corresponding to the input picture according to the picture.

[0302] For example, the three-dimensional reconstruction model is based on a structured light principle for three-dimensional reconstruction; or, the three-dimensional reconstruction model is based on a triangulation principle for three-dimensional reconstruction; or, the three-dimensional reconstruction model is based on a binocular vision principle for three-dimensional reconstruction; or, the three-dimensional reconstruction model is based on a deep learning method for three-dimensional reconstruction. The model structure and reconstruction principle of the three-dimensional reconstruction model are not limited in the embodiments of the present application. For example, the three-dimensional reconstruction model is based on 3D Gaussian Splatting for three-dimensional reconstruction.

[0303] Step 930: At least one external environment picture is captured from the virtual environment.

[0304] Optionally, the vehicle device constructs a virtual camera, and adjusts camera parameters of the virtual camera to capture the at least one external environment picture.

[0305] Optionally, the three-dimensional reconstruction model is also used for rendering, that is, the three-dimensional reconstruction model is also used for capturing the at least one external environment picture from the virtual environment.

[0306] In some embodiments, step 930 can be implemented as step 931 and step 932. Step 931 can be performed before or simultaneously with step 910, or can be exchanged with step 920 or performed simultaneously.

[0307] Step 931: Obtain camera parameters of the electronic glasses through a communication connection.

[0308] The camera parameters include at least one of position information, a rotation attitude, and zoom information.

[0309] Optionally, the camera parameters are all or part of parameters in a Transform matrix of the electronic glasses.

[0310] Step 932: Based on the camera parameters of the electronic glasses, at least one external environment picture is captured from the virtual environment.

[0311] Based on the camera parameters of the electronic glasses, a virtual camera is constructed in the virtual environment, the camera parameters of the virtual camera are consistent with the camera parameters of the electronic glasses; and at least one external environment picture is collected from the virtual environment through the virtual camera.

[0312] Step 940: sending the at least one external environment picture to the electronic glasses through the communication connection.

[0313] The auxiliary data includes the at least one external environment picture, that is, step 730 can be implemented as step 940.

[0314] Optionally, the at least one external environment picture is sent to the electronic glasses after performing picture compression; or the at least one external environment picture is directly sent to the electronic glasses.

[0315] In summary, the method provided by the embodiments of the present application shows how to obtain the at least one external environment picture when the auxiliary data includes the at least one external environment picture, and the method of reconstructing the virtual environment and then collecting the external environment picture from the virtual environment according to the camera parameters of the electronic glasses can ensure that the collected external environment picture is the same view angle as the external environment picture seen when the electronic glasses are not worn, thereby laying a foundation for superimposing the external environment picture and the picture collected by the electronic glasses.

[0316] In addition, the three-dimensional reconstruction is used instead of directly splicing the images, which can reduce the image distortion caused by the physical reasons of the camera, avoid the misjudgment of the wearer on the distance, angle, etc. due to the distorted images displayed in the glasses picture, and thus improve the safety of driving.

[0317] Method two: the electronic glasses synthesize the external environment picture and the driver picture with perspective content (the auxiliary data at least includes the at least one environment picture collected by the camera).

[0318] FIG. 26 shows a flowchart of an auxiliary driving method provided by an example embodiment of the present application. The method is performed by the electronic glasses, which can be the electronic glasses in the computer system described above. The method includes:

[0319] In some embodiments, step 1031 can be performed before or simultaneously with step 1010; step 1031 can be exchanged with step 1020 or performed simultaneously; step 1041 can be performed before or simultaneously with step 1010; step 1041 can be performed between step 1010 and step 1020; step 1041 can also be performed simultaneously with step 1020; and step 1041 can be performed before or simultaneously with step 1031.

[0320] Step 1010: obtaining the at least one environment picture collected by the camera through the communication connection.

[0321] Step 1020: constructing a virtual environment based on the at least one environment picture captured by the camera.

[0322] Step 1030: obtaining at least one external environment picture from the virtual environment.

[0323] In some embodiments, step 1030 can be implemented as step 1031 and step 1032.

[0324] Step 1031: obtaining camera parameters of the electronic glasses.

[0325] The camera parameters include at least one of position information, rotation posture and zoom information.

[0326] Step 1032: obtaining at least one external environment picture from the virtual environment based on the camera parameters of the electronic glasses.

[0327] The steps 1010 to 1030 are similar to the steps 910 to 930 described above, except for the difference in the execution subject, which will not be repeated here.

[0328] Step 1040: obtaining a glasses picture with perspective content based on the at least one external environment picture.

[0329] In some embodiments, step 1040 can be implemented as step 1041 and step 1042.

[0330] Step 1041: obtaining a picture captured by the electronic glasses.

[0331] Step 1042: performing perspective display on the occlusion in the picture captured by the electronic glasses based on the at least one external environment picture to obtain a glasses picture with perspective content.

[0332] In some embodiments, performing perspective display on the occlusion in the picture captured by the electronic glasses based on the at least one external environment picture to obtain a glasses picture with perspective content includes: performing perspective display on a part of the picture corresponding to the occlusion to obtain a processed picture; and superimposing the at least one external environment picture and the processed picture according to the position of the occlusion to obtain the glasses picture with perspective content.

[0333] The step 1040 is similar to the steps 820 and 830 described above, which will not be repeated here.

[0334] In summary, the method provided by the embodiment of the application is responsible for the synthesis of the entire perspective content by the electronic glasses, and the vehicle only provides the environment picture collected by at least one camera, so that the transmission times of the above data, such as the camera parameters of the electronic glasses and the picture collected by the electronic glasses, can be reduced, the time delay of data provision is minimized, and the real-time performance of the displayed glasses picture is better.

[0335] Method three: the vehicle terminal synthesizes the external environment picture and the driver picture with perspective content (the auxiliary data at least includes the driver picture with perspective content).

[0336] FIG. 27 shows the overall flowchart of the auxiliary driving method provided by an example embodiment of the application. The method is performed by the electronic glasses, and the method further includes:

[0337] Step 1110: at least one of the camera parameters of the electronic glasses and the picture collected by the electronic glasses is transmitted to the vehicle device through the communication connection.

[0338] The camera parameters are used to determine at least one external environment picture, the picture collected by the electronic glasses is used to be superimposed with the at least one external environment picture to obtain the glasses picture with perspective content, and the camera parameters include at least one of position information, rotation posture and zoom information.

[0339] Optionally, the camera parameters of the electronic glasses and the picture collected by the electronic glasses are simultaneously transmitted to the vehicle device through the communication connection, or the camera parameters of the electronic glasses and the picture collected by the electronic glasses are sequentially transmitted to the vehicle device through the communication connection, or the camera parameters of the electronic glasses are transmitted to the vehicle device through the communication connection when a request of the vehicle device for the camera parameters is received, or the picture collected by the electronic glasses is transmitted to the vehicle device through the communication connection when a request of the vehicle device for the picture collected by the electronic glasses is received.

[0340] Step 1120: the glasses picture with perspective content is acquired through the communication connection.

[0341] In summary, the method provided by the embodiment of the application only needs to provide the camera parameters and the picture collected by the electronic glasses to the vehicle device, so that the glasses picture with perspective content synthesized by the vehicle device can be acquired, the demand for the processing performance of the electronic glasses is reduced, and the electronic glasses with a lower configuration can also display the glasses picture with perspective content.

[0342] FIG. 27 shows the overall flowchart of the picture collection method provided by an example embodiment of the application. The method is performed by the vehicle device, and the method further includes:

[0343] In some embodiments, step 1231 can be performed before or simultaneously with step 1210; step 1231 can be performed between step 1210 and step 1220; step 1231 can be performed simultaneously with step 1220; step 1241 can be performed before or simultaneously with step 1210; step 1241 can be performed between step 1210 and step 1220; step 1241 can be performed simultaneously with step 1220; step 1241 can be performed simultaneously with step 1231.

[0344] Step 1210: Obtain at least one environment picture captured by a camera through a communication connection.

[0345] Step 1220: Construct a virtual environment based on the at least one environment picture captured by the camera.

[0346] Step 1230: Capture at least one external environment picture from the virtual environment.

[0347] In some embodiments, step 1230 can be implemented as step 1231 and step 1232.

[0348] Step 1231: Obtain camera parameters of the electronic glasses through a communication connection.

[0349] Step 1232: Capture at least one external environment picture from the virtual environment based on the camera parameters of the electronic glasses.

[0350] Wherein, steps 1210 to 1230 are similar to steps 910 to 930 described above, and will not be repeated here.

[0351] Step 1240: Obtain a glasses picture with perspective content based on the at least one external environment picture.

[0352] In some embodiments, step 1240 can be implemented as step 1241 and step 1242.

[0353] Step 1241: Obtain a picture captured by the electronic glasses through a communication connection.

[0354] Step 1242: Perform perspective display on an occlusion in the picture captured by the electronic glasses based on the at least one external environment picture, to obtain a glasses picture with perspective content.

[0355] In some embodiments, based on the at least one external environment picture, performing perspective display on the occlusion in the picture collected by the electronic glasses to obtain a glasses picture with perspective content, comprising: performing perspective display on the part of the area corresponding to the occlusion in the picture collected by the electronic glasses to obtain a processed picture; according to the position of the occlusion, superimposing the at least one external environment picture and the processed picture to obtain the glasses picture with perspective content.

[0356] Step 1240 is similar to steps 820 and 830 described above, except that the execution subject is different, which will not be described here.

[0357] Step 1250: sending the glasses picture with perspective content to the electronic glasses through the communication connection.

[0358] In summary, the method provided by the embodiments of the present application can reduce the performance requirements of the electronic glasses by the carrier device to synthesize the glasses picture with perspective content. Moreover, since the processing performance of the carrier device is usually high, the synthesis result can be obtained more quickly when synthesizing the glasses picture, even if there is a certain time delay due to transmission speed. Compared with processing by the electronic glasses with poor performance, the total time delay can be guaranteed to be low, the viewing experience of the wearer can be improved, and the real-time performance of the glasses picture can be ensured.

[0359] Next, taking a car as the vehicle and the wearer as the driver as an example. The user can eliminate the blind area caused by the vehicle body structure or the driving road conditions during driving by wearing electronic glasses.

[0360] 1. Open the perspective view of the electronic glasses (open the auxiliary driving mode)

[0361] The user sets to open or close the auxiliary driving mode on the electronic glasses. After opening the auxiliary driving mode, the picture observed by the user through the electronic glasses is a picture in the perspective view, which can be called a perspective picture. The perspective picture can also be called a glasses picture with perspective content. The perspective picture is a picture showing the outside of the vehicle physically occluded by the vehicle body in the normal picture. The normal picture refers to the picture observed by the user when not wearing electronic glasses or not opening the auxiliary driving mode of the electronic glasses.

[0362] For example, when the auxiliary driving mode is not turned on, or in other words, when the auxiliary driving mode is turned off, the user observes the scene through the electronic glasses as shown in (1) of FIG. 9, and in this case, there is a vehicle blind area. After the auxiliary driving mode is turned on, the user observes the scene through the electronic glasses as shown in (2), (3) or (4) of FIG. 9, and in this case, the vehicle is displayed as fully transparent or semi-transparent, so that the user can directly observe the scene in the vehicle blind area, or in other words, the user can directly observe the scene outside the vehicle, which is equivalent to eliminating the vehicle blind area.

[0363] For the entire vehicle body, the perspective scene after the auxiliary driving mode is turned on is to make the vehicle body transparent, so that the user can directly observe the scene blocked by the vehicle body, as shown in (2), (3) or (4) of FIG. 9. For the outside rearview mirror (including the left and right rearview mirrors), the perspective scene is to make the vehicle body displayed in the outside rearview mirror transparent, and the perspective scene when the auxiliary driving mode is not turned on is shown in (1) of FIG. 15, and the perspective scene when the auxiliary driving mode is turned on is shown in (2) of FIG. 15. For the inside rearview mirror, if the inside rearview mirror displays a vehicle body scene, the perspective scene is to make the vehicle body displayed in the inside rearview mirror transparent.

[0364] 2. Complete the viewing angle (expand the scene, extend the scene)

[0365] For the vehicle blind area and the rearview mirror blind area caused by the small observation range of the rearview mirror, after the auxiliary driving mode is turned on, the electronic glasses can display an expanded scene, which can be referred to as a glasses scene with rearview mirror extension. When the electronic glasses detect that the user is looking at the rearview mirror (usually using eye movement recognition technology, when the user's eyeball is directed to the rearview mirror area, it is determined that the user is looking at the rearview mirror), the expanded scene is displayed around the rearview mirror, and the expanded scene is an expansion of the scene displayed in the rearview mirror, which can be understood as expanding the scene range of the scene displayed in the rearview mirror. As shown in FIG. 15 and FIG. 17, FIG. 15 is an expanded scene for the outside rearview mirror, when the auxiliary driving mode is not turned on, the user can observe a small scene range through the outside rearview mirror, as shown in (1) of FIG. 15; when the auxiliary driving mode is turned on and the expanded scene is displayed, the expanded scene is displayed around the rearview mirror, as shown in (3) or (4) of FIG. 15. Alternatively, the scene in the outside rearview mirror is displayed as a perspective scene, as shown in (3) of FIG. 15; or, the scene in the outside rearview mirror is a non-perspective scene, as shown in (4) of FIG. 15. FIG. 17 is an expanded scene for the inside rearview mirror, which will not be described here.

[0366] Optionally, the display range of the expanded picture is manually set by the user; or, the display range of the expanded picture is fixed. For example, the user sets the display range of the expanded picture through a pre-agreed gesture; or, the user sets the display range of the expanded picture through a voice instruction.

[0367] 3. Obstacle / hazard / abnormality prompt

[0368] The electronic glasses display a warning picture when there is an obstacle or an abnormality in the warning area of the vehicle; or, the electronic glasses display a warning picture when there is an obstacle or an abnormality on the predicted driving route of the vehicle. The abnormality includes an abnormal pedestrian and an abnormal vehicle. The abnormal pedestrian is, for example, a pedestrian suddenly appearing in the warning area, a pedestrian in the warning area when the vehicle starts, and a pedestrian on the predicted driving route. The abnormal vehicle is, for example, a vehicle suddenly appearing in the warning area, a vehicle in the warning area when the vehicle starts, and a vehicle on the predicted driving route. The predicted driving route refers to the driving route of the vehicle within a predicted distance or a predicted time, which is predicted by the vehicle or the electronic glasses based on at least one of the driving habits recorded by the vehicle and the current driving speed, the current deflection angle of the steering wheel of the vehicle.

[0369] The warning picture is used to indicate at least one of the position of the obstacle, the position of the abnormality, the type of the obstacle, the type of the abnormality, the distance to the obstacle, and the distance to the abnormality. Optionally, the warning picture is a picture marking the obstacle, and / or, the warning picture is a picture marking the abnormality, and / or, the warning picture is a picture marking the type of the obstacle, and / or, the warning picture is a picture marking the type of the abnormality, and / or, the warning picture is a picture marking the distance to the obstacle, and / or, the warning picture is a picture marking the distance to the abnormality. The warning picture includes at least one of a virtual warning element and a virtual obstacle avoidance element.

[0370] Optionally, the electronic glasses display an obstacle avoidance picture when there is an obstacle or an abnormality in the warning area of the vehicle; or, the electronic glasses display an obstacle avoidance picture when there is an obstacle or an abnormality on the predicted driving route of the vehicle. The obstacle avoidance picture is used to indicate the user to adjust the driving route, and / or, the obstacle avoidance picture is used to indicate the user to avoid the obstacle by a driving method, and / or, the obstacle avoidance picture is used to indicate the user to avoid the abnormality by a driving method.

[0371] Optionally, the warning area of the vehicle is not blocked in the electronic glasses; or, the warning area of the vehicle is blocked in the electronic glasses. The predicted driving route of the vehicle is not blocked in the electronic glasses; or, the predicted driving route of the vehicle is blocked in the electronic glasses. If there is an obstacle or abnormal situation in the warning area of the vehicle, and the area where the obstacle or abnormal situation is located is blocked in the picture displayed by the electronic glasses, a partial perspective picture in the area corresponding to the obstacle or abnormal situation is displayed; or, if there is an obstacle or abnormal situation in the predicted driving route of the vehicle, and the area where the obstacle or abnormal situation is located is blocked in the picture displayed by the electronic glasses, a partial perspective picture in the area corresponding to the obstacle or abnormal situation is displayed. That is, for the area where the obstacle or abnormal situation exists and is blocked, the perspective processing of the blocked area is performed.

[0372] Optionally, in the case that the electronic glasses display a warning picture or an obstacle avoidance picture, the electronic glasses give a voice prompt for the obstacle or abnormal situation. The content of the voice prompt includes the position of the obstacle, the position of the abnormal situation, the type of the obstacle, the type of the abnormal situation, the distance to the obstacle, and the distance to the abnormal situation.

[0373] As shown in FIG. 19, a child suddenly appears in the A-pillar blind area of the vehicle. The vehicle detects that the child is located in the warning area (i.e., the A-pillar blind area) of the vehicle, and sends the warning information to the electronic glasses. The electronic glasses perform perspective processing on the area where the child appears in the warning area, and display a warning picture in the range of the warning area. The warning picture includes a highlighted warning icon. At the same time, the electronic glasses also give a voice prompt “a child is passing from the left front, please pay attention to the speed and distance”.

[0374] 4. Reserved area setting

[0375] In the case that the user expects to be able to perform perspective on part of the area in the vehicle, the user can set a reserved area by himself / herself. After starting the auxiliary driving mode, the reserved area is displayed as non-transparent; or, the transparency of the reserved area is lower than that of the non-reserved area; or, the non-transparency of the reserved area is higher than that of the non-reserved area.

[0376] Optionally, the setting of the reserved area can be started by a fixed gesture; or, the setting of the reserved area can be started by a voice instruction; or, the setting of the reserved area can be started by a physical button; or, the setting of the reserved area can be started by a virtual button.

[0377] When the user sets the reserved area, the electronic glasses perform image recognition on the selection range of the user, and automatically fit to obtain the reserved area.

[0378] After the user sets the reserved area, if an obstacle or an abnormal situation appears in the warning area corresponding to the reserved area, the reserved area is made transparent. Optionally, a warning picture is displayed in the area corresponding to the reserved area.

[0379] As shown in FIG. 21, the user can set the reserved area by gestures. After confirming the reserved area, the reserved area is kept visible in the range of the picture seen by the user. The visibility of the reserved area is cancelled only when an obstacle or an abnormal situation appears in the warning area corresponding to the reserved area, and a warning icon is displayed.

[0380] Please refer to FIG. 28, which shows a structural block diagram of an auxiliary driving device provided in an example embodiment of the present application. The device has the functions of implementing the above-mentioned example of the auxiliary driving method, which can be implemented by hardware or executed by corresponding software implemented by hardware. The device can be the electronic glasses introduced above, or can be arranged in the electronic glasses, which are the electronic glasses worn by the wearer inside the vehicle. As shown in FIG. 28, the device can include a first communication module 1310, a first acquisition module 1320 and a first display module 1330.

[0381] The first communication module 1310 is configured to establish a communication connection with a vehicle device, the vehicle device including at least one camera facing a blind area of the vehicle, the blind area being an external area that the wearer cannot observe due to an obstruction.

[0382] The first acquisition module 1320 is configured to acquire auxiliary data through the communication connection, the auxiliary data being based on an environment picture obtained by the at least one camera.

[0383] The first display module 1330 is configured to display a wearer picture with transparent content based on the auxiliary data, the transparent content being an environment picture of the blind area displayed after the obstruction is made transparent in a picture collected by the electronic glasses.

[0384] In some embodiments, the first display module 1330 is further configured to display the glasses picture with the transparent content based on the auxiliary data when a transparent condition is met, the transparent condition being a condition for triggering the display of the transparent content.

[0385] In some embodiments, the see-through condition comprises at least one of: a line of sight of the wearer is directed towards the occlusion; the occlusion appears in the eyeglass view; a human-computer interaction indication of the wearer requiring to see through the occlusion; presence of warning information in a warning area corresponding to the occlusion, the warning area being an area within a range of a side of the vehicle; presence of warning information in a predicted moving route corresponding to the occlusion, the predicted moving route being a route that the vehicle will move within a time threshold or a distance threshold.

[0386] In some embodiments, the see-through content further displays at least one of a semi-transparent full occlusion, a semi-transparent partial occlusion, a semi-transparent full blind area environment view, and a semi-transparent partial blind area environment view.

[0387] In some embodiments, the device further comprises a second display module.

[0388] The second display module is configured to display, in response to a zoom triggering operation on a first area view in the eyeglass view, an eyeglass view with zoomed content, wherein the first area view is partial view content in the eyeglass view, and the zoomed content is view content after zooming the first area view.

[0389] In some embodiments, the eyeglass view displays a rearview mirror content, the rearview mirror content being a partial view of an external area of a non-forward perspective of the wearer, and the device further comprises a third display module.

[0390] The third display module is configured to display, in response to a triggering operation on a view expansion mode of the rearview mirror content, an eyeglass view with rearview mirror extended content, the rearview mirror extended content being view content after extending the partial view in the rearview mirror content.

[0391] In some embodiments, the triggering operation on the view expansion mode comprises at least one of: an eyeball of the wearer is directed towards the rearview mirror content; a head of the wearer is directed towards the rearview mirror content; presence of warning information in a warning area corresponding to the rearview mirror extended content; a human-computer interaction indication of the wearer requiring to turn on the view expansion mode.

[0392] In some embodiments, the device further comprises an adjustment module.

[0393] The adjustment module is configured to adjust, in response to a zoom operation on the rearview mirror extended content, a view range of the rearview mirror extended content.

[0394] In some embodiments, the device further comprises a fourth display module.

[0395] The fourth display module is configured to display a virtual warning element in the glasses picture in response to a warning trigger operation, the virtual warning element being used to indicate picture content that needs attention of the wearer.

[0396] In some embodiments, the see-through content includes see-through content corresponding to a plurality of the occlusions, and the device further includes a fifth display module.

[0397] The fifth display module is configured to display a glasses picture with reserved picture content in response to a reservation operation for a partial occlusion, the reserved picture content being picture content with higher opacity than the see-through content and corresponding to the partial occlusion.

[0398] In some embodiments, the auxiliary data includes at least one external environment picture, the external environment picture being generated based on environment pictures captured by the at least one camera; and the first display module includes a first acquisition submodule and a first see-through submodule.

[0399] The first acquisition submodule is configured to acquire pictures captured by the electronic glasses.

[0400] The first see-through submodule is configured to perform see-through display on occlusions in the pictures captured by the electronic glasses based on the at least one external environment picture, to obtain a glasses picture with see-through content.

[0401] In some embodiments, the auxiliary data includes at least one environment picture captured by a camera; and the first display module further includes a first construction submodule, a first acquisition submodule, and a first determination submodule.

[0402] The first construction submodule is configured to construct a virtual environment based on the at least one environment picture captured by the camera.

[0403] The first acquisition submodule is configured to acquire the at least one external environment picture from the virtual environment.

[0404] The first determination submodule is configured to obtain the glasses picture with see-through content based on the at least one external environment picture.

[0405] In some embodiments, the first acquisition submodule includes a first acquisition unit and a first acquisition unit.

[0406] The first acquisition unit is configured to acquire camera parameters of the electronic glasses, the camera parameters including at least one of position information, a rotation attitude, and zoom information.

[0407] The first acquisition unit is configured to acquire camera parameters of the electronic glasses, the camera parameters including at least one of position information, a rotation attitude, and zoom information.

[0408] In some embodiments, the first construction submodule is further configured to input the environment picture captured by the at least one camera into a three-dimensional reconstruction model to obtain the virtual environment.

[0409] In some embodiments, the first determination submodule includes a second acquisition unit.

[0410] The second acquisition unit is configured to acquire a picture captured by the electronic glasses.

[0411] The first perspective submodule is further configured to perform perspective display on an occlusion in the picture captured by the electronic glasses based on the at least one external environment picture to obtain a glasses picture with perspective content.

[0412] In some embodiments, the first perspective submodule is further configured to perform perspective display on a part of an area corresponding to the occlusion in the picture captured by the electronic glasses to obtain a processed picture; and superimpose the at least one external environment picture and the processed picture according to a position of the occlusion to obtain the glasses picture with perspective content.

[0413] In some embodiments, the auxiliary data includes the glasses picture with perspective content; and the device further includes a first sending module.

[0414] The first sending module is configured to send at least one of a camera parameter of the electronic glasses and a picture captured by the electronic glasses to the vehicle device through the communication connection, the camera parameter being used to determine at least one external environment picture, the picture captured by the electronic glasses being used to superimpose the at least one external environment picture to obtain the glasses picture with perspective content, and the camera parameter including at least one of position information, a rotation attitude, and zoom information.

[0415] Referring to FIG. 29, a structural block diagram of a picture acquisition device according to an example embodiment of the present application is shown. The device has the functions of the above-described picture acquisition method example, which can be implemented by hardware or by execution of corresponding software by hardware. The device can be the vehicle device introduced above or can be arranged in the vehicle device. The vehicle device has a first communication connection with at least one camera facing a blind area of the vehicle, which refers to an external area that cannot be observed by the wearer due to occlusion.

[0416] The second communication module 1410 is configured to establish a second communication connection with the electronic glasses worn by the wearer inside the vehicle.

[0417] The first determining module 1420 is configured to determine auxiliary data based on an environment picture collected by the at least one camera; the environment picture is acquired based on the first communication connection, and the auxiliary data is used to enable the electronic glasses to display a glasses picture with perspective content, wherein the perspective content is an environment picture of a blind area presented after performing perspective display on the occlusion in a picture collected by the electronic glasses.

[0418] The second sending module 1430 is configured to send the auxiliary data to the electronic glasses through the second communication connection.

[0419] It should be noted that, in implementing the functions, the apparatuses provided in the above embodiments are only used as examples for the division of the above functional modules, and in actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions. In addition, the apparatuses and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is described in detail in the method embodiments, which will not be repeated here.

[0420] FIG. 30 shows a structural block diagram of electronic glasses provided in an example embodiment of the present application.

[0421] Generally, the electronic glasses include one or more processors 1501 and one or more memories 1502.

[0422] The processor 1501 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1501 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 1501 can also include a main processor and a coprocessor. The main processor is a processor for processing data in a wake-up state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1501 can be integrated with a GPU (Graphics Processing Unit) that is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, the processor 1501 can also include an AI (Artificial Intelligence) processor for processing machine learning-related computing operations.

[0423] The memory 1502 can include one or more computer-readable storage media. The computer-readable storage media can be non-transitory. The memory 1502 can also include high-speed random access memory and can include nonvolatile memory, such as one or more magnetic disk storage devices, optical storage devices, flash memory devices, or other nonvolatile solid-state storage devices. In some embodiments, the non-transitory computer-readable storage medium of the memory 1502 is used to store at least one computer program for being executed by the processor 1501 to implement the assisted driving method provided by the method embodiments of the present application.

[0424] In some embodiments, the electronic glasses can further optionally include a peripheral device interface 1503 and at least one peripheral device. The processor 1501, the memory 1502 and the peripheral device interface 1503 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1503 through a bus, a signal line or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 1504, a display screen 1505, a camera assembly 1506, an audio circuit 1507 and a power supply 1508.

[0425] The peripheral device interface 1503 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1501 and the memory 1502. In some embodiments, the processor 1501, the memory 1502 and the peripheral device interface 1503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1501, the memory 1502 and the peripheral device interface 1503 can be implemented on a separate chip or circuit board, and the present embodiment does not limit this.

[0426] The RF circuit 1504 is used to receive and send RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1504 communicates with communication networks and other communication devices through electromagnetic signals. The RF circuit 1504 converts electrical signals to electromagnetic signals for transmission, or vice versa. Optionally, the RF circuit 1504 includes antenna system, RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a code converter chipset, a user identity module card, and the like. The RF circuit 1504 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network, and / or a WiFi network. In some embodiments, the RF circuit 1504 can also include NFC (Near Field Communication) related circuitry, which is not limited in the present application.

[0427] The optical display component 1505 is used to display the picture content. The picture content can be real picture content, virtual picture content, or a combination of real picture content and virtual picture content. In some embodiments, the optical display component 1505 is mounted on the lens of the electronic glasses, and the electronic glasses further comprise a projector for projecting the picture content onto the optical display component 1505 to realize the display of the picture content; that is, the electronic glasses are transparent electronic glasses. In another embodiment, the optical display component 1505 is used to replace the lens of the electronic glasses, that is, the optical display component 1505 is mounted on the frame of the electronic glasses; that is, the electronic glasses are non-transparent electronic glasses. In some embodiments, the optical display component 1505 can be one, which is arranged on the frame or lens of the electronic glasses, and only one eye of the electronic glasses can display the picture content; or, the picture content of both eyes of the electronic glasses shares one optical display component 1505, that is, the optical display component 1505 of the electronic glasses is divided into left and right areas, and the picture content is displayed on both areas; in other embodiments, the optical display component 1505 can be at least two, which are arranged on the two frames or lenses of the electronic glasses; in other embodiments, the optical display component 1505 can be at least four, that is, the forward and backward optical display components 1505 are arranged on the two frames of the electronic glasses, respectively, the forward optical display component 1505 is used to observe the eye picture of the wearer by others, and the backward optical display component 1505 is used to watch the picture of the wearer by the wearer. In some embodiments, the optical display component 1505 can be a flexible display screen, that is, the optical display component 1505 on the electronic glasses has a certain curvature. Even, the optical display component 1505 can also be arranged in an irregular shape, that is, a special-shaped screen. The display screen 1505 can be made of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc.

[0428] The camera assembly 1506 is configured to capture images or videos. Optionally, the camera assembly 1506 includes a front-facing camera and a built-in camera. Generally, the front-facing camera is disposed at the front of the electronic glasses, and is configured to capture an electronic glasses view, which can be understood as capturing a view in front of the electronic glasses in a current wearing perspective. The electronic glasses view can be used to display a view in front of the wearer in the display screen 1505, and can also be used to recognize a gesture of the wearer, and the human-computer interaction with the electronic glasses is realized through the gesture. The built-in camera is disposed inside the electronic glasses, and is configured to capture an eye view of the wearer, which can be used to estimate a line of sight of the wearer. In addition, the eye view can also be used for iris recognition of the wearer to confirm identity information of the wearer. In another embodiment, the camera assembly 1506 includes a camera mounted below the glasses, which is configured to capture images or videos. In some embodiments, the camera assembly 1506 can also include a flash. The flash can be a single-color-temperature flash or a dual-color-temperature flash. The dual-color-temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0429] The audio circuit 1507 is configured to provide an audio interface between the user and the electronic glasses. The audio circuit 1507 can include a microphone and a speaker. The microphone is configured to capture sound waves of the user and the environment, and convert the sound waves into an electrical signal input to the processor 1501 for processing, or input to the radio frequency circuit 1504 to realize voice communication. For the purpose of stereo sound collection or noise reduction, the microphone can be multiple, and is respectively disposed at different parts of the electronic glasses. The microphone can also be an array microphone or an omnidirectional collection microphone. The speaker is configured to convert an electrical signal from the processor 1501 or the radio frequency circuit 1504 into sound waves. The speaker can be a traditional diaphragm speaker, or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert an electrical signal into a sound wave audible to humans, but also convert an electrical signal into a sound wave inaudible to humans for ranging purposes. In some embodiments, the audio circuit 1507 can also include a headphone jack.

[0430] The power supply 1508 is configured to supply power to each component in the electronic glasses. The power supply 1508 can be alternating current, direct current, disposable battery or rechargeable battery. When the power supply 1508 includes a rechargeable battery, the rechargeable battery can be a wired charging battery or a wireless charging battery. The wired charging battery is a battery charged through a wired line, and the wireless charging battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology and wireless charging technology.

[0431] In some embodiments, the electronic glasses further include one or more sensors 1509. The one or more sensors 1509 include, but are not limited to, a gesture sensor 1510, a gyroscope sensor 1511, a pressure sensor 1512, an optical sensor 1513, and a proximity sensor 1514.

[0432] The gesture sensor 1510 can detect the gesture action of the wearer to realize the human-computer interaction between the electronic glasses and the wearer.

[0433] The gyroscope sensor 1511 can detect the direction and rotation angle of the body of the electronic glasses. The gyroscope sensor 1511 can be used to collect the 3D action of the user to the electronic glasses. For example, the gyroscope sensor 1511 can be used to identify the head posture of the wearer.

[0434] The pressure sensor 1512 can be arranged at the side frame of the electronic glasses and / or the lower layer of the display screen 1505. When the pressure sensor 1512 is arranged at the side frame of the electronic glasses, the touch signal of the user to the electronic glasses can be detected, and the processor 1501 can perform the shortcut operation according to the touch signal collected by the pressure sensor 1512.

[0435] The optical sensor 1513 is used to collect the ambient light intensity. In one embodiment, the processor 1501 can control the display brightness of the optical display assembly 1505 according to the ambient light intensity collected by the optical sensor 1513. Specifically, when the ambient light intensity is high, the display brightness of the optical display assembly 1505 is increased; when the ambient light intensity is low, the display brightness of the optical display assembly 1505 is decreased. In another embodiment, the processor 1501 can also dynamically adjust the shooting parameter of the camera assembly 1506 according to the ambient light intensity collected by the optical sensor 1513.

[0436] The proximity sensor 1514, also known as the distance sensor, is usually arranged at the front of the electronic glasses. The proximity sensor 1514 is used to collect the distance between the user and the front of the electronic glasses.

[0437] Those skilled in the art can understand that the structure shown in FIG. 30 does not constitute a limitation on the electronic glasses, and can include more or fewer components than those shown, or combine certain components, or adopt different component arrangements.

[0438] FIG. 31 shows a structural block diagram of a carrier device according to an example embodiment of the present application.

[0439] Generally, the carrier device includes one or more processors 1601 and one or more memories 1602.

[0440] The processor 1601 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1601 can be implemented in at least one of a hardware form of a DSP, an FPGA, a PLA. The processor 1601 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also referred to as a CPU, and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 1601 can be integrated with a GPU, the GPU being responsible for rendering and drawing of content required to be displayed by the display screen. In some embodiments, the processor 1601 can further include an AI processor for processing computing operations related to machine learning.

[0441] The memory 1602 can include one or more computer-readable storage media that can be non-transitory. The memory 1602 can also include a high-speed random access memory, and a nonvolatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1602 is used to store at least one computer program for being executed by the processor 1601 to implement the picture acquisition method provided by the method embodiments in the present application.

[0442] In some embodiments, the vehicle device can further optionally include a peripheral device interface 1603 and at least one peripheral device. The processor 1601, the memory 1602 and the peripheral device interface 1603 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1603 through a bus, a signal line or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 1604, a display screen 1605, a camera assembly 1606, an audio circuit 1607 and a power supply 1608.

[0443] The peripheral device interface 1603 can be used to connect at least one I / O-related peripheral device to the processor 1601 and the memory 1602. In some embodiments, the processor 1601, the memory 1602 and the peripheral device interface 1603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1601, the memory 1602 and the peripheral device interface 1603 can be implemented on a separate chip or circuit board, and the present embodiment does not limit this.

[0444] The radio frequency circuit 1604 is configured to receive and send RF signals, also known as electromagnetic signals. The radio frequency circuit 1604 performs conversion between electrical signals and electromagnetic signals. The radio frequency circuit 1604 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 1604 converts electrical signals into electromagnetic signals for transmission, or converts electromagnetic signals received into electrical signals. Optionally, the radio frequency circuit 1604 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 1604 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network, and / or a WiFi network. In some embodiments, the radio frequency circuit 1604 can also include NFC related circuitry, which is not limited by the present application.

[0445] The display screen 1605 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 1605 is a touch display screen, the display screen 1605 also has the ability to collect touch signals on or above the surface of the display screen 1605. The touch signals can be input as control signals to the processor 1601 for processing. At this time, the display screen 1605 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1605 can be one, arranged on the front panel of the carrier device; in other embodiments, the display screen 1605 can be at least two, arranged on different surfaces of the carrier device or in a folding design; in some embodiments, the display screen 1605 can be a flexible display screen, arranged on a curved surface or a folding surface of the carrier device. Even, the display screen 1605 can also be arranged in an irregular shape, that is, a special-shaped screen. The display screen 1605 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0446] The camera assembly 1606 is configured to capture images or videos. Optionally, the camera assembly 1606 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the carrier device, and the rear camera is arranged on the back of the carrier device. In other embodiments, the camera assembly can be a camera mounted on a vehicle in communication connection with the carrier device. In some embodiments, the camera assembly 1606 can also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. The dual-color temperature flash refers to a combination of warm light flash and cold light flash, which can be used for light compensation at different color temperatures.

[0447] The audio circuit 1607 is configured to provide an audio interface between a user and the carrier device. The audio circuit 1607 can include a microphone and a speaker. The microphone is configured to collect sound waves from the user and the environment, and convert the sound waves into an electrical signal input to the processor 1601 for processing, or to the radio frequency circuit 1604 for voice communication. The microphone can be multiple microphones arranged at different positions of the carrier device for stereo sound collection or noise reduction. The microphone can also be an array microphone or an omnidirectional collection microphone. The speaker is configured to convert an electrical signal from the processor 1601 or the radio frequency circuit 1604 into sound waves. The speaker can be a traditional thin-film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals into sound waves that are audible to humans, or sound waves that are inaudible to humans for ranging purposes. In some embodiments, the audio circuit 1607 can further include a headphone jack.

[0448] The power supply 1608 is configured to supply power to various components in the carrier device. The power supply 1608 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1608 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. The wired rechargeable battery is a battery that is charged through a wired line, and the wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology and wireless charging technology.

[0449] In some embodiments, the carrier device further includes one or more sensors 1609. The one or more sensors 1609 include, but are not limited to, an acceleration sensor 1610, a gyroscope sensor 1611, a pressure sensor 1612, an optical sensor 1613, and a proximity sensor 1614.

[0450] The acceleration sensor 1610 can detect the acceleration in three coordinate axes of a coordinate system established by the carrier device.

[0451] The gyroscope sensor 1611 can detect the orientation and rotation angle of the carrier device. The gyroscope sensor 1611 can work with the acceleration sensor 1610 to collect 3D movements of the user on the carrier device.

[0452] The pressure sensor 1612 can be arranged at the side frame of the carrier device and / or the lower layer of the display screen 1605. When the pressure sensor 1612 is arranged at the side frame of the carrier device, the holding signal of the user to the carrier device can be detected, and the left-hand / right-hand recognition or shortcut operation is performed by the processor 1601 according to the holding signal collected by the pressure sensor 1612. When the pressure sensor 1612 is arranged at the lower layer of the display screen 1605, the operable control on the UI interface is controlled by the processor 1601 according to the pressure operation of the user to the display screen 1605. The operable control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0453] The optical sensor 1613 is used to collect the ambient light intensity. In an embodiment, the processor 1601 can control the display brightness of the display screen 1605 according to the ambient light intensity collected by the optical sensor 1613. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1605 is increased; when the ambient light intensity is low, the display brightness of the display screen 1605 is decreased. In another embodiment, the processor 1601 can also dynamically adjust the shooting parameter of the camera assembly 1606 according to the ambient light intensity collected by the optical sensor 1613.

[0454] The proximity sensor 1614, also known as a distance sensor, is usually arranged at the front of the carrier device. The proximity sensor 1614 is used to collect the distance between the user and the front of the carrier device.

[0455] Those skilled in the art can understand that the structure shown in FIG. 31 does not constitute a limitation on the carrier device, and can include more or fewer components than shown, or combine certain components, or adopt different component arrangements.

[0456] In an exemplary embodiment, a non-transitory computer readable storage medium having a computer program stored thereon is also provided, the computer program being executed by a processor to implement the above-mentioned assisted driving method, and / or, the picture collection method. For example, the computer readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0457] In an exemplary embodiment, a computer program product is also provided, the computer program product including computer instructions stored in a computer readable storage medium, and a processor reading and executing the computer instructions from the computer readable storage medium, for implementing the above-mentioned assisted driving method, and / or, the picture collection method.

[0458] It should be understood that "multiple" mentioned herein refers to two or more. The "and / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A existing alone, A and B existing together, and B existing alone. The character " / " generally represents that the associated objects before and after it are in an "or" relationship. In addition, the step numbers described herein only exemplarily show a possible execution order between steps, and in some other embodiments, the above steps can also be executed in a non-numbered order, such as two different numbered steps being executed simultaneously, or two different numbered steps being executed in an order opposite to that shown in the figure, and the embodiments of the present application are not limited in this regard.

[0459] The above only describes optional embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An assisted driving method, the method being performed by electronic glasses worn by a wearer inside a vehicle, the method comprising: establishing a communication connection with a vehicle device, the vehicle device comprising at least one camera towards a blind area of the vehicle, the blind area referring to an external area that the wearer cannot observe due to an obstruction; obtaining assistance data via the communication connection, the assistance data being based on an environment picture captured by the at least one camera; displaying a glasses picture with see-through content based on the assistance data, the see-through content being an environment picture of the blind area displayed in a see-through manner through the obstruction in a picture captured by the electronic glasses.

2. The method of claim 1, wherein, The displaying of the glasses picture with see-through content based on the assistance data comprises: displaying the glasses picture with see-through content based on the assistance data if a see-through condition is met, the see-through condition being a condition for triggering the display of the see-through content.

3. The method of claim 2, wherein, The see-through condition comprises at least one of: a line of sight of the wearer being towards the obstruction; the obstruction appearing in the glasses picture; a human-computer interaction indication of the wearer requiring to see through the obstruction; presence of warning information in a warning area corresponding to the obstruction, the warning area being an area within a range of sides of the vehicle; presence of the warning information in a predicted moving route corresponding to the obstruction, the predicted moving route being a route that the vehicle will move within a time threshold or a distance threshold.

4. The method according to any one of claims 1 to 3, wherein, The see-through content further displays the obstruction in a semi-transparent manner.

5. The method according to any one of claims 1 to 4, wherein, The method further comprises: displaying a glasses picture with zoom content in response to a zoom triggering operation on a first area picture in the glasses picture; wherein the first area picture is a partial picture content in the glasses picture, and the zoom content is a picture content after zooming the first area picture.

6. The method according to any one of claims 1 to 5, wherein, The glasses picture displays a rearview mirror content, the rearview mirror content being a partial picture of an external area that is not in a forward view of the wearer. The method further comprises: displaying a glasses picture with rearview mirror extended content in response to a triggering operation of a picture expansion mode on the rearview mirror content, the rearview mirror extended content being a picture content after extending a partial picture in the rearview mirror content.

7. The method of claim 6, wherein, The triggering operation of the picture expansion mode comprises at least one of: an eye of the wearer being towards the rearview mirror content; a head of the wearer being towards the rearview mirror content; presence of warning information in a warning area corresponding to the rearview mirror extended content; a human-computer interaction indication of the wearer requiring to turn on the picture expansion mode.

8. The method of claim 6 or 7, wherein, The method further comprises: adjusting a picture range of the rearview mirror extended content in response to a zoom operation on the rearview mirror extended content.

9. The method according to any one of claims 1 to 8, wherein, The method further comprises: displaying a virtual warning element in the glasses picture in response to a warning triggering operation, the virtual warning element being used to indicate a picture content that the wearer needs to pay attention to.

10. The method of any one of claims 1 to 9, wherein, The see-through content comprises a plurality of see-through contents corresponding to the obstructions, and the method further comprises: In response to the reservation operation for the partial occlusion, a glasses picture with reserved picture content is displayed, the reserved picture content being picture content with higher opacity than the see-through content and corresponding to the partial occlusion.

11. The method of any one of claims 1 to 10, wherein, The auxiliary data includes at least one external environment picture, the external environment picture being generated based on an environment picture captured by the at least one camera; The displaying of the glasses picture with see-through content based on the auxiliary data includes: capturing a picture by the electronic glasses; performing see-through display on an occlusion in the captured picture by the electronic glasses based on the at least one external environment picture to obtain the glasses picture with see-through content.

12. The method of any one of claims 1 to 10, wherein, The auxiliary data includes at least one environment picture captured by a camera; The displaying of the glasses picture with see-through content based on the auxiliary data includes: constructing a virtual environment based on the at least one environment picture captured by the camera; capturing the at least one external environment picture from the virtual environment; obtaining the glasses picture with see-through content based on the at least one external environment picture.

13. The method of claim 12, wherein, The capturing of the at least one external environment picture from the virtual environment includes: obtaining camera parameters of the electronic glasses, the camera parameters including at least one of position information, a rotation attitude, and zoom information; capturing the at least one external environment picture from the virtual environment based on the camera parameters of the electronic glasses.

14. The method of claim 12, wherein, The constructing of the virtual environment based on the at least one environment picture captured by the camera includes: inputting the at least one environment picture captured by the camera into a three-dimensional reconstruction model to obtain the virtual environment.

15. The method of claim 12, wherein, The obtaining of the glasses picture with see-through content based on the at least one external environment picture includes: capturing a picture by the electronic glasses; performing see-through display on an occlusion in the captured picture by the electronic glasses based on the at least one external environment picture to obtain the glasses picture with see-through content.

16. The method of claim 11 or 15, wherein, The performing of see-through display on the occlusion in the captured picture by the electronic glasses based on the at least one external environment picture to obtain the glasses picture with see-through content includes: performing see-through display on a partial region corresponding to the occlusion in the captured picture by the electronic glasses to obtain a processed picture; superimposing the at least one external environment picture and the processed picture according to a position of the occlusion to obtain the glasses picture with see-through content.

17. The method of any one of claims 1 to 16, wherein, The auxiliary data includes the glasses picture with see-through content; The method further includes: sending at least one of camera parameters of the electronic glasses and a captured picture by the electronic glasses to the carrier device through the communication connection, the camera parameters being used to determine at least one external environment picture, the captured picture by the electronic glasses being used to superimpose the at least one external environment picture to obtain the glasses picture with see-through content, the camera parameters including at least one of position information, a rotation attitude, and zoom information.

18. A picture acquisition method, the method being performed by a vehicle device, the vehicle device having a first communication connection with at least one camera facing a blind area of a vehicle, the blind area referring to an external area that a wearer cannot observe due to an obstruction, the method comprising: establishing a second communication connection with electronic glasses worn by the wearer inside the vehicle; determining auxiliary data based on an environment picture acquired by the at least one camera, the environment picture being obtained based on the first communication connection, the auxiliary data being used to cause the electronic glasses to display a glasses picture with see-through content, the see-through content being an environment picture of the blind area presented in a picture acquired by the electronic glasses after performing see-through display on the obstruction; sending the auxiliary data to the electronic glasses through the second communication connection.

19. The method of claim 18, wherein, The auxiliary data is used to cause the electronic glasses to display the glasses picture with see-through content when a see-through condition is met.

20. An auxiliary driving device, the device being performed by electronic glasses worn by a wearer inside a vehicle, the device comprising: a first communication module configured to establish a communication connection with a vehicle device, the vehicle device comprising at least one camera facing a blind area of the vehicle, the blind area referring to an external area that the wearer cannot observe due to an obstruction; a first obtaining module configured to obtain auxiliary data through the communication connection, the auxiliary data being obtained based on an environment picture acquired by the at least one camera; a first display module configured to display a glasses picture with see-through content based on the auxiliary data, the see-through content being an environment picture of the blind area presented in a picture acquired by the electronic glasses after performing see-through display on the obstruction.

21. A picture acquisition device, the device being performed by a vehicle device, the device having a first communication connection with at least one camera facing a blind area of a vehicle, the blind area referring to an external area that a wearer cannot observe due to an obstruction, the device comprising: a second communication module configured to establish a second communication connection with electronic glasses worn by the wearer inside the vehicle; a first determining module configured to determine auxiliary data based on an environment picture acquired by the at least one camera; the environment picture being obtained based on the first communication connection, the auxiliary data being used to cause the electronic glasses to display a glasses picture with see-through content, the see-through content being an environment picture of the blind area presented in a picture acquired by the electronic glasses after performing see-through display on the obstruction; a second sending module configured to send the auxiliary data to the electronic glasses through the second communication connection.

22. An electronic eyewear comprising: a processor and a memory, the memory storing at least one program; the processor is configured to execute the at least one program in the memory to implement the auxiliary driving method as claimed in any of claims 1 to 17.

23. A carrier device comprising: a processor and a memory, the memory storing at least one program; the processor is configured to execute the at least one program in the memory to implement the auxiliary driving method as claimed in any of claims 1 to 17. The processor is configured to execute the at least one program in the memory to implement the screen capturing method of claim 18 or 19. 24.A computer readable storage medium, having stored therein executable instructions, which are loaded and executed by a processor to implement the assisted driving method of any one of claims 1 to 17, and / or the screen capturing method of claim 18 or 19. 25.A computer program product, comprising computer instructions stored in a computer readable storage medium, which are read and executed by a processor to implement the assisted driving method of any one of claims 1 to 17, and / or the screen capturing method of claim 18 or 19.

Images