Auxiliary driving method, picture acquisition method, device, equipment, medium and product

Abstract

The invention discloses an auxiliary driving method, a picture acquisition method, a picture acquisition device, equipment, a medium and a product, and belongs to the field of auxiliary driving. The method comprises the steps that communication connection with carrier equipment is established, the carrier equipment comprises at least one camera facing a blind area of the vehicle, and the blind area refers to an external area which cannot be observed by a wearer due to shielding of a shielding object; obtaining auxiliary data through the communication connection, wherein the auxiliary data is obtained based on an environment picture collected by the at least one camera; a glasses picture with perspective content is displayed based on the auxiliary data, and the perspective content is an environment picture of the blind area presented after perspective display of the shelter in pictures collected by the electronic glasses. According to the method disclosed by the invention, the wearer can see through the shielding object in all directions after wearing the electronic glasses, and the effect of comprehensively eliminating the blind area is achieved, so that the driving safety is improved.

Description

Technical Field

[0001] This application relates to the field of driver assistance, and in particular to a driver assistance method, image acquisition method, device, equipment, medium and product. Background Technology

[0002] With the evolution of electronic products and the upgrading of computer software, vehicle driver assistance technologies have also developed, such as radar sensing, intelligent parking, high-speed cameras, and intelligent headlights. However, in actual driving, blind spots remain the biggest challenge for drivers.

[0003] Solutions to address blind spots in some vehicles have emerged in related technologies. These solutions often eliminate some blind spots by installing displays on the vehicle. For example, for the A-pillar blind spot, related technologies use a display embedded in the A-pillar to show the real-time images captured by the camera on the outside of the A-pillar; for the rearview mirror blind spot, two cameras are used to replace the rearview mirrors on both sides of the vehicle, and a display is installed on each side of the vehicle to show the images captured by the corresponding camera.

[0004] The above methods can eliminate some blind spots of vehicles by adding a display screen. However, any blind spot of a vehicle can lead to serious traffic accidents while driving. It is hoped that a method can be designed to completely eliminate vehicle blind spots. Summary of the Invention

[0005] This application provides a method for assisting driving, a method for capturing images, a device, equipment, a medium, and a product, the technical solution of which is as follows:

[0006] According to one aspect of this application, a driving assistance method is provided, the method being performed by electronic glasses worn by a wearer located inside a vehicle, the method comprising:

[0007] Establish a communication connection with a vehicle device, the vehicle device including at least one camera facing the blind spot of the vehicle, the blind spot being an external area that the wearer cannot observe due to obstruction;

[0008] Auxiliary data is acquired through the communication connection, and the auxiliary data is obtained based on environmental images captured by the at least one camera.

[0009] The auxiliary data display shows the glasses image with perspective content, which is the environmental image of the blind spot presented after the obstruction is displayed through the screen in the image captured by the electronic glasses.

[0010] According to one aspect of this application, a method for acquiring blind spot images is provided, the method being performed by a vehicle device having a first communication connection with at least one camera facing the blind spot of a vehicle, the blind spot referring to an external area that cannot be observed by a wearer due to obstruction, the method comprising:

[0011] A second communication connection is established with electronic glasses, which are worn by the wearer located inside the vehicle;

[0012] Based on the environmental images captured by the at least one camera, auxiliary data is determined; the environmental images are obtained based on the first communication connection, and the auxiliary data is used to enable the electronic glasses to display a glasses image with see-through content, wherein the see-through content is the environmental image of the blind spot presented after the obstruction is displayed through the image captured by the electronic glasses;

[0013] The auxiliary data is sent to the electronic glasses via the second communication connection.

[0014] According to one aspect of this application, a driver assistance device is provided, the device being executed by electronic glasses worn by a wearer located inside a vehicle, the device comprising:

[0015] A first communication module is used to establish a communication connection with a vehicle device, the vehicle device including at least one camera facing the blind spot of the vehicle, the blind spot being an external area that the wearer cannot observe due to obstruction by an obstruction.

[0016] The first acquisition module is used to acquire auxiliary data through the communication connection, the auxiliary data being obtained based on environmental images captured by the at least one camera;

[0017] The first display module is used to display a glasses image with perspective content based on the auxiliary data display. The perspective content is the environmental image of the blind spot presented after the obstruction is displayed in the image captured by the electronic glasses.

[0018] According to one aspect of this application, a blind spot image acquisition device is provided, the device being executed by a vehicle device, the device having a first communication connection with at least one camera facing the blind spot of the vehicle, the blind spot referring to an external area that cannot be observed by a wearer due to obstruction, the device comprising:

[0019] A second communication module is used to establish a second communication connection with electronic glasses, which are worn by the wearer located inside the vehicle;

[0020] The first determining module is used to determine auxiliary data based on the environmental images captured by the at least one camera; the environmental images are obtained based on the first communication connection, and the auxiliary data is used to enable the electronic glasses to display a glasses image with see-through content, wherein the see-through content is the environmental image of the blind spot presented after the obstruction is displayed through the image captured by the electronic glasses;

[0021] The second transmitting module is used to transmit the auxiliary data to the electronic glasses via the second communication connection.

[0022] According to one aspect of this application, an electronic pair of glasses is provided, the electronic pair of glasses comprising: a processor and a memory, the memory storing at least one program; the processor being configured to execute the at least one program in the memory to implement the above-described assisted driving method.

[0023] According to one aspect of this application, a vehicle device is provided, the vehicle device comprising: a processor and a memory, the memory storing at least one program; the processor being configured to execute the at least one program in the memory to implement the above-described method for acquiring blind spot images.

[0024] According to one aspect of this application, a computer-readable storage medium is provided, wherein executable instructions are stored in the computer-readable storage medium, the executable instructions being loaded and executed by a processor to implement the above-described assisted driving method, and / or, the blind spot image acquisition method.

[0025] According to one aspect of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium, wherein a processor reads from the computer-readable storage medium and executes the computer instructions to implement the above-described assisted driving method and / or, blind spot image acquisition method.

[0026] The beneficial effects of the technical solution provided in this application include at least the following:

[0027] The electronic glasses, based on a communication connection with the vehicle, acquire auxiliary data determined from environmental images captured by at least one camera. Using this auxiliary data, they display a see-through view through the glasses, allowing the wearer to directly see through blind spots obstructing the view. This eliminates blind spots previously unseen, thus improving driving safety. The definition of the see-through view is further illustrated: based on the image captured by the electronic glasses, obstructions (objects appearing in front of the wearer's eyes after wearing the glasses) are displayed through the glasses, revealing the environmental view of the blind spots. This allows the wearer to drive based on the see-through view displayed on the glasses, improving driving safety and minimizing the incidence of traffic accidents caused by blind spots. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A schematic diagram of vehicle blind spots in related technologies is shown;

[0030] Figure 2 A schematic diagram of a car structure in the related art is shown;

[0031] Figure 3 A schematic diagram of the blind spot of a large vehicle in the related technology is shown;

[0032] Figure 4 A schematic diagram of the blind spot of a large vehicle in the related technology is shown;

[0033] Figure 5 A schematic diagram of the overtaking blind spot in the relevant technology is shown;

[0034] Figure 6 A schematic diagram of the inner wheel difference blind zone in the related technology is shown;

[0035] Figure 7 An architectural diagram of a computer system provided in an exemplary embodiment of this application is shown;

[0036] Figure 8 A flowchart of an exemplary embodiment of the driving assistance method provided in this application is shown;

[0037] Figure 9A schematic diagram of an exemplary embodiment of the driving assistance method provided in this application is shown;

[0038] Figure 10 A schematic diagram showing the visible range provided by an exemplary embodiment of this application is illustrated;

[0039] Figure 11 A schematic diagram of an exemplary embodiment of the driving assistance method provided in this application is shown;

[0040] Figure 12 A flowchart of an exemplary embodiment of the driving assistance method provided in this application is shown;

[0041] Figure 13 A flowchart of an exemplary embodiment of the driving assistance method provided in this application is shown;

[0042] Figure 14 This illustration shows a schematic diagram of scaled content provided in an exemplary embodiment of this application;

[0043] Figure 15 A schematic diagram illustrating the rearview mirror content and extended content provided in an exemplary embodiment of this application is shown;

[0044] Figure 16 A flowchart of an exemplary embodiment of the driving assistance method provided in this application is shown;

[0045] Figure 17 A schematic diagram illustrating the rearview mirror content and extended content provided in an exemplary embodiment of this application is shown;

[0046] Figure 18 A flowchart of an exemplary embodiment of the driving assistance method provided in this application is shown;

[0047] Figure 19 A schematic diagram of a glasses screen with virtual warning elements provided in an exemplary embodiment of this application is shown;

[0048] Figure 20 A flowchart of an exemplary embodiment of the driving assistance method provided in this application is shown;

[0049] Figure 21 This illustration shows a schematic diagram of the retained screen content provided in an exemplary embodiment of this application;

[0050] Figure 22 A flowchart illustrating a blind spot image acquisition method provided in an exemplary embodiment of this application is shown;

[0051] Figure 23 A flowchart illustrating the assisted driving method and blind spot image acquisition method provided in an exemplary embodiment of this application is shown.

[0052] Figure 24 This illustration shows a schematic diagram of the visible range of the human eye provided in an exemplary embodiment of this application;

[0053] Figure 25 This illustration shows a schematic diagram of the hierarchical structure of the glasses screen provided in an exemplary embodiment of this application;

[0054] Figure 26 A flowchart illustrating the assisted driving method and blind spot image acquisition method provided in an exemplary embodiment of this application is shown.

[0055] Figure 27 A flowchart illustrating the assisted driving method and blind spot image acquisition method provided in an exemplary embodiment of this application is shown.

[0056] Figure 28 This invention provides a structural block diagram of an assisted driving device according to an exemplary embodiment of the present application.

[0057] Figure 29 This paper illustrates a structural block diagram of a blind spot image acquisition device provided in an exemplary embodiment of this application;

[0058] Figure 30 A schematic diagram of the structure of electronic glasses provided in an exemplary embodiment of this application is shown;

[0059] Figure 31 A schematic diagram of the structure of a vehicle device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0061] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0062] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0063] It should be noted that the user information (including but not limited to user device information, user personal information, vehicle driving information, and image information captured by cameras) and data (including but not limited to data used for analysis, stored data, and displayed data) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the biometric operations, image acquisition operations, and human-computer interaction recognition operations involved in this application were all obtained under full authorization.

[0064] It should be understood that although the terms first, second, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, a first parameter may also be referred to as a second parameter without departing from the scope of this disclosure, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0065] First, let me introduce the relevant terms used in this application:

[0066] Vehicle blind spots: These are areas that a driver cannot directly observe from the driver's seat because their view is obstructed by the vehicle body or road obstacles. The following section will provide a detailed explanation of the classification of vehicle blind spots.

[0067] (1) Front blind spot: The area directly in front of the vehicle that cannot be seen from the driver's seat. For passenger cars, this is usually the area not visible in front of the hood. Figure 1 The blind spot at the front of the vehicle is shown in Figure 10. The size of the blind spot at the front of the vehicle is usually affected by the vehicle height, seat height, hood length, and the driver's height.

[0068] (2) Rear blind spot: The area behind the vehicle that cannot be seen from the driver's seat using the rearview mirror. The size of the rear blind spot varies depending on the position and curvature of the rearview mirror, and is also affected by the extent of obstruction by the rear windshield. The approximate shape of the rear blind spot is as follows: Figure 1 The blind spot at the rear of the vehicle is shown in Figure 11. Reversing radar is commonly used in related technologies to eliminate the blind spot at the rear of the vehicle when reversing.

[0069] (3) Blind spot in rearview mirrors: The area to the side and rear of the vehicle that cannot be seen through the exterior rearview mirrors from the driver's seat. Exterior rearview mirrors include the left and right rearview mirrors. For example... Figure 1 The left rearview mirror blind spot 12 and the right rearview mirror 13 are shown in the diagram.

[0070] (4) Blind Spot: Areas that cannot be observed from the driver's seat due to obstruction by the vehicle's vertical beams. The vehicle's vertical beams are components connecting the vehicle body and roof; depending on the vehicle's structure, such as... Figure 2 As shown, the vehicle body vertical beams typically include the A-pillar 20 located between the windshield and the front door, the B-pillar 21 located between the front door and the rear door, and the C-pillar 22 located between the rear windshield and the rear door. This example uses a common four-seater sedan. For six-seater and eight-seater sedans, several more sets of vehicle body vertical beams may be provided. This application does not limit the location or name of the vehicle body vertical beams. The blind spots caused by the A, B, and C pillars are as follows... Figure 1 The blind spots are shown in Figure 14 (left A-pillar), 15 (right A-pillar), 16 (left B-pillar), 17 (right B-pillar), 18 (left C-pillar), and 19 (right C-pillar). These blind spots can also be called pillar blind spots, AB pillar blind spots, etc.

[0071] The above examples of vehicle blind spots are based on sedans, which are the most common vehicles encountered by ordinary people in their daily lives. However, due to differences in vehicle structure and size, the blind spots of large vehicles such as trucks, buses, and semi-trailers are somewhat different from those of sedans. Therefore, the blind spots of large vehicles will be introduced separately here.

[0072] (5) Front blind spot: The area directly in front of the vehicle that cannot be seen from the driver's seat. This area is also called the front blind spot, and is similar to the front blind spot of sedans mentioned above, so it will not be repeated here. The range of the front blind spot is as follows: Figure 3 As shown in section C. When there are pedestrians or road obstacles in the blind spot in front of the vehicle, the driver can hardly see them. Figure 4 As shown in section (1), the size of the blind spot in front of the vehicle also depends on the height of the corresponding obstacle.

[0073] (6) Left-side blind spot: The area on the left side of the driver's seat that cannot be seen from the driver's seat. The extent of the left-side blind spot is as follows: Figure 3 As shown in area B. Because the driver's seat of a large vehicle is usually higher than that of a small vehicle, its corresponding left and right blind spots are also larger, as shown in area B. Figure 4 As shown in part (2) of the document.

[0074] (7) Right-side blind spot: The area to the right of the driver's seat that cannot be seen from the driver's seat. The right-side blind spot is as follows: Figure 3 As shown in section A.

[0075] (8) Blind Spot: An area that cannot be seen from the driver's seat due to obstruction by the vehicle's vertical beams. The blind spot of a large vehicle is similar to that of a small car; however, for trucks and semi-trailers, there is usually only an A-pillar blind spot. Blind spots include... Figure 3The E1 area (i.e., the blind spot of the left A-pillar) and the E2 area (i.e., the blind spot of the right A-pillar) are shown in the figure.

[0076] (9) Blind spot in rearview mirrors: The area to the side and rear of the vehicle that cannot be observed through the exterior rearview mirrors from the driver's seat. Blind spots in rearview mirrors include... Figure 3 The image shows the D1 area (i.e., the blind spot of the left rearview mirror) and the D2 area (i.e., the blind spot of the right rearview mirror).

[0077] (10) Rear blind spot: The area behind the vehicle that cannot be seen from the driver's seat through the rearview mirror. Examples of rear blind spots include... Figure 3 Shown in D3 area.

[0078] It should be noted that the above Figure 3 The blind spot range shown is for illustrative purposes only. In actual driving scenarios, the area that the driver cannot observe may be larger or smaller than the shown blind spot range. This application does not limit this.

[0079] In addition, the following blind spots may be created during vehicle operation due to vehicle body structure or road conditions.

[0080] (11) Overtaking blind spot: The area that cannot be observed when overtaking due to obstruction by the vehicle in front. For example... Figure 5 As shown, there are four vehicles traveling on the road: car 31, car 32, car 33, and car 34. Cars 31, 32, and 33 are traveling in the same direction, while car 34 is traveling in the opposite direction. The driver of car 31 has a line of sight of 35 when driving. When car 31 needs to overtake car 32, the driver of car 31 has difficulty seeing cars 33 and 34 due to the obstruction of car 32. That is, cars 33 and 34 are in the overtaking blind spot of car 31.

[0081] (12) Inner wheel difference blind spot: The area where the vehicle's movement cannot be accurately judged due to the presence of the inner wheel difference when turning. The inner wheel difference refers to the difference between the turning radius of the front inner wheel and the turning radius of the rear inner wheel when the vehicle is turning. The inner wheel difference blind spot is usually caused by the driver's difficulty in visually estimating the inner wheel difference or the turning path of the rear inner wheel. Figure 6 As shown, when car 36 turns, its corresponding right front wheel trajectory is trajectory 37, left front wheel trajectory is trajectory 38, and right rear wheel trajectory is trajectory 39. The right front wheel trajectory 37 and right rear wheel trajectory 39 form an inner wheel difference blind spot 40. The driver of car 36 usually has difficulty observing pedestrians or cars in the inner wheel difference blind spot 40, or has difficulty being aware of the existence of the inner wheel difference blind spot 40 (that is, it is difficult to judge whether the right rear wheel will cause harm to pedestrians or cars in the inner wheel difference blind spot 40 during a right turn).

[0082] (13) Mountain road blind spot: The area of ​​the road surface that cannot be seen due to the obstruction of rocks on the side of the road. Mountain road blind spots usually occur on mountain roads with curves, where the driver cannot see the road conditions on the other side of the rocks.

[0083] (14) Blind spots: Areas that cannot be observed due to changes in light. For example, when exiting a tunnel, the light environment suddenly changes from dark to bright, or when entering a tunnel, the light environment suddenly changes from bright to dark, or various strong light stimuli (such as the high beams of oncoming vehicles) when driving at night may cause the driver's vision to go black, resulting in temporary blind spots.

[0084] 3D reconstruction is a method of modeling an object to obtain a virtual 3D model of that object. This virtual 3D model is a computer-readable model, which can be understood as a mathematical model that a computer can represent and process. 3D reconstruction is the foundation for using computers to analyze objects in the real world. In the field of computer vision, 3D reconstruction specifically refers to the process of reconstructing the original 3D information from a single view or multiple views. A single view refers to a photograph taken by a camera with a fixed perspective of the object to be reconstructed; multiple views refer to photographs taken by cameras with multiple different perspectives of the object to be reconstructed. Generally speaking, multiple view reconstruction yields better results, but it takes longer than single view reconstruction. 3D reconstruction technology has been widely applied in many fields, including video entertainment, virtual try-on, smart homes, cultural relic reconstruction, AR (Augmented Reality) tourism, autonomous driving, and large-scale 3D reconstruction.

[0085] With the development of 3D reconstruction technology, 3D reconstruction methods have gradually shifted from studying how to extract 3D information from 2D information to extracting 2D information from other perspectives from the reconstructed 3D information, i.e., generating new perspectives. In general, this involves performing 3D reconstruction on an object based on images from partial perspectives to obtain a virtual 3D model of the object, i.e., a virtual object; and then constructing a virtual camera based on the desired new perspective to capture images of the virtual object, thus obtaining images from the new perspective.

[0086] A 3D engine is a pre-written software framework or core component for creating and rendering 3D models. This framework provides designers with various tools for creating and rendering 3D models, aiming to enable them to quickly create 3D models without needing to understand how to derive them from computer languages. A 3D engine includes the following systems: a rendering engine (i.e., a "renderer," containing both 2D and 3D graphics engines), a physics engine, a collision detection system, sound effects, a scripting engine, computer animation, artificial intelligence, a network engine, and scene management.

[0087] Rendering engine: In the field of graphics technology, a rendering engine refers to the process of rendering a 3D model of a virtual object into a 2D image, while maintaining the stereoscopic effect of the 3D model in the 2D image. Typically, the model data of the completed 3D model is imported into the rendering engine, which then drives the rendering pipeline in the GPU (Graphics Processing Unit) to perform rendering, thereby visualizing the object indicated by the 3D model on the display screen of the gaming device.

[0088] Unity: One of the most widely used 3D engines. It provides powerful tools and features that enable developers to create realistic 3D environments and interactive experiences. Here are some of Unity's main applications in scene virtualization:

[0089] (1) Virtual Reality (VR) and Augmented Reality: Unity supports the creation of immersive VR and AR experiences, allowing users to enter virtual worlds through devices such as headsets or smartphones. In these applications, Unity's 3D rendering capabilities and interactive design tools enable developers to design various virtual scenes, such as games, educational simulations, and training scenarios.

[0090] (2) Digital Twin: A digital twin is a virtual copy of a physical object or system that can be simulated, analyzed, and optimized in a virtual environment. Unity can be used to build digital twin models, such as buildings, factories, and cities, which can be used for planning, monitoring, and maintenance.

[0091] (3) Architectural Visualization: In the fields of architecture and urban planning, Unity can be used to create 3D visualization models of architectural projects. This not only helps designers and clients better understand design intent, but can also be used for marketing and presentations.

[0092] (4) Game development: Unity is one of the preferred platforms for game developers. It provides a rich library of resources, a physics engine and animation tools, enabling developers to quickly build and iterate game scenes.

[0093] (5) Education and Training: Unity can be used to create educational software and training simulations, such as medical simulations and flight simulators. These virtual scenarios provide a safe environment that allows users to learn and practice without real risk.

[0094] (6) Industrial design and simulation: In the industrial field, Unity can be used to simulate production lines, machine operations, etc., to help engineers test and optimize during the design phase.

[0095] Image segmentation: The process of dividing an image into multiple segments. Based on the granularity of the segmentation, or the granularity of the segments, it can be divided into semantic segmentation, instance segmentation, and part segmentation.

[0096] (1) Semantic Segmentation: Semantic segmentation refers to pixel-level image segmentation. For a given image, semantic segmentation assigns a category to each pixel, ultimately outputting a semantic segmentation map. In the semantic segmentation map, the image is divided into several non-overlapping regions according to the categories assigned to each pixel. Each region consists of pixels belonging to the same category, and pixels within the same region exhibit consistency or similarity, while different regions show obvious differences. For example, separating the target (such as a transparent object) from the background in an image is a type of semantic segmentation targeting the target. The purpose of semantic segmentation is to understand the content of an image at the pixel level and assign an object class to each pixel in the image. For example, an image containing elements such as people, vehicles, and buildings can be segmented into multiple regions, each region representing a specific object.

[0097] (2) Instance segmentation: Dividing the image into regions corresponding to specific instances. The target region for instance segmentation is the object (e.g., a specific car or a specific building).

[0098] (3) Component segmentation: Dividing the image into regions corresponding to certain components of the instance. The target region for component segmentation is the object component (e.g., car door, body, wheels, etc.).

[0099] Object detection: The process of automatically locating and identifying specific targets in an image or video.

[0100] Target recognition: The process of identifying a specific target in a series of image or video frames.

[0101] Image Composition: This involves cutting out the foreground of an image and pasting it onto a background image to create a composite image.

[0102] Transform matrix: A mathematical tool used to represent changes in the position, orientation, and size of a graphical object in two-dimensional or three-dimensional space. In computer graphics, the transform matrix is ​​closely related to Position, Rotation, and Scale. The transform matrix is ​​composed of these three factors.

[0103] Position: Represents the position of an object in 3D space. Position is a 3D vector containing three components: x, y, and z.

[0104] Rotation: Represents the rotation of an object in 3D space. Rotation is a quaternion containing four components: w, x, y, and z. Quaternions can represent rotations in 3D space, avoiding gimbal lock issues and providing smoother interpolation.

[0105] Scale: Represents the size of an object in 3D space. Scale is a 3D vector containing three components: x, y, and z.

[0106] The Transform matrix is ​​composed of these three factors. It is a 4×4 matrix that can represent transformations such as translation, rotation, and scaling. The structure of the Transform matrix is ​​as follows: Transform = Position × Rotation × Scale.

[0107] Electronic glasses: Glasses with an independent operating system, controlled by voice or gestures to achieve different functions. Similar to smartphones, electronic glasses can install various applications and typically also have communication connectivity options such as Bluetooth and Wi-Fi to support communication with other electronic devices. Electronic glasses can also be called smart glasses.

[0108] VR, also known as Virtual Reality or Virtual Reality Technology, is a computer simulation system that can create and experience virtual environments. VR technology encompasses computer science, electronic information, and simulation technology. Its basic implementation relies primarily on computer technology, utilizing and integrating the latest advancements in various high-tech fields such as 3D graphics, multimedia, simulation, display, and server technologies. With the help of computers and other equipment, it generates a realistic 3D virtual environment that provides a multi-sensory experience, including visual, tactile, and olfactory sensations. By combining the virtual and real worlds, it creates a sense of immersion for those within the virtual environment.

[0109] AR: A technology that cleverly integrates virtual information with the real world. It widely uses various technologies such as multimedia, 3D modeling, real-time tracking and registration, intelligent interaction, and sensing to simulate and apply computer-generated text, images, 3D models, music, videos, and other virtual information to the real world. The two types of information complement each other, thereby achieving "augmentation" of the real world.

[0110] MR (Mixed Reality): MR technology is a further development of VR technology. MR technology enhances the realism of the user experience by presenting real-world scene information in a virtual scene and building an interactive feedback loop between the real world, the virtual world and the user.

[0111] Transportation vehicles: refers to all man-made devices used for human transportation. This includes widely used vehicles such as automobiles, ships, and airplanes, as well as vehicles still under development such as low-altitude manned aircraft and unmanned manned aircraft, and less common vehicles such as submarines, rockets, and manned satellites.

[0112] Communication connection: The connection state in which multiple communication devices transmit and exchange information through some means (wired or wireless). Based on the signal transmission method, it can be broadly divided into two categories: wireless communication connections and wired communication connections.

[0113] (1) Wireless Communication Connection: A communication connection that transmits information through electromagnetic waves propagating in space without the need for physical wiring. There are many types of wireless communication connection technologies, each with its own characteristics, such as WiFi (Wireless Fidelity), Bluetooth, mobile communication networks, satellite communication, and V2X (Vehicle to X, where X can represent anything). WiFi is a technology that allows electronic devices to connect to a wireless local area network (WLAN), widely used in homes, offices, and other scenarios to provide high-speed internet access. Bluetooth is a short-range wireless communication technology used to connect and transmit data between devices, such as mobile phones, headphones, and computers. Mobile communication networks include 2G, 3G, 4G, and 5G, which communicate with mobile devices (such as mobile phones and tablets) through base stations to achieve voice calls and data transmission. Satellite communication refers to using artificial Earth satellites as relay stations to forward radio signals, enabling communication between different locations on Earth. V2X is a communication technology that enables vehicles to exchange data with various entities in their surrounding environment. V2X can include: vehicle-to-vehicle (V2V) communication, vehicle-to-infrastructure (V2I) communication, vehicle-to-pedestrian (V2P) communication, or vehicle-to-network (V2N) communication.

[0114] (2) Wired Communication Connection: A communication connection that transmits information through physical lines. Wired communication connection technologies are generally more stable and have faster transmission speeds, but are limited by the physical length and layout of the lines. Wired communication connection technologies include Ethernet connections, fiber optic communication, serial port connections, USB (Universal Serial Bus) connections, and CAN (Controller Area Network) buses. Ethernet connections are local area network communication technologies based on the IEEE (Institute of Electrical and Electronics Engineers) 802.3 standard, widely used in offices, factories, and other scenarios, providing high-speed and stable data transmission. Fiber optic communication uses light waves to transmit information in optical fibers, and has advantages such as high transmission speed, large capacity, and strong anti-interference ability. Serial port connections are a commonly used serial communication interface standard used to connect computers to external devices (such as printers, modems, etc.). USB connections are a widely used communication interface between computers and peripherals, featuring plug-and-play functionality and high-speed transmission. CAN bus is a serial communication network that effectively supports distributed control systems and is commonly used in the automotive field to handle communication between internal automotive components.

[0115] Human-computer interaction methods refer to the methods by which humans exchange information and control operations with computers or other intelligent devices. There are many corresponding interaction methods, including: mouse and keyboard input, touchscreen input, gesture interaction, eye-tracking interaction, voice interaction, and brain-computer interfaces. Among these, mouse and keyboard input and touchscreen input are currently the most widely used input methods, but these two interaction methods have lower compatibility with electronic glasses and will not be discussed further here. The remaining interaction methods are all suitable for electronic glasses, and their application in electronic glasses will be briefly introduced below.

[0116] (1) Gesture interaction refers to the interaction between the user and the electronic glasses through hand movements. For example, gesture recognition technology can recognize the user's gestures such as waving, pinching, and rotating, and convert them into corresponding instructions to change the content of the screen displayed by the electronic glasses.

[0117] (2) Eye-tracking interaction refers to human-computer interaction by collecting the user's eye movements. For example, after wearing electronic glasses, the user can select or control the electronic glasses to perform corresponding operations by looking at a specific area.

[0118] (3) Voice interaction refers to users interacting with electronic glasses by speaking, and the electronic glasses understanding the user's intentions through voice recognition technology and responding accordingly (e.g., displaying corresponding screen content). Voice interaction is also a widely used method at present.

[0119] (4) Brain-computer interface (BCI / BMI) refers to the communication and control between a person and electronic glasses by directly reading and decoding the electrical signals of the brain. Brain-computer interface is a human-computer interaction method that is not yet widely used but has great potential.

[0120] Figure 7 An architectural diagram of a computer system provided in an exemplary embodiment of this application is shown. The computer system includes: electronic glasses 110 and a vehicle device 120.

[0121] Electronic glasses 110, also known as smart glasses, wearable devices, portable devices, etc., can be VR devices, AR devices, or MR devices, etc. Electronic glasses 110 can be divided into two categories: one is electronic glasses using a transparent display screen (which can be called see-through electronic glasses), and the other is electronic glasses using an opaque display screen (which can be called non-see-through electronic glasses). This application embodiment does not limit the specific type. Electronic glasses 110 are used to display a view of glasses with see-through content based on auxiliary data; or, electronic glasses 110 are used to enable the wearer to observe a view of glasses with see-through content based on auxiliary data.

[0122] The vehicle equipment 120 includes at least one camera facing the blind spot of the vehicle 130. The at least one camera in the vehicle equipment 120 is used to capture environmental images, and auxiliary data is related to the environmental images captured by the at least one camera.

[0123] Optionally, the vehicle equipment 120 also includes a control terminal for the vehicle 130. This control terminal can be an electronic device such as a mobile phone, tablet computer, vehicle-mounted terminal (vehicle system), wearable device, PC (Personal Computer), unmanned reservation terminal, or computer equipped with a ship's main engine control system. The vehicle 130 can be any of the aforementioned vehicles (including large and small cars), ships, airplanes, submarines, low-altitude manned aircraft, or unmanned manned aircraft; this embodiment does not limit the specific vehicle to these. The control terminal is used to establish a communication connection with at least one camera and electronic glasses 110.

[0124] For example, at least one camera 1 is used to capture an environmental image 2, which is processed by the control terminal in the electronic glasses 110 or the vehicle device 120 to obtain auxiliary data 3. Before the electronic glasses 110 acquires the auxiliary data 3, the electronic glasses 110 can display a glasses image 4 without perspective content. In this glasses image 4, there is a blind spot caused by the vehicle body 6. The blind spot refers to the external area of ​​the vehicle that the wearer cannot observe due to the obstruction of the obstruction (vehicle body). After the electronic glasses 110 acquires the auxiliary data 3, it can display a glasses image 5 with perspective content based on the auxiliary data 3. The perspective content is as shown in area 7. The perspective content is a perspective display processing of the vehicle body 6, making the entire vehicle body 6 appear semi-transparent, so that the wearer can observe what was originally obstructed by the vehicle body 6. Thus, the wearer can observe the ball 8 that was originally located in the blind spot. The glasses image with perspective content allows the wearer to notice sudden situations in the blind spot in time.

[0125] Figure 8 A flowchart illustrating an exemplary embodiment of this application is shown. The method is performed by electronic glasses worn by a user inside a vehicle; these electronic glasses may be those within the aforementioned computer system. The method includes:

[0126] Step 210: Establish a communication connection with the vehicle equipment, which includes at least one camera facing the blind spot of the vehicle, whereby the blind spot is an external area that the wearer cannot observe due to obstruction.

[0127] The vehicle equipment includes at least one camera facing the blind spot of the vehicle, whereby the blind spot is an external area of ​​the vehicle that cannot be observed by a wearer due to obstruction; wherein the wearer is located inside the vehicle, and the obstruction is at least one of a road barrier, a component of the vehicle, an occupant, or an item located inside the vehicle. It can be understood that the blind spot is an external area of ​​the vehicle that cannot be observed by a wearer located inside the vehicle due to obstruction. The external area of ​​the vehicle refers to the entire exterior of the vehicle. For example, if the vehicle is a car, the external area of ​​the car refers to the area outside the car frame; if the vehicle is a ship, the external area of ​​the ship refers to the area outside the entire ship frame; if the vehicle is an airplane, the external area of ​​the airplane refers to the area outside the entire airplane frame.

[0128] Generally, the wearer is located in the driver's seat. For example, the "vehicle blind spot" mentioned above refers to the area outside the vehicle that the wearer in the driver's seat cannot directly observe due to obstruction of their view by the vehicle body (i.e., parts of the vehicle) or road obstacles. However, this application embodiment does not limit the wearer's position inside the vehicle; that is, in a car driving scenario, the wearer can also be a passenger in the front passenger seat or a rear passenger. In other words, the blind spot in this application embodiment applies to any wearer inside the vehicle, not just the driver (i.e., the wearer in the driver's seat). For example, when the wearer is in the front passenger seat, the blind spot refers to the area within the wearer's line of sight. In general, when different wearers (height, upper and lower body proportions, etc.) are in the same position or the same wearer is in different positions, the areas they cannot observe may differ, that is, their blind spots (position, range, etc.) may differ.

[0129] At least one camera is positioned facing the blind spot of the vehicle, meaning at least one camera is used to capture the environmental image of the vehicle's blind spot; alternatively, at least one camera is used to capture the environmental image of the blind spot corresponding to the wearer. In reality, the environmental image captured by at least one camera should include either the vehicle's blind spot or the wearer's blind spot; that is, the area captured by at least one camera is larger than the blind spot itself. If discontinuous (or non-intersecting) blind spots are considered a single blind spot, then one camera corresponds to one blind spot, or one camera corresponds to multiple blind spots, or multiple cameras correspond to one blind spot. Taking the vehicle blind spot mentioned above as an example, a camera positioned directly in front of the vehicle can capture the environmental image of both the forward blind spot and any obstructed blind spot for any wearer; cameras on the left and right rearview mirrors can capture the environmental image of the rearview mirror blind spot; a camera at the rear of the vehicle can capture the environmental image of the rear blind spot, and so on.

[0130] Optionally, at least one camera includes at least one of the following: a camera configured with the vehicle at the factory, a camera added to the vehicle by the user, and a camera located in a driving environment. For example, the vehicle is a car. To eliminate the vehicle blind spots shown above, the camera can be mounted directly in front of the car, on the left and right rearview mirrors, on the A and B pillars, on the vehicle base, etc.; for vehicle blind spots while driving, the camera can be mounted on road infrastructure (such as streetlights, utility poles, etc.) or on other vehicles, and this application embodiment does not limit this.

[0131] Transportation vehicles include cars, ships, and airplanes. The driving environment is the environment in which a transportation vehicle is in motion. For example, for a car, the corresponding driving environment is usually a road; for a ship, the corresponding driving environment is usually a waterway; and for airplanes and manned aircraft, the corresponding driving environment is usually an airway.

[0132] Optionally, the communication connection established between the electronic glasses and the vehicle device includes at least one of wireless communication connection and wired communication connection; the connection method of the communication connection established between the electronic glasses and the vehicle device includes at least one of direct connection and indirect connection, where direct connection means that the communication connection between the electronic glasses and the vehicle device does not pass through other communication devices, and indirect connection means that the communication connection between the electronic glasses and the vehicle device is relayed through other communication devices.

[0133] Taking a car as an example, the vehicle equipment includes only at least one camera facing the blind spot of the vehicle, and the communication connection between the at least one camera and the electronic glasses is a wireless communication connection, with the connection method being a direct connection; or, the vehicle equipment includes only at least one camera facing the blind spot of the vehicle, and the communication connection between the at least one camera and the vehicle terminal is a wired communication connection, while the communication connection between the vehicle terminal and the electronic glasses is a wireless communication connection. In this case, the communication connection established between the electronic glasses and the vehicle equipment includes both wireless and wired communication connections, with the connection method being an indirect connection; or, at least one camera is installed on the car, and the communication connection between at least one camera and the vehicle terminal is a wired communication connection, such as a CAN bus or USB connection, while the communication connection between the vehicle terminal and the electronic glasses is a wireless communication connection; or, some cameras are installed on the car, and the connection between these cameras and the vehicle terminal is a wired communication connection, while other cameras are installed on road infrastructure or other vehicles, and the communication connection between these cameras and the vehicle terminal is a wireless connection, such as a Bluetooth connection or a V2X connection. This application does not limit this specific approach.

[0134] Step 220: Obtain auxiliary data through a communication connection. The auxiliary data is based on environmental images captured by at least one camera.

[0135] The auxiliary data is obtained based on environmental images captured by at least one camera. This can be understood as: auxiliary data is data obtained by encapsulating environmental images captured by at least one camera; or, auxiliary data is data obtained by digitizing environmental images captured by at least one camera; or, auxiliary data is data obtained by processing environmental images captured by at least one camera, such as auxiliary data being digitized data of glasses images with perspective features, or auxiliary data being data obtained by encapsulating environmental images of blind spots, etc.

[0136] For example, the auxiliary data acquired by the electronic glasses at the current moment is determined based on the environmental images captured by at least one camera at the current moment; or, the auxiliary data acquired by the electronic glasses at the current moment is determined based on the environmental images captured by at least one camera in the previous n seconds, where n is a positive integer. Optionally, the value of n is determined based on at least one of the following: the vehicle's speed, road conditions, and driving scenario. For example, in high-speed driving scenarios (such as a car speed exceeding 100 km / h), the value of n is small, such as 0.01; in low-speed driving scenarios (such as a car speed below 10 km / h), the value of n is large, such as 5; in scenarios where there is no driving, the value of n can be even larger, such as 10 or even 60; in complex road conditions (such as traffic jams, continuous sharp curves, etc.), the value of n is small; in simple road conditions, the value of n is large; in driving scenarios where accidents frequently occur (such as starting, reversing, etc.), the value of n is small. It can be understood that the smaller the value of n, the higher the timeliness of the auxiliary data.

[0137] Optionally, the environmental footage captured by at least one camera is at least one of an image, a video, or a video frame.

[0138] Step 230: Based on the auxiliary data display, the glasses image with perspective content is the environmental image of the blind spot presented after the obstruction is displayed through the image captured by the electronic glasses.

[0139] The blind spot environment shown in the perspective content refers to the environment corresponding to all or part of the blind spot of the vehicle. The aforementioned partial blind spot refers to the blind spot selected by the wearer, or the blind spot set by the developer, or the blind spot existing in the image captured by the electronic glasses.

[0140] The environmental view of the blind spot is determined based on the environmental view captured by at least one camera. Optionally, the environmental view (or perspective content) of the blind spot is directly determined based on the environmental view captured by at least one camera, that is, the environmental view of the blind spot includes a portion of the environmental view captured by at least one camera; or, the environmental view of the blind spot includes a portion of the environmental view captured by some cameras; or, the environmental view of the blind spot includes a portion of the environmental view captured by some cameras. Alternatively, the environmental view of the blind spot is determined after processing the environmental view captured by at least one camera.

[0141] Glasses images without perspective, such as Figure 9 As shown in part (1), the obstruction 90 will obstruct the road, preventing the wearer from observing the road conditions obstructed by the obstruction 90. That is, for the wearer, Figure 9 The image shown in part (1) has blind spots; the image through the glasses with perspective content is as follows: Figure 9As shown in section (2), the entire obstruction 90 is made completely transparent, allowing the wearer to have a clear view of the current driving scene, thereby eliminating blind spots caused by the obstruction 90. Figure 9 In the corresponding scenario, the obstruction 90 refers to the parts of the car and the occupants. That is, the parts of the car are the entire car body, and the occupants refer to the part of the driver's body, namely the wearer's arm.

[0142] The "glasses screen" refers to the image seen by the wearer of the electronic glasses. Alternatively, the "glasses screen" refers to the image displayed on the screen of the electronic glasses. Optionally, the content of the image included in the glasses screen is set by the user; or, the content of the image included in the glasses screen is set by the developer.

[0143] The field of view of the glasses may be the same as or different from the field of view of the direct view, which refers to the content seen without wearing the electronic glasses. For example, such as Figure 10 As shown, there is a row of 5 trees. When observer 41 is not wearing electronic glasses, the direct view they see is image 42. When observer 41 is wearing electronic glasses, the view they see can be image 43 (with a wider field of view than the direct view), image 42 (with a field of view equal to the direct view), or image 44 (with a field of view smaller than the direct view). It should be noted that... Figure 10 The visible range shown includes variations in the length and width of the image, but in reality, the factors influencing the visible range include at least one of the length, width, and the center of the image. That is, the difference in the visible range between glasses and direct viewing is reflected in the difference in length and / or width and / or the center of the image.

[0144] In summary, the method provided in this application involves electronic glasses that, based on a communication connection with a vehicle device, acquire auxiliary data determined from environmental images captured by at least one camera. This auxiliary data is then used to display a view through the glasses, allowing the wearer to directly see through obstructions and eliminate blind spots, thus improving safety while driving. Furthermore, the basic processing method for the see-through content is illustrated: based on the image captured by the electronic glasses, obstructions (i.e., those appearing in front of the wearer's eyes after wearing the glasses) are rendered with a see-through effect, revealing the environmental image of the blind spot. This allows the wearer to drive based on the see-through view displayed in the electronic glasses, thereby improving driving safety and minimizing the incidence of traffic accidents caused by blind spots. Furthermore, the auxiliary data is obtained based on environmental images captured by at least one camera, which is a camera facing the blind spot of the vehicle. This demonstrates a method for eliminating blind spots—indirectly obtaining a view through the glasses (or indirectly obtaining the view through the glasses) by capturing environmental images from at least one camera facing the blind spot of the vehicle. In addition, the blind spots eliminated by this embodiment not only include blind spots caused by vehicle components but also blind spots caused by occupants, items inside the vehicle, etc., comprehensively considering blind spot factors during vehicle driving and achieving all-round blind spot elimination.

[0145] In some embodiments, the perspective content also displays semi-transparent, wholly or partially obstructing objects. That is, when the perspective content shows obstructing objects, the obstructing objects are not completely transparent, but have a certain degree of transparency. Furthermore, the environmental view of the blind spot displayed by the obstructing objects in the perspective content can also be semi-transparent; that is, the environmental view of all or part of the blind spot displayed in the perspective content is semi-transparent.

[0146] Electronic glasses can be divided into two categories: those using transparent displays (called see-through electronic glasses) and those using opaque displays (called non-see-through electronic glasses). For see-through electronic glasses, the image displayed is based on the real scene directly observed by the wearer; that is, it displays a semi-transparent view of the environment in areas where obstructions are located, showing all or part of the blind spot. For non-see-through electronic glasses, all image content in the see-through view is processed; therefore, the see-through content can display at least one of the following: a semi-transparent view of all obstructions, a semi-transparent view of part of an obstruction, a semi-transparent view of the entire blind spot, or a semi-transparent view of a partially blind spot.

[0147] Optionally, semi-transparency refers to having transparency, but the value of transparency is not limited. For example, if there are objects a1 to a3, and the transparency of object a1 displayed in the perspective content is 25%, the transparency of object a2 is 50%, and the transparency of object a3 is 75%, then in terms of display effect, the transparency of object a1 < the transparency of object a2 < the transparency of object a3. Semi-transparency can also indicate opacity. For example, if there are objects a1 to a3, and the opacity of object a1 displayed in the perspective content is 25%, the opacity of object a2 is 50%, and the opacity of object a3 is 75%, then in terms of display effect, the transparency of object a1 > the transparency of object a2 > the transparency of object a3.

[0148] For example, in perspective content, there are semi-transparent, fully obscured objects, such as... Figure 9 As shown, Figure 9 Part (1) shows the view through the glasses without perspective, i.e., the view with a blind spot. Figure 9 Parts (3) and (4) show the view through the glasses with perspective content, which includes all semi-transparent obstructions. Figure 9 The transparency of the obstruction shown in section (3) is higher than that of the obstruction. Figure 9 The obstruction shown in section (4) of the document.

[0149] Optionally, in addition to the vehicle equipment including at least one camera, the driving environment includes at least one environmental camera, and the at least one environmental camera establishes a communication connection with the electronic glasses; in this case, the obstruction also includes road barriers outside the vehicle, and the perspective view also shows semi-transparent road barriers. For example, as Figure 11 As shown, the means of transportation is a car, and the car is driving on a mountain road with sharp bends, such as... Figure 11 As shown in part (1), part of the vehicle body 70 can be observed, while another part of the vehicle body 70 is obscured by the mountain 71; after wearing electronic glasses, the glasses image with perspective content is displayed as follows. Figure 11 As shown in section (2), the mountain 71 is an obstruction. Figure 11 Part (2) is shown as a semi-transparent mountain 72. The environment of the blind spot obscured by the mountain includes the entire body of the object vehicle 70 and the mountain road obscured by the mountain.

[0150] Optionally, the environment of all blind spots displayed in the perspective content is semi-transparent, such as... Figure 11 As shown in part (3), relative to Figure 11 In part (1), the mountain 71, which is an obstruction, is not transparent, while the objects in the blind spot environment, such as the vehicle 70 and the road, are transparent.

[0151] In summary, the method provided in this application displays a semi-transparent obstruction or blind spot in the viewed content, rather than a fully transparent obstruction or blind spot. On the one hand, the wearer can observe the environment in the blind spot; on the other hand, it prevents the wearer from misjudging the actual driving conditions of the vehicle. If the obstruction were fully transparent, such as… Figure 9 As shown in section (2), it may cause the wearer to have a vague sense of the fact that they are currently driving a vehicle, which may lead to accidents.

[0152] In some embodiments, after wearing the electronic glasses, the glasses display a view with see-through content based on auxiliary data. However, on the one hand, for wearers unaccustomed to see-through content, the normally displayed see-through content may cause confusion while driving. For example, a wearer accustomed to driving a regular car might feel like they are driving a convertible when viewing the glasses with see-through content, making it difficult for them to adapt to driving while wearing electronic glasses. Therefore, methods such as... Figure 12 The method shown displays the view through the glasses with perspective content. The perspective content is only displayed when the perspective conditions are met. Step 230 can be implemented as step 231.

[0153] Step 231: When the perspective condition is met, display the glasses image with perspective content based on the auxiliary data display. The perspective condition is the condition that triggers the display of perspective content.

[0154] In other words, when the electronic glasses detect that the perspective conditions are met, the auxiliary data displays an image of the glasses with perspective content. When the perspective conditions are not met, the auxiliary data displays an image of the glasses without perspective content; or, when the perspective conditions are not met, the auxiliary data displays an image of the glasses in which no perspective content exists.

[0155] Optionally, the see-through conditions include at least one of the following: the wearer needs to observe the see-through content; there is a dangerous situation in the blind spot of the vehicle; the traffic scene requires the wearer to observe the blind spot of the vehicle; or the driver assistance mode is activated. The driver assistance mode can also be called the see-through driving mode or the see-through mode.

[0156] The determination of perspective conditions is made by electronic glasses or a vehicle. For example, electronic glasses are used to collect the wearer's sensory information and, based on this information, detect whether the wearer needs to observe perspective content. The sensory information includes at least one of eye movement information, gesture information, voice information, and electroencephalogram (EEG) information. Alternatively, the vehicle is used to identify whether there are dangerous situations in the vehicle's blind spots and sends the identification results to the electronic glasses. Auxiliary data includes the vehicle's identification results, and the electronic glasses determine whether the vehicle's blind spots are dangerous based on these results. Alternatively, auxiliary data includes the environmental image of the blind spots, and the electronic glasses determine whether there are dangerous situations in the vehicle's blind spots based on this image. Alternatively, the vehicle is used to identify whether the traffic scene requires the wearer to observe the vehicle's blind spots and sends the identification results to the electronic glasses. Auxiliary data includes the vehicle's identification results, and the electronic glasses determine whether the traffic scene requires the wearer to observe the vehicle's blind spots. Alternatively, the electronic glasses are used to detect whether the assisted driving mode is activated.

[0157] The wearer has a need to observe perspective content, which can also be described as the wearer's observation needs, observation habits, perspective habits, perspective preferences, or observation preferences. Observation habits (also referred to as perspective habits, perspective preferences, or observation preferences) refer to driving scenarios in which the wearer requests the display of perspective content at a frequency that reaches a preference threshold. For example, if the wearer requests the display of perspective content every time they wear the electronic glasses while passing through a certain road segment, and requests that the perspective content not be displayed when leaving that road segment, this can be recorded as the wearer's observation habits. Subsequently, when the wearer wears the electronic glasses again, the electronic glasses will automatically display perspective content every time they pass through that road segment.

[0158] Optionally, the observation habits stored in the electronic glasses do not distinguish the wearer's identity; or, the electronic glasses store the corresponding observation habits for each wearer. After the wearer puts on and turns on the electronic glasses, the electronic glasses identify the wearer's identity and load the corresponding observation habits. The electronic glasses can use biometric technology, such as iris, face, voiceprint, etc., to identify the wearer. This application embodiment does not limit this.

[0159] A blind spot of a vehicle presents a dangerous situation, which can also be described as a dangerous situation being obscured by an obstacle, or a dangerous situation existing within the visual environment of the blind spot. A dangerous situation refers to a traffic condition that affects the continued movement of the vehicle. This could be an obstacle suddenly appearing in the vehicle's blind spot, an obstacle located in the vehicle's blind spot at the time of starting, or an obstacle in the predicted path of movement. The predicted path of movement refers to the path predicted based on at least one of the following: recorded driving habits, current speed, and the current steering wheel deflection angle.

[0160] Traffic scenarios require the wearer to observe the blind spots of the vehicle. This can be described as a high-visibility-requirement traffic scenario, or an accident-prone traffic scenario. A traffic scenario refers to the environment in which the vehicle is situated, and is related to the driving situation, road conditions, etc. For example, a traffic scenario might be driving on a road near a school; driving on a mountain road with many sharp bends; a narrow road; or going uphill or downhill.

[0161] Alternatively, the perspective conditions may be set by the wearer; or the perspective conditions may be fixed.

[0162] In summary, the method provided in this application sets a trigger condition, namely a perspective condition, for the display of perspective content. This ensures that when the wearer wears the electronic glasses, perspective content will only be displayed if the perspective condition is met. If the perspective condition is not met, the perspective content will not be displayed. Without affecting the wearer's normal driving, displaying perspective content only when the perspective condition is met can avoid the discomfort caused to the wearer by normal perspective content.

[0163] Furthermore, the perspective settings take into account two aspects. Firstly, they consider user needs, displaying perspective content only when the wearer expects to see it. They can also record the wearer's observation habits, allowing the glasses to display perspective content directly based on those habits without requiring the wearer to actively provide instructions. This improves human-computer interaction efficiency and provides a better user experience. It also allows the wearer to be more focused on driving, avoiding the need to distract themselves by instructing the glasses to display perspective content. Secondly, they also consider scenarios where perspective is needed in actual driving situations, such as when there is danger in blind spots, requiring the display of perspective content to alert the wearer to potential hazards.

[0164] The perspective conditions will now be explained in more detail.

[0165] 1. The wearer has a need to observe see-through content.

[0166] The conditions for perspective (or the wearer's need to observe perspective content) include at least one of the following:

[0167] • The wearer's gaze is directed towards the obstruction;

[0168] • The wearer's head should be facing the obstruction;

[0169] • An obstruction appears in the view through the glasses;

[0170] • Human-computer interaction instructions that allow the wearer to see through obstructions.

[0171] The following sections will introduce each of the aforementioned perspective conditions.

[0172] 1.1 The wearer's line of sight is directed towards the obstruction.

[0173] When the wearer's line of sight is directed toward an obstruction, it can eliminate the blind spot caused by the obstruction in the wearer's line of sight.

[0174] The wearer's gaze is directed toward the obstruction, including at least one of the following: the wearer's gaze is directed toward the obstruction; the wearer's gaze is on the obstruction; the wearer's eyes are directed toward the obstruction.

[0175] Electronic glasses perform gaze recognition by employing visual estimation methods to determine the wearer's gaze target, gaze point, or eye direction. Once the gaze point or eye direction is determined, target detection is needed to identify any obstructions pointing to the gaze point or along the eye's direction. Gaze recognition or visual estimation methods require a camera inside the electronic glasses to capture images of the wearer's eyes. For example, in a car, with the wearer in the driver's seat and the obstruction being a passenger in the front passenger seat, when the wearer's gaze is directed towards the passenger, the glasses display the environment through the passenger's blind spot (i.e., the environment outside the vehicle obscured by the passenger). Alternatively, in a car, when the wearer looks towards the left front of the car, if the detected gaze target is the left A-pillar, the glasses display the environment through the left A-pillar's blind spot (i.e., the environment outside the vehicle obscured by the left A-pillar).

[0176] In summary, the method provided in this application embodiment can improve human-computer interaction efficiency and driving safety because the wearer's line of sight can largely represent the wearer's observation needs. Whenever the wearer's line of sight is directed toward an obstruction, it is assumed that the wearer wants to see through the obstruction to eliminate the blind spot caused by the obstruction, that is, to see the environmental scene obscured by the obstruction.

[0177] 1.2 The wearer's head should face the obstruction.

[0178] When the wearer's head is facing an obstruction, head pose estimation can be performed by the electronic glasses themselves, such as using physical sensors (e.g., gyroscopes) to detect the wearer's head pose. Alternatively, head pose estimation can be performed by the vehicle itself, such as using a camera inside the vehicle facing the wearer, and performing head pose estimation based on the image of the wearer captured by that camera. For example, if the vehicle is a car and the obstruction is the car's A-pillar (including the left and right A-pillars), when the wearer's head is facing left, the system displays an environmental view of the blind spot through the left A-pillar.

[0179] In summary, the method provided in this application, compared to being triggered by the wearer's gaze toward an obstruction, detects the wearer's head toward the obstruction and determines the wearer's desired perspective direction based on the wearer's head orientation. This provides the wearer with the desired perspective content while preventing the wearer's gaze from leaving the road surface, thereby reducing the risk of traffic accidents caused by the wearer's gaze leaving the road surface and improving driving safety.

[0180] 1.3 Obstructions appear in the image of the glasses.

[0181] When an obstruction appears in the viewfinder, the glasses eliminate the blind spot caused by the obstruction within the wearer's field of vision. The obstruction refers to at least one of the following: a component of a vehicle, a passenger, or an item located within the vehicle. A component of a vehicle refers to the various parts that make up the vehicle; for example, in a car, components include the body and steering wheel; in a ship, components include the control panel and hull. A passenger is a person located on or inside the vehicle. Items inside the vehicle refer to items installed inside or placed inside the vehicle after it leaves the factory; for example, in a car, items inside the car include decorations on the center console, decorations hanging on the rearview mirror, and decorations placed in front of the rear windshield.

[0182] In summary, the method provided in this application eliminates blind spots caused by all obstructions within the eyeglass image. This method treats all obstructions within the entire eyeglass image, or the image content observed by the wearer, uniformly. On one hand, it eliminates the need for other identification methods to segment different obstructions within the image content, thereby improving processing efficiency and reducing processing difficulty. On the other hand, compared to eyeglass images that only process partial obstructions, uniformly processed eyeglass images can reduce the sense of disjointedness caused to the wearer. For example, with a car, the entire frame is an obstruction. If only part of the frame is visible, it might appear as if a car is missing or has a semi-transparent left A-pillar, conflicting with the wearer's common sense about cars and potentially causing a strong sense of disjointedness. This not only results in a poor user experience but could also lead to traffic accidents due to the wearer's disorientation. Processing the entire car body, i.e., displaying a transparent or semi-transparent body, creates a more harmonious color scheme for the entire eyeglass image, reducing the sense of disjointedness.

[0183] 1.4 Human-computer interaction instructions for wearers to see through the obstruction

[0184] The wearer can use different human-computer interaction methods to issue human-computer interaction instructions. These methods include at least one of voice interaction, gesture interaction, eye-tracking interaction, and brain-computer interface. Furthermore, given the specific context of wearing electronic glasses on public transportation, the human-computer interaction methods may also include the wearer's interaction with the vehicle itself.

[0185] In cases involving multiple obstructions or multiple obstruction types, the human-computer interaction indication can be used to indicate the obstruction or obstruction type. The obstruction type indicates the type of obstruction, such as obstruction types including vehicle parts, occupants, and items within the vehicle. For example, obstruction types include type 1, type 2, and type 3; obstructions include obstructions a1 to a6, wherein obstructions a1, a2, and a3 belong to type 1, obstructions a4 and a5 belong to type 2, and obstruction a6 belongs to type 3; the wearer can indicate a see-through obstruction a1, or an obstruction of see-through type 1 (i.e., obstructions a1, a2, and a3).

[0186] Human-computer interaction instructions are human-computer interaction commands that instruct electronic glasses to display see-through content. Human-computer interaction instructions are commands that use human-computer interaction methods.

[0187] The wearer requests human-computer interaction instructions to see through an obstruction, including at least one of the following: the wearer's voice command to see through the obstruction; the wearer's gesture selection command to see through the obstruction; the wearer's directional operation to a vehicle, where the directional operation corresponds to the obstruction; the wearer's eye movement selection command to see through the obstruction; and the wearer's brainwave command to see through the obstruction.

[0188] Among them, voice commands allow the wearer to see through objects, such as saying "see through the entire car body," "see through the A-pillar of the car," or "I want to see the front of the car." Gesture commands allow the wearer to touch or point to a part of the vehicle or circle an obstruction with their finger. Eye commands allow the wearer to look at an obstruction for more than one second. Brainwave commands allow the wearer to express their thoughts about seeing through obstructions. Directional operations for the vehicle allow the wearer to turn the steering wheel to the left, turn on the left turn signal, or start the vehicle.

[0189] Optionally, the human-computer interaction instructions are multimodal, that is, multiple of the above-mentioned human-computer interaction instructions are used. For example, combining voice perspective instructions and eye-tracking selection instructions, the wearer looks at an obstruction and says "perspective"; or, combining voice perspective instructions and gesture selection instructions, the wearer touches an obstruction with their left hand and says "perspective the obstruction pointed to by the left hand", etc.

[0190] In summary, the method provided in this application uses diverse human-computer interaction instructions to confirm the wearer's need for see-through content, taking the wearer's needs as the foundation, thereby improving interaction efficiency and user experience. At the same time, electronic glasses are usually equipped with various human-computer interaction-based sensors. With the electronic glasses equipped with human-computer interaction sensors, the method detects the wearer's human-computer interaction instructions to see through the obstruction and provides see-through of the obstruction based on the wearer's wishes. This can both avoid the need for normally uncommon see-through content and meet the wearer's occasional see-through needs.

[0191] 2. Warning information exists in the blind spots of the vehicle.

[0192] The perspective condition (or the presence of warning information in the blind spot of the vehicle) includes at least one of the following: the presence of warning information in the warning area corresponding to the obstruction, the warning area being the area around the vehicle; or the presence of warning information in the predicted movement route corresponding to the obstruction, the predicted movement route being the route the vehicle will move within a time threshold or a distance threshold.

[0193] The warning area is the area surrounding the vehicle. Optionally, the surrounding area refers to a circular area centered on the center of the vehicle, and the warning area corresponding to an obstruction is the blind spot caused by the obstruction within this surrounding area. For example, such as... Figure 1 As shown, the warning area is a circular area with the center of the vehicle as the center, that is, the warning area is circle 9. The obstruction is the front of the vehicle, so the warning area corresponding to the front of the vehicle is the blind spot 10.

[0194] A predicted route is the path a vehicle will take within a time or distance threshold. For example, a predicted route is the path a vehicle will take within 5 seconds; or, a predicted route is the path a vehicle will take within 100 meters.

[0195] Electronic glasses or vehicles can predict the route a vehicle will take within a time or distance threshold. The prediction is based on at least one of the following: driving habits and current speed, current steering wheel deflection angle, and road conditions.

[0196] Warning information includes at least one of the following: obstacles and unusual situations. Obstacles are usually immovable and fixed, such as utility poles, walls, hillsides, and green belts. Unusual situations are rare or sudden traffic occurrences, such as pedestrians suddenly appearing, vehicles suddenly turning, or children in the driver's blind spot when starting.

[0197] That is, a warning message exists in the warning area corresponding to the obstruction, including at least one of the following situations: an obstacle appears in the warning area corresponding to the obstruction; an abnormal situation occurs in the warning area corresponding to the obstruction. A warning message exists in the predicted movement path corresponding to the obstruction, including at least one of the following situations: an obstacle exists in the predicted movement path corresponding to the obstruction; an abnormal situation occurs in the predicted movement path corresponding to the obstruction.

[0198] In summary, the method provided in this application displays the see-through content corresponding to the warning information to the wearer when the electronic glasses or vehicle detects the presence of warning information. When no warning information is detected, the see-through content is not displayed. This represents a minimal change compared to current daily driving practices, making it easier for the wearer to adapt to the see-through content displayed in the electronic glasses. Furthermore, displaying the see-through content only when a warning information is detected fully considers the potential for traffic accidents caused by warning information. Displaying the see-through content alerts the wearer to the presence of warning information in blind spots, allowing the wearer to more intuitively notice the warning information in the blind spot's environment and make appropriate decisions, thereby improving driving safety.

[0199] 3. In traffic scenarios, wearers need to observe the blind spots of vehicles.

[0200] Optionally, the electronic glasses or vehicle can identify the current traffic scene in which the wearer is located.

[0201] Traffic scenarios require the wearer to observe the blind spots of the vehicle. This can be described as a high-visibility-requirement traffic scenario, or an accident-prone traffic scenario. A traffic scenario refers to the environment in which the vehicle is situated, and is related to the driving situation, road conditions, etc. For example, a traffic scenario might be driving on a road near a school; driving on a mountain road with many sharp bends; a narrow road; or going uphill or downhill.

[0202] In summary, the method provided in this application involves electronic glasses or a vehicle judging traffic scenarios and displaying perspective content in scenarios where the wearer needs to observe blind spots. This eliminates the need for the wearer to instruct or make judgments. On one hand, having the electronic glasses or vehicle make the judgments frees up the wearer's mental energy, allowing them to focus more on driving or do other things. On the other hand, for wearers with limited driving experience, who often struggle to determine when to observe blind spots, the electronic glasses or vehicle assistance helps them develop awareness of blind spots, thus improving both driving skills and safety.

[0203] In addition to the perspective content mentioned above, to further enhance vehicle driving safety and meet human-computer interaction needs, the electronic glasses also provide other types of screen content, such as zoomed-in content, rearview mirror content, extended rearview mirror content, virtual warning elements, and retained screen content. The following sections will introduce each of these screen contents and their display scenarios (trigger conditions).

[0204] 4.1 Scaling Content

[0205] In some embodiments, such as Figure 13 As shown, the method also includes:

[0206] Step 310: In response to a zoom trigger operation on the first area of ​​the screen in the glasses, display the screen with zoomed content.

[0207] The first area image refers to a portion of the image content displayed on the glasses, while the scaled-up content refers to the image content after scaling down the first area image.

[0208] Optionally, the first area image is a portion of the image content in the glasses image; or, the first area image is the image content of the first area in the glasses image, where the first area is a certain area in the glasses image.

[0209] Scaling refers to the content of the image after scaling down the first area of ​​the image. For example, the image of glasses... Figure 14 As shown in part (1), the first area of ​​the screen is screen 45; after performing a zoom trigger operation on the first area of ​​the screen, the displayed screen with zoomed content is as follows: Figure 14 As shown in part (2), the scaled content is screen 46, which is enlarged relative to the first area screen 45.

[0210] Optionally, in response to a scaling trigger operation on content scaled within the glasses' display, the display size of the scaled content is changed. For example... Figure 14 As shown in section (3), the scaled content has been enlarged (or, in other words, the first area of ​​the image has been further enlarged). The scaling degree of the scaled content 47 relative to the first area of ​​the image is compared to... Figure 14 The zoomed content 46 shown in section (2) is larger; the zoomed content can also be reduced, such as Figure 14 As shown in section (4), the scaling of the scaled content 48 relative to the first area of ​​the image is compared to... Figure 14 The zoomed content 46 shown in section (2) is smaller.

[0211] Optionally, the zooming operation for the first area of ​​the image on the glasses screen can be triggered by a human-computer interaction command. For example, the wearer can select the first area of ​​the image on the glasses screen using a gesture and then say "zoom in on the selected area" via voice command; or, the wearer can use brainwave commands to select the first area of ​​the image and instruct to zoom in on the first area to obtain the zoomed content. The form of the corresponding human-computer interaction command is similar to "1.4 Human-computer interaction instructions for wearers requesting to see through obstructions", which will not be described in detail here.

[0212] Optionally, when scaling the first area of ​​the image, the entire first area of ​​the image can be scaled; or a specific element within the first area of ​​the image can be scaled. That is, the scaled content refers to the image content after scaling the entire content of the first area of ​​the image; or, the scaled content refers to the image content after scaling a specific element within the first area of ​​the image. For example... Figure 14 As shown, Figure 14 Parts (2) to (4) in the text are zoomed in on the entire screen content; Figure 14 Part (5) only scales the "puppy" element in the first area of ​​the image.

[0213] In summary, the method provided in this application provides for scaling the first area of ​​the image in the glasses, enabling the wearer to better observe the first area of ​​the image in the glasses. For example, when the wearer is about to start moving, if they notice what appears to be a child in front of them, but it is difficult to observe clearly due to the distance, they can select the area where the child is located as the first area of ​​the image and zoom in on it for clearer observation. Or, when piloting a fighter jet, if they notice a black dot that appears to be an enemy in the distance, they can select the first area of ​​the image where the black dot is located and zoom in on it for better observation of the enemy situation.

[0214] 4.2 Rearview Mirror Contents

[0215] The rearview mirror content is a partial view of the external environment from the wearer's non-forward perspective. The rearview mirror content is displayed through at least one of optical rearview mirrors, electronic rearview mirrors, and virtual rearview mirrors.

[0216] Optical rearview mirrors are rearview mirrors that form images based on optical principles; they are currently the most widely used type of rearview mirror in the automotive industry. Electronic rearview mirrors display the external environment from a non-forward-facing perspective on an electronic screen; their imaging usually requires a camera with a communication connection to capture images of the external environment. Virtual rearview mirrors display the external environment from a non-forward-facing perspective in a second area of ​​the screen displayed by electronic glasses; that is, virtual rearview mirrors are invisible without the electronic glasses being worn.

[0217] Optionally, the virtual rearview mirror is displayed normally on the glasses screen; or, based on rearview mirror display conditions, a virtual rearview mirror is displayed on the glasses screen, located in a second area of ​​the glasses screen. The rearview mirror display conditions include at least one of the following: the wearer's gaze is directed towards the trigger area; the trigger area appears on the glasses screen; the wearer requests a human-computer interaction instruction to display the virtual rearview mirror; a warning message appears in the rearview mirror content; or the traffic scenario requires the wearer to observe the rearview mirror content. The trigger area is an area inside the vehicle, i.e., a real-world area. For example, if the vehicle is a car and there is no longer an interior rearview mirror, the trigger area is the original rearview mirror's location. The rearview mirror display conditions are similar to the perspective conditions and will not be elaborated further here.

[0218] Optionally, the position and size of the virtual rearview mirror in the glasses screen can be adjusted, that is, the position and size of the second area can be adjusted.

[0219] A blind spot in a rearview mirror refers to an external area that cannot be seen by the wearer due to obstructions within the mirror's view. In other words, the obstruction may not actually be located in front of the wearer, but is highly likely to be behind, to the side, or rear-to-side of the wearer. For example... Figure 15 In the rearview mirror content shown in the middle (1) section, the obstruction shown in the rearview mirror 80 is the rear side vehicle body 81.

[0220] Optionally, if there is an obstruction in the rearview mirror view, the obstruction is not shown. Alternatively, if there is an obstruction in the rearview mirror view, the obstruction is shown to display the environment of the blind spot. Figure 15 As shown, when the vehicle is a car and the obstruction is the car body; if there is an obstruction in the rearview mirror, the obstruction is not visible, such as... Figure 15 As shown in part (1), the vehicle body 81 displayed in the rearview mirror 80 is not shown in perspective; when there are obstructions in the content of the rearview mirror, the environment of the blind spot is shown through the obstructions, such as... Figure 15 As shown in section (2), for the vehicle body 81 displayed in the rearview mirror 80, the vehicle body 81 is rendered in perspective to display the external environment 82, which is not in a forward-looking view and is obscured by the vehicle body. That is, the content of the rearview mirror may or may not include perspective content.

[0221] Optionally, whether the rearview mirror content includes see-through content is set by the wearer or by the developer.

[0222] In summary, the method provided in this application illustrates the classification of rearview mirror content displayed on the glasses screen. Different types of rearview mirrors can be displayed on the glasses screen, allowing the wearer to observe the external environment of the vehicle from a non-forward perspective (or the external environment behind, to the sides, and to the sides of the vehicle), thus expanding the range of the external environment observable on the glasses screen. Furthermore, this assisted driving method using electronic glasses is adaptable to various vehicle models, whether optical or electronic rearview mirrors, or even in the absence of a rearview mirror, it can add a virtual rearview mirror to observe the external environment from a non-forward perspective while wearing electronic glasses, effectively eliminating blind spots in all directions.

[0223] 4.3 Extended Contents of Rearview Mirrors

[0224] In some embodiments, the glasses display rearview mirror content, such as... Figure 16 As shown, the method also includes:

[0225] Step 410: In response to the triggering operation of the rearview mirror content screen extension mode, display the glasses screen with rearview mirror extension content, which is the screen content after extending a part of the rearview mirror content.

[0226] Optionally, the extended rearview mirror content is a view of the rearview mirror content that has been extended from a portion of the rearview mirror content; or, the extended rearview mirror content is a view of the rearview mirror content that has been extended from the rearview mirror content. The extended rearview mirror content and the rearview mirror content are view of the same perspective, and the center of the extended rearview mirror content is the same as the center of the rearview mirror content. The extended rearview mirror content is as follows: Figure 15 As shown, without displaying the extended content of the rearview mirror, the image related to the rearview mirror in the glasses' view when the wearer observes the rearview mirror is as follows: Figure 15 As shown in part (1), the rearview mirror 80 displays the rearview mirror content, while outside the rearview mirror 80 is the view in front of the vehicle, which shows a guardrail and a tree 83 located outside the guardrail; after triggering the screen expansion mode, as shown Figure 15As shown in part (3), what is displayed outside the rearview mirror 80 is not the front view of the vehicle, but the external area under the same view as the rearview mirror content, which is a non-forward view. Only the guardrail 84 is displayed in this external area. It can be understood that the rearview mirror content is the image content obtained by reflection from the first mirror, and the extended content of the rearview mirror is the image content obtained by reflection from the second mirror. The first mirror and the second mirror are placed in the same position, angle and mirror type, and the size of the first mirror is smaller than that of the second mirror. Or, the rearview mirror content is based on the data collected by the first camera, and the extended content of the rearview mirror is based on the data collected by the second camera. The first camera and the second camera are placed in the same position and angle, and the zoom of the first camera is smaller than that of the second camera.

[0227] Optionally, the extended content of the rearview mirror does not include the rearview mirror content; or, the extended content of the rearview mirror includes the rearview mirror content. In the case where the extended content of the rearview mirror does not include the rearview mirror content, such as... Figure 15 As shown in section (3), the extended content of the rearview mirror is Figure 15 In part (3), the content of the screen other than the rearview mirror 80 is displayed. This content shows an external area outside the range of the rearview mirror, in which a guardrail 84 is displayed; or, if the extended content of the rearview mirror includes the content of the rearview mirror, such as Figure 15 As shown in part (3), the content of the rearview mirror is the content displayed in the mirror area of ​​the rearview mirror 80. At this time, the extended content of the rearview mirror is... Figure 15 All content shown in section (3) of the document. That is, the rearview mirror extension content is the image content displayed outside the rearview mirror content after matching with the rearview mirror content; or, the rearview mirror extension content is the image content that is overlaid on the rearview mirror content after matching with the rearview mirror content.

[0228] Optionally, if the rearview mirror content includes perspective content, the extended content of the rearview mirror also includes perspective content, such as... Figure 15 As shown in section (2), the extended content of the rearview mirror is as follows: Figure 15 As shown in section (4); or, where the rearview mirror content includes perspective content, the extended rearview mirror content includes perspective content, and the rearview mirror content is as follows: Figure 15 As shown in section (2), the extended content of the rearview mirror is as follows: Figure 15 As shown in section (3); or, where the rearview mirror content does not include perspective content, the extended rearview mirror content also does not include perspective content, and the rearview mirror content is as follows: Figure 15 As shown in section (1), the extended content of the rearview mirror is as follows: Figure 15 As shown in section (3); or, where the rearview mirror content does not include perspective content, the extended rearview mirror content includes perspective content, and the rearview mirror content is as follows: Figure 15As shown in section (1), the extended content of the rearview mirror is as follows: Figure 15 As shown in part (4) of the document.

[0229] For example, rearview mirror content 86 and rearview mirror extension content 87 are as follows: Figure 17 As shown, the extended content 87 of the rearview mirror is an extension of the content 86 of the rearview mirror. That is, the screen range of the extended content 87 of the rearview mirror is larger than that of the content 86 of the rearview mirror. The screen range of the extended content 87 of the rearview mirror is usually much smaller than that of the screen range 85 of the glasses. Especially for the interior rearview mirror, the extended content 87 of the rearview mirror will not obstruct the external environment screen of the forward view displayed in the glasses screen too much.

[0230] Optionally, the triggering operation for the extended view mode includes at least one of the following: the wearer's eyes are facing the rearview mirror content; the wearer's head is facing the rearview mirror content; there is a warning message in the warning area corresponding to the extended rearview mirror content; there is a warning message in the predicted movement route corresponding to the extended rearview mirror content; the traffic scene requires the wearer to observe the extended rearview mirror content; the wearer requests a human-computer interaction instruction to activate the extended view mode. Among these, "there is a warning message in the warning area corresponding to the extended rearview mirror content; there is a warning message in the predicted movement route corresponding to the extended rearview mirror content" is similar to "2. There is a warning message in the blind spot of the vehicle" above; "the traffic scene requires the wearer to observe the extended rearview mirror content" is similar to "3. the traffic scene requires the wearer to observe the blind spot of the vehicle" above; and "the wearer's eyes are facing the rearview mirror content; the wearer's head is facing the rearview mirror content; the wearer requests a human-computer interaction instruction to activate the extended view mode" is similar to "1. the wearer has a need to observe perspective content" above, and will not be elaborated further here.

[0231] In some embodiments, the display range of the extended content in the rearview mirror is adjustable; the display range of the extended content is determined in response to a scaling operation. The scaling operation can be triggered by a human-computer interaction command, such as the wearer selecting the corresponding extended content via gesture and saying "expand the display range of the extended content" via voice command; or the wearer instructing the extended content via EEG and requesting a reduction in the display range. The form of the corresponding human-computer interaction command is similar to "1.4 Human-computer interaction instruction for wearer requesting to see through obstructions," and will not be elaborated further here. Optionally, the minimum display range of the extended content is the display range corresponding to the rearview mirror content, and the maximum is the display range of the glasses' view.

[0232] In summary, the method provided in this application can display extended content of the rearview mirror, solving the problem of the limited reflection range of optical rearview mirrors, allowing the wearer to observe more of the scene in the blind spot of the rearview mirror; unlike electronic rearview mirrors, which display a large area of ​​the external environment on a small screen, this method displays extended content of the rearview mirror outside the original rearview mirror, without reducing the size of the rearview mirror content or the extended content, avoiding misjudgment caused by different scaling ratios, and improving driving safety.

[0233] Furthermore, the trigger conditions are set with two aspects in mind. Firstly, user needs are taken into consideration: the extended content of the rearview mirror is only displayed when the wearer expects to see it. It can also record the wearer's observation habits, so that if these habits are met, the wearer doesn't need to actively provide instructions to display the extended content; the glasses can directly display the perspective content based on the wearer's observation habits, improving human-computer interaction efficiency and providing a better user experience. This also allows the wearer to be more immersed in driving, avoiding the need to distract themselves by instructing the glasses to display the extended content. Secondly, it considers scenarios where perspective is needed in actual driving situations, i.e., when there is a danger in the blind spot corresponding to the extended rearview mirror, the extended content needs to be displayed to alert the wearer to the dangerous situation.

[0234] In addition, scaling the field of view of the extended content of the rearview mirror allows the wearer to adjust it freely according to the driving scenario, which can improve the wearer's experience. On the other hand, for some scenarios that require careful observation of non-forward perspectives, such as reversing and changing lanes, the wearer can adjust the field of view of the extended content of the rearview mirror to see a larger field of view of non-forward perspectives, which indirectly improves driving safety.

[0235] 4.4 Virtual Alert Elements

[0236] In some embodiments, the wearer of the electronic glasses should be alerted when a dangerous situation is detected, such as... Figure 18 As shown, the method also includes:

[0237] Step 510: In response to the alert trigger operation, display a virtual alert element on the glasses screen. The virtual alert element is used to indicate the content on the screen that the wearer needs to pay attention to.

[0238] Virtual warning elements are visual elements with warning meanings, used to indicate what the wearer needs to pay attention to. The visual content the wearer needs to pay attention to includes warning information, namely obstacles and / or abnormal situations. This visual content can be content within a blind spot (or the environment within the blind spot) or content outside the blind spot (i.e., the environment not obstructed by objects).

[0239] Virtual alert elements can be at least one of icons, graphics, or animations. For example... Figure 19 As shown, the virtual warning elements displayed on the glasses screen include icons 61 and graphics 62. Virtual warning elements can be highlighted, such as having highlights, borders, or highlighted borders; virtual warning elements can also be dynamic, such as constantly flashing virtual warning elements or warning elements whose colors change over time.

[0240] Optionally, the display effect of the virtual warning element is related to the distance between the vehicle and the warning information, the speed of the vehicle, and the estimated collision time between the vehicle and the warning information. For example, if the virtual warning element is an icon, the closer the vehicle is to the warning information, the larger the icon's display effect; or, if the virtual warning element is an icon, initially green, the icon's color gradually changes from green to orange, and finally to red, as the estimated collision time between the vehicle and the warning information decreases; or, if the virtual warning element has a highlighted border, the brighter the highlighted border becomes, the faster the vehicle travels; or, if the virtual warning element is an icon displaying the distance between the vehicle and the warning information or the estimated collision time between the vehicle and the warning information, the number on the icon will change accordingly as the distance or the estimated collision time changes.

[0241] Optionally, in response to a warning trigger, a virtual obstacle avoidance element is displayed on the glasses screen. This virtual obstacle avoidance element instructs the wearer to avoid the warning information. The virtual obstacle avoidance element indicates at least one of an obstacle avoidance route or obstacle avoidance plan. For example, if a pedestrian is detected crossing a crosswalk, a stop icon is displayed on the glasses screen; or, if a large rock is detected in the middle of the road, a route around the rock is displayed.

[0242] In addition to providing virtual warning or obstacle avoidance elements on the screen, it can also play corresponding voice prompts to remind the wearer.

[0243] Optionally, in response to a warning triggering operation, a warning voice message is played; and / or, in response to a warning triggering operation, an obstacle avoidance voice message is played. For example, in Figure 19In the scenarios shown, a warning voice message will be played: "A child is passing by on the left front, please keep your distance"; or, an obstacle avoidance voice message will be played: "A child is passing by on the left front, please slow down." Alternatively, when driving on a narrow road, a voice prompt will be given based on the distance to the roadside obstacle, such as: "You can pass safely," or "The distance between the left side mirror and the obstacle on the left is too close, only 20cm, please keep your distance."

[0244] The virtual warning elements and / or warning voice are used to indicate at least one of the following: the location of the warning information, the type of the warning information, and the distance to the warning information. The virtual obstacle avoidance elements and / or obstacle avoidance voice are used to indicate at least one of the following: the obstacle avoidance route and the obstacle avoidance plan.

[0245] In summary, the method provided in this application, upon detecting a warning trigger operation, displays virtual warning elements to alert the wearer to the warning information. This allows wearers accustomed to displaying transparent content normally to notice the warning information; it also alerts wearers with poor driving habits, such as those easily distracted; and in the case of autonomous driving, displaying warning information through electronic glasses allows wearers engaged in other activities to notice the warning information and take over driving in a timely manner, thereby improving driving safety.

[0246] 4.5 Retain screen content

[0247] In some embodiments, the perspective content includes perspective content corresponding to multiple occlusions, such as... Figure 20 As shown, the method also includes:

[0248] Step 610: In response to the retention operation for partial occlusion, display the glasses screen with retained image content, which refers to the image content with higher opacity than the perspective content and corresponding to the partial occlusion.

[0249] Optionally, the operation of retaining partial obstruction can also be described as a human-computer interaction instruction from the wearer requesting the retention of partial obstruction; that is, in response to the operation of retaining partial obstruction, the glasses screen displaying the content to be retained is shown; or, in response to the human-computer interaction instruction from the wearer requesting the retention of partial obstruction, the glasses screen displaying the content to be retained is shown. The human-computer interaction instruction from the wearer requesting the retention of partial obstruction is similar to the aforementioned "1.4 Human-computer interaction instruction from the wearer requesting to see through obstructions," and will not be repeated here.

[0250] Retaining the original content refers to the content whose opacity is higher than that of the perspective content; for example, if the opacity of an object in the perspective content is 0%, the opacity of the object in the retained content is 100%; or, if the opacity of an object in the perspective content is 25%, the opacity of the object in the retained content is 75%. Alternatively, retaining the original content can also refer to the content whose opacity is lower than that of the perspective content; for example, if the opacity of an object in the perspective content is 100%, the opacity of the object in the retained content is 0%; or, if the opacity of an object in the perspective content is 75%, the opacity of the object in the retained content is 25%.

[0251] When there are multiple obstructions in the viewfinder, the wearer can adjust the perspective (or the degree of preservation) of each obstruction. For example, ... Figure 21 As shown, the wearer selected four objects to cover using their fingers: the steering wheel, their hands, and the two dashboard panels, totaling three areas. Figure 21 As shown in section (1) of the document. After selecting the desired obstruction, the electronic glasses will automatically recognize the obstruction selected by the wearer and display the glasses screen containing the content to be retained after confirmation, such as... Figure 21 As shown in section (2) of the document. After the wearer confirms that the image displayed on the glasses with the retained image content is the desired image, the retention operation ends, and the image displayed on the glasses with the retained image content is displayed normally, as shown in section (2). Figure 21 As shown in part (3) of the document.

[0252] Optionally, if the conditions for cancellation are met, the display of the retained screen content is cancelled on the glasses screen. The conditions for cancellation include at least one of the following: a warning message exists in the warning area corresponding to the retained screen content; a warning message exists in the predicted movement route corresponding to the retained screen content; or the wearer provides a human-computer interaction instruction to cancel the retained screen content. The conditions “a warning message exists in the warning area corresponding to the retained screen content; a warning message exists in the predicted movement route corresponding to the retained screen content” are similar to “2. A warning message exists in the blind spot of the vehicle” above, and the conditions “human-computer interaction instruction to cancel the retained screen content” are similar to “1.4 Human-computer interaction instruction for the wearer to request to see through obstructions” above, and will not be elaborated further here. For example, such as... Figure 21 As shown in section (4), during driving, a puppy appears in the warning area corresponding to the retained screen content (the blind spot caused by the obstruction corresponding to the retained screen content). The retention effect (or opacity effect) of the retained screen content in this part is canceled, and its transparency is changed to be consistent with the transparency of the transparent content. Optionally, virtual warning elements, such as warning icons and circled graphics, are displayed on the puppy.

[0253] In summary, the method provided in this application allows the wearer to define whether to allow or retain the obstruction, thus increasing the wearer's autonomy in controlling the view through the glasses. For some wearers, if all parts of a vehicle are visible through the glasses, the rapidly changing road markings and road signs at high speeds may distract them and potentially lead to traffic accidents. In such cases, they can retain the base and side parts without allowing them to be seen through, thereby improving driving safety.

[0254] Figure 22 A flowchart illustrating a method for acquiring blind spot images according to an exemplary embodiment of this application is shown. The method is performed by a vehicle device, which may be a vehicle device within the aforementioned computer system. The method includes:

[0255] The vehicle-mounted device has a first communication connection with at least one camera facing the blind spot of the vehicle. The blind spot refers to an external area that cannot be observed by the wearer due to obstruction, such as a road barrier, a component of the vehicle, an occupant, or an item located within the vehicle. For example, the vehicle is a car. To eliminate the vehicle blind spot shown above, the camera can be mounted on the front of the car, on the left and right rearview mirrors, on the A and B pillars, or on the vehicle base. For the vehicle blind spot while driving, the camera can be mounted on road infrastructure (such as streetlights, utility poles, etc.) or on other vehicles; this application embodiment does not limit this. For cameras mounted on vehicles, the first communication connection between the vehicle-mounted device and the camera is typically a wired communication connection, such as a CAN bus or USB connection. For cameras mounted on road infrastructure or other vehicles, the first communication connection between the vehicle-mounted device and the camera is typically a wireless communication connection, such as a Bluetooth connection or a V2X connection. It should be noted that the communication connection between the vehicle-mounted device and the camera mounted on the vehicle can also be a wireless communication connection; this application embodiment does not limit this.

[0256] Step 710: Establish a second communication connection with the electronic glasses, which are worn by a wearer located inside the vehicle.

[0257] Optionally, the second communication connection established between the vehicle device and the electronic glasses includes at least one of a wireless communication connection and a wired communication connection; the connection method of the second communication connection established between the vehicle device and the electronic glasses includes at least one of a direct connection and an indirect connection, wherein a direct connection means that the communication connection between the vehicle device and the electronic glasses does not pass through other communication devices, and an indirect connection means that the communication connection between the vehicle device and the electronic glasses is relayed through other communication devices.

[0258] Optionally, the first communication connection established between the vehicle device and at least one camera includes at least one of wireless communication connection and wired communication connection; the connection method of the first communication connection established between the vehicle device and at least one camera includes at least one of direct connection and indirect connection, where direct connection means that the communication connection between the vehicle device and at least one camera does not pass through other communication devices, and indirect connection means that the communication connection between the vehicle device and at least one camera is relayed through other communication devices.

[0259] The communication connection methods used by the first communication connection and the second communication connection can be the same or different, and the embodiments of this application do not limit this.

[0260] Step 720: Determine auxiliary data based on environmental images captured by at least one camera.

[0261] Among them, the environmental image is acquired based on the first communication connection, and the auxiliary data is used to enable the electronic glasses to display an image with perspective content. The perspective content is the environmental image of the blind spot presented after performing perspective display on the obstruction in the image captured by the electronic glasses.

[0262] The auxiliary data is obtained based on environmental images captured by at least one camera. This can be understood as: auxiliary data is data obtained by encapsulating environmental images captured by at least one camera; or, auxiliary data is data obtained by digitizing environmental images captured by at least one camera; or, auxiliary data is data obtained by processing environmental images captured by at least one camera, such as auxiliary data being digitized data of glasses images with perspective features, or auxiliary data being data obtained by encapsulating environmental images of blind spots, etc.

[0263] Optionally, auxiliary data is determined based on environmental images captured by at least one camera at the current moment; or, auxiliary data is determined based on environmental images captured by at least one camera in the previous n seconds, where n is a positive integer.

[0264] Optionally, the value of n is determined based on at least one of the following: vehicle speed, road conditions, and driving scenario. For example, in high-speed driving scenarios (e.g., vehicle speed exceeding 100 km / h), the value of n is smaller, such as 0.01; in low-speed driving scenarios (e.g., vehicle speed below 10 km / h), the value of n is larger, such as 5; in no-driving scenarios, the value of n can be even larger, such as 10 or even 60; in complex road conditions (e.g., traffic jams, consecutive sharp curves), the value of n is smaller; in simple road conditions, the value of n is larger; in driving scenarios where accidents frequently occur (e.g., starting, reversing), the value of n is smaller. It can be understood that the smaller the value of n, the higher the timeliness of the auxiliary data.

[0265] Optionally, the environmental footage captured by at least one camera is at least one of an image, a video, or a video frame.

[0266] Step 730: Send auxiliary data to the electronic glasses via the second communication connection.

[0267] The environmental imagery of the blind spot is determined based on the environmental images captured by at least one camera.

[0268] In summary, the method provided in this application embodiment establishes a communication connection between the carrier device and the electronic glasses, and transmits auxiliary data for displaying see-through content. This enables the electronic glasses to display see-through content smoothly, while also handling some of the preparation work for the see-through content, reducing the power consumption of the electronic glasses and improving their battery life.

[0269] In some embodiments, auxiliary data is used to display a view with perspective content on the electronic glasses when perspective conditions are met. Perspective conditions are as described above: "1. The wearer needs to observe perspective content," "2. Warning information exists in the blind spots of the vehicle," and "3. The traffic scene requires the wearer to observe the blind spots of the vehicle," and will not be repeated here. Furthermore, trigger conditions, i.e., perspective conditions, are set for the display of perspective content. This ensures that after the wearer puts on the electronic glasses, perspective content will only be displayed when the perspective conditions are met; otherwise, it will not be displayed. This avoids the discomfort caused by constantly displaying perspective content, ensuring that perspective content is displayed only when the perspective conditions are met without affecting the wearer's normal driving.

[0270] Depending on the chosen method, the content of the auxiliary data varies. This auxiliary data includes at least one of the following: an external environment image, an environment image captured by at least one camera, and a view through glasses with see-through capabilities. Different auxiliary data correspond to different methods of synthesizing the driver's view, while synthesizing the view through glasses requires the assistance of an external environment image. This application embodiment will illustrate the following three methods of synthesizing view through glasses:

[0271] Method 1: The vehicle-mounted terminal synthesizes the external environment image, and the electronic glasses synthesize the glasses image with see-through content (the auxiliary data includes at least one external environment image).

[0272] Method 2: Electronic glasses synthesize the external environment image and the glasses image with see-through content (auxiliary data includes at least one environmental image captured by a camera).

[0273] Method 3: The vehicle terminal synthesizes the external environment image and the glasses image with see-through content (the auxiliary data should at least include the glasses image with see-through content).

[0274] The following section will introduce the three methods mentioned above, but the order in which they are introduced does not imply any superiority or inferiority among them.

[0275] Method 1: The vehicle-mounted terminal synthesizes the external environment image, and the electronic glasses synthesize the glasses image with see-through content (the auxiliary data includes at least the external environment image).

[0276] Figure 23 An overall flowchart of an assisted driving method provided in an exemplary embodiment of this application is shown. The method is performed by electronic glasses and further includes:

[0277] In some embodiments, the auxiliary data includes at least one external environment image. Step 220 can be implemented as step 810.

[0278] In some embodiments, steps 810 and 820 may be performed in an alternate order or simultaneously.

[0279] Step 810: Obtain at least one external environment image through the communication connection.

[0280] Step 820: Acquire the image captured by the electronic glasses.

[0281] The electronic glasses include a camera with the same viewing angle as the wearer, used to capture images that align with the wearer's line of sight. It should be noted that the electronic glasses may include one or more cameras, and this application does not limit this. If the electronic glasses include only one camera, only the image captured by that camera is acquired; if the electronic glasses include multiple cameras, the images captured by all cameras are acquired. Alternatively, the image captured by the electronic glasses refers to the image directly observed by the wearer through the lenses of the electronic glasses.

[0282] For example, the electronic glasses include two cameras, one for capturing the image seen by the wearer's left eye and the other for capturing the image seen by the wearer's right eye. The camera capturing the image seen by the left eye can be called the left-eye camera, and the camera capturing the image seen by the right eye can be called the right-eye camera. Based on the image captured by the left-eye camera (left-eye image), the image displayed on the lens (or screen) corresponding to the left eye in the electronic glasses is determined; based on the image captured by the right-eye camera (right-eye image), the image displayed on the lens (or screen) corresponding to the right eye in the electronic glasses is determined. Alternatively, the image captured by the electronic glasses (or binocular image) is obtained by sampling the image captured by the left-eye camera (left-eye image) and the image captured by the right-eye camera (right-eye image), and the image of the glasses is determined based on the image captured by the electronic glasses (or binocular image).

[0283] For example, the electronic glasses include a camera located at the center of the glasses or the center of the wearer's face. Left-eye and right-eye images are sampled from the images captured by the camera (or electronic glasses). The image displayed on the lens (or screen) corresponding to the left eye in the electronic glasses is determined based on the left-eye image; the image displayed on the lens (or screen) corresponding to the right eye in the electronic glasses is determined based on the right-eye image. Alternatively, the image is determined based on the images captured by the camera (or electronic glasses).

[0284] The aforementioned phrases, such as "sampled from the image captured by the left-eye camera (left-eye image) and the image captured by the right-eye camera (right-eye image) to obtain the image captured by the electronic glasses (or binocular image)" or "sampled from the image captured by the camera (or electronic glasses) (or binocular image) to obtain the left-eye image and right-eye image," relate to the field of view observed by the human eye. Typically, such as... Figure 24 As shown, the vertical visual field of the human eye is considered to be 150°, and the horizontal visual field of a single eye is 156°. That is, the visual field of the left eye is 156°, and the visual field of the right eye is also 156°. The maximum visual field of both eyes is 188°. Therefore, for the situation where "the image captured by the electronic glasses (or binocular image) is obtained by sampling the image captured by the left-eye camera (left-eye image) and the image captured by the right-eye camera (right-eye image)," feature matching can be performed on the left-eye and right-eye images to find the overlapping parts of the images, such as... Figure 24 The image is processed by cropping or overlaying overlapping parts of the image to obtain a binocular image that conforms to the binocular field of view. For the "left eye image and right eye image obtained by sampling the image (or binocular image) captured by the camera (or electronic glasses)," the left eye image that conforms to the left eye field of view is obtained by sampling directly from left to right according to the field of view of the left and right eyes, and the right eye image that conforms to the right eye field of view is obtained by sampling from right to left.

[0285] In reality, due to physical limitations such as the camera lens, the area captured by the camera may be larger or smaller than the field of view that the human eye can observe. That is, the area of ​​the view displayed by electronic glasses may be larger or smaller than the direct view seen by the human eye. Figure 10 As shown, there is a row of 5 trees. When observer 41 is not wearing electronic glasses, the direct view they see is image 42. When observer 41 is wearing electronic glasses, the view they see through the glasses can be image 43 (which has a larger area than the direct view), image 42, or image 44 (which has a smaller area than the direct view). It should be noted that... Figure 10The image range shown includes variations in the length and width of the image, but in reality, the influencing factors of the image range include at least one of the length, width, and image center. That is, the difference in image range between the glasses image and the direct-view image is reflected in the difference in length and / or width and / or image center. Specifically, in cases where "the image captured by the electronic glasses (or binocular image) is obtained by sampling the image captured by the left-eye camera (left-eye image) and the image captured by the right-eye camera (right-eye image)" or "the left-eye image and right-eye image are obtained by sampling the image captured by the camera (or electronic glasses) (or binocular image)," the visual field used during sampling may not be the visual field corresponding to the human eye, but rather a visual field size designed by the electronic glasses developer that is larger or smaller than the human eye's visual field. This application's embodiments do not limit this.

[0286] Step 830: Based on at least one external environment image, perform perspective display on the obstructions in the image captured by the electronic glasses to obtain a glasses image with perspective content.

[0287] An external environment view refers to a view of the environment outside the vehicle, and at least one external environment view is used to indicate at least one of the external environment from a forward-looking perspective and the external environment from a non-forward-looking perspective. Optionally, the external environment view includes a view of the environment in the blind spot.

[0288] Optionally, if the viewing angle of the external environment image is consistent with the viewing angle of the image captured by the electronic glasses (meaning the camera parameters of the camera capturing the external environment image are consistent with the camera parameters of the camera in the electronic glasses), then feature points in the image captured by the electronic glasses and at least one external environment image are matched to determine the perspective method of the image captured by the electronic glasses and at least one external environment image. Based on this perspective method and at least one external environment image, obstructions in the image captured by the electronic glasses are displayed using perspective rendering to obtain an image from the glasses with perspective content. The perspective method includes at least one of the following: displaying obstacles in the image captured by the electronic glasses using perspective rendering, displaying the image captured by the electronic glasses using perspective rendering, displaying obstacles in the external environment image using perspective rendering, displaying the environment image corresponding to the obstacles in the external environment image (i.e., the environment image of the blind spot), and displaying the external environment image using perspective rendering.

[0289] Optionally, if the viewing angle of the external environment image is inconsistent with the viewing angle of the image captured by the electronic glasses, the superposition position of the external environment image is determined; based on the superposition position of the external environment image, the obstructions in the image captured by the electronic glasses are displayed in a perspective manner to obtain a glasses image with perspective content.

[0290] In this scenario, the area of ​​the external environment image is greater than, equal to, or smaller than the area of ​​the image captured by the electronic glasses. If the area of ​​the external environment image is greater than the area of ​​the image captured by the electronic glasses, the position of the image captured by the electronic glasses within the external environment image is determined. The external environment image is then cropped to obtain an external environment image with an area equal to that captured by the electronic glasses. Based on at least one cropped external environment image, obstructions in the image captured by the electronic glasses are displayed using perspective rendering, resulting in a glasses image with perspective content. Alternatively, if the area of ​​the external environment image is greater than the area of ​​the image captured by the electronic glasses, based on at least one external environment image, obstructions in the image captured by the electronic glasses are displayed using perspective rendering, resulting in a glasses image with perspective content, where the area of ​​the glasses image is equal to the area of ​​the external environment image. If the area of ​​the external environment image is equal to the area of ​​the image captured by the electronic glasses, based on at least one external environment image, obstructions in the image captured by the electronic glasses are displayed using perspective rendering, resulting in a glasses image with perspective content. When the area of ​​the external environment is smaller than the area of ​​the image captured by the electronic glasses, the position of the external environment in the image captured by the electronic glasses is determined. Based on at least one external environment and its position in the image captured by the electronic glasses, the obstructions in the image captured by the electronic glasses are displayed in a perspective manner to obtain a glasses image with perspective content.

[0291] In some embodiments, based on at least one external environment image, a perspective display is performed on an obstruction in the image captured by the electronic glasses to obtain a glasses image with perspective content. This includes: performing perspective display on a partial area corresponding to the obstruction in the image captured by the electronic glasses to obtain a processed image; and superimposing at least one external environment image and the processed image according to the position of the obstruction to obtain a glasses image with perspective content. Perspective display refers to reducing the opacity of all or part of the image captured by the electronic glasses; or, perspective display refers to increasing the transparency of all or part of the image captured by the electronic glasses. The partial area refers to the area corresponding to the obstruction that needs to be seen through.

[0292] Optionally, when overlaying at least one external environment image and the processed image, the layer corresponding to the at least one external environment image is lower than the layer corresponding to the image captured by the electronic glasses; or, the layer corresponding to the at least one external environment image is higher than the layer corresponding to the processed image; or, the layers corresponding to some external environment images are lower than the layers corresponding to the processed image, and the layers corresponding to some external environment images are higher than the layers corresponding to the processed image. For example, such as Figure 25As shown, the layer corresponding to the aforementioned perspective content is lower than the layer corresponding to the processed image; the layer corresponding to the aforementioned rearview mirror content is higher than the layer corresponding to the processed image; the layer corresponding to the rearview mirror extension content is higher than or equal to the layer corresponding to the processed image; and the layer corresponding to the aforementioned virtual warning element is higher than the layers corresponding to the rearview mirror content and the rearview mirror extension content.

[0293] When the first layer is lower than the second layer, the content of the first layer will be obscured by the content of the second layer. If the content of the second layer has transparency, the content of the first layer can be seen through the content of the second layer. The higher the transparency of the content in the second layer, the clearer the content of the first layer will be after being superimposed. The lower the transparency of the content in the second layer, the blurrier the content of the first layer will be after being superimposed.

[0294] In summary, the method provided in this application illustrates how to obtain a glasses image with perspective content when the auxiliary data includes at least one external environment image. By superimposing the external environment image and the image captured by the electronic glasses, the glasses image can display the environment image of the blind spot included in the external environment image, and can also display the environment image of the vehicle interior captured by the electronic glasses. This ensures that the goal of eliminating blind spots is achieved, and also allows the wearer to better adapt to the glasses image with perspective content.

[0295] Furthermore, by overlaying rather than merging images to obtain the glasses' image, there is no need for excessive alignment and calibration of some image content, reducing the difficulty of obtaining the glasses' image and enabling even low-performance electronic glasses to achieve this assisted driving method.

[0296] Furthermore, the processing of the image captured by the electronic glasses during overlay, rather than processing the external environment or other elements, is because this image is captured automatically by the electronic glasses themselves, without requiring multiple communication connections. This results in lower acquisition latency and better real-time performance. This ensures the real-time nature of the glasses' image and reduces driving risks caused by display latency.

[0297] Figure 23 A flowchart illustrating an exemplary embodiment of this application for acquiring blind spot images is shown. The method is performed by a vehicle device and further includes:

[0298] Step 910: Obtain environmental images captured by at least one camera via a communication connection.

[0299] In some embodiments, step 910 is performed before step 720 described above.

[0300] Optionally, the environmental images captured by at least one camera are captured by at least one camera at the same time; or, the environmental images captured by at least one camera are captured by at least one camera at different times.

[0301] Optionally, at least one camera includes a camera facing the blind spot of the vehicle and a reconstruction-assisted camera. The reconstruction-assisted camera, similar to the camera facing the blind spot, is used to capture environmental images. The captured environmental images will be used to construct the virtual environment. The reconstruction-assisted camera has a different orientation than the camera facing the blind spot; it supplements the capture of environmental images not captured by the camera facing the blind spot.

[0302] Step 920: Construct a virtual environment based on environmental images captured by at least one camera.

[0303] Optionally, a virtual environment is constructed by stitching together environmental images captured by at least one camera. That is, feature points in the environmental images captured by at least one camera are identified; feature points in each environmental image are matched to determine the mapping relationship between the environmental images; based on the mapping relationship between the environmental images, at least one environmental image is fused to obtain a virtual environment, which is a panoramic image obtained by fusing at least one environmental image.

[0304] In some embodiments, environmental images captured by at least one camera are input into a 3D reconstruction model to obtain a virtual environment. The 3D reconstruction model is used to generate a machine learning model of the virtual 3D model corresponding to the input images.

[0305] For example, the 3D reconstruction model may be based on the principle of structured light; or, it may be based on the principle of triangulation; or, it may be based on the principle of binocular vision; or, it may be based on deep learning methods. This application does not limit the model structure or reconstruction principle of the 3D reconstruction model. For example, the 3D reconstruction model may be based on 3D Gaussian Splatting.

[0306] Step 930: Acquire at least one external environment scene from the virtual environment.

[0307] Optionally, the vehicle equipment constructs a virtual camera and adjusts the camera parameters of the virtual camera to capture at least one image of the external environment.

[0308] Optionally, the 3D reconstruction model is also used for rendering, that is, the 3D reconstruction model is also used to capture at least one image of the external environment from the virtual environment.

[0309] In some embodiments, step 930 may be implemented as steps 931 and 932. Step 931 may be performed before or simultaneously with step 910, or may be performed in a different order or simultaneously with step 920.

[0310] Step 931: Obtain the camera parameters of the electronic glasses through the communication connection.

[0311] The camera parameters include at least one of the following: position information, rotation attitude, and zoom information.

[0312] Optionally, the camera parameters are all or some of the parameters in the Transform matrix of the electronic glasses.

[0313] Step 932: Based on the camera parameters of the electronic glasses, acquire at least one external environment image from the virtual environment.

[0314] Based on the camera parameters of the electronic glasses, a virtual camera is constructed in the virtual environment, and the camera parameters of the virtual camera are consistent with those of the electronic glasses; at least one external environment image is captured from the virtual environment through this virtual camera.

[0315] Step 940: Send at least one external environment image to the electronic glasses via a communication connection.

[0316] The auxiliary data includes at least one external environment image, that is, step 730 can be implemented as step 940.

[0317] Optionally, at least one external environment image is compressed and then sent to the electronic glasses; or at least one external environment image is sent directly to the electronic glasses.

[0318] In summary, the method provided in this application embodiment demonstrates how to obtain at least one external environment image when the auxiliary data includes at least one external environment image. By reconstructing a virtual environment and then acquiring the external environment image from the virtual environment according to the camera parameters of the electronic glasses, it is possible to ensure that the acquired external environment image is from the same perspective as the external environment image seen when not wearing the electronic glasses, thus laying the foundation for superimposing the external environment image and the image acquired by the electronic glasses.

[0319] Furthermore, using 3D reconstruction instead of directly stitching images can reduce image distortion caused by the physical limitations of the camera, preventing misjudgments of distance, angle, etc., by the wearer due to distorted images displayed on the glasses screen, thereby improving driving safety.

[0320] Method 2: Electronic glasses synthesize an external environment image and a driver image with perspective (auxiliary data includes at least one environmental image captured by a camera).

[0321] Figure 26 A flowchart illustrating an exemplary embodiment of the driving assistance method provided in this application is shown. The method is performed by electronic glasses, which may be electronic glasses within the aforementioned computer system. The method includes:

[0322] In some embodiments, step 1031 may be performed before or simultaneously with step 1010; step 1031 may be swapped with or performed simultaneously with step 1020; step 1041 may be performed before or simultaneously with step 1010; step 1041 may be performed between step 1010 and step 1020; step 1041 may also be performed simultaneously with step 1020; step 1041 may be performed before or simultaneously with step 1031.

[0323] Step 1010: Obtain environmental images captured by at least one camera via a communication connection.

[0324] Step 1020: Construct a virtual environment based on environmental images captured by at least one camera.

[0325] Step 1030: Acquire at least one external environment scene from the virtual environment.

[0326] In some embodiments, step 1030 may be implemented as steps 1031 and 1032.

[0327] Step 1031: Obtain the camera parameters of the electronic glasses.

[0328] The camera parameters include at least one of the following: position information, rotation attitude, and zoom information.

[0329] Step 1032: Based on the camera parameters of the electronic glasses, acquire at least one external environment image from the virtual environment.

[0330] Steps 1010 to 1030 are similar to steps 910 to 930 above, except that the executing entity is different, and will not be described again here.

[0331] Step 1040: Obtain a glasses image with perspective content based on at least one external environment image.

[0332] In some embodiments, step 1040 may be implemented as steps 1041 and 1042.

[0333] Step 1041: Acquire the image captured by the electronic glasses.

[0334] Step 1042: Based on at least one external environment image, perform perspective display on the obstructions in the image captured by the electronic glasses to obtain a glasses image with perspective content.

[0335] In some embodiments, based on at least one external environment image, a perspective display is performed on an obstruction in the image captured by the electronic glasses to obtain a glasses image with perspective content, including: performing perspective display on a portion of the image captured by the electronic glasses corresponding to the obstruction to obtain a processed image; and superimposing at least one external environment image and the processed image according to the position of the obstruction to obtain a glasses image with perspective content.

[0336] Step 1040 is similar to steps 820 and 830 above, and will not be described again here.

[0337] In summary, the method provided in this application embodiment uses electronic glasses to synthesize the entire perspective content, while the vehicle only provides environmental images captured by at least one camera. This reduces the number of times data such as the camera parameters of the electronic glasses and the images captured by the electronic glasses are transmitted, minimizing the latency of data delivery and improving the real-time performance of the displayed glasses image.

[0338] Method 3: The vehicle terminal synthesizes the external environment image and the driver's image with perspective content (the auxiliary data should at least include the driver's image with perspective content).

[0339] Figure 27 A general flowchart of an assisted driving method provided in an exemplary embodiment of this application is shown. The method comprises electronic glasses and further includes:

[0340] Step 1110: Send at least one of the camera parameters of the electronic glasses and the images captured by the electronic glasses to the vehicle device via the communication connection.

[0341] The camera parameters are used to determine at least one external environment image, and the image captured by the electronic glasses is used to overlay the image with the at least one external environment image to obtain a glasses image with perspective content. The camera parameters include at least one of position information, rotation posture, and zoom information.

[0342] Optionally, the camera parameters of the electronic glasses and the images captured by the electronic glasses can be sent to the vehicle device simultaneously via a communication connection; or, the camera parameters of the electronic glasses and the images captured by the electronic glasses can be sent to the vehicle device sequentially via a communication connection; or, when a request for camera parameters is received from the vehicle device, the camera parameters of the electronic glasses can be sent to the vehicle device via a communication connection; when a request for images captured by the electronic glasses is received from the vehicle device, the images captured by the electronic glasses can be sent to the vehicle device via a communication connection.

[0343] Step 1120: Obtain the view through the glasses with perspective content via the communication connection.

[0344] In summary, the method provided in this application embodiment allows electronic glasses to obtain a glasses image with see-through content synthesized by the vehicle device by simply providing camera parameters and the image captured by the electronic glasses to the vehicle device. This reduces the processing performance requirements of the electronic glasses and enables even low-configuration electronic glasses to display glasses images with see-through content.

[0345] Figure 27 This illustration shows an overall flowchart of a blind spot image acquisition method provided in an exemplary embodiment of this application. The method comprises a vehicle device and further includes:

[0346] In some embodiments, step 1231 may be performed before or simultaneously with step 1210; step 1231 may be performed between step 1210 and step 1220; step 1231 may be performed simultaneously with step 1220; step 1241 may be performed before or simultaneously with step 1210; step 1241 may be performed between step 1210 and step 1220; step 1241 may be performed simultaneously with step 1220; step 1241 may be performed simultaneously with step 1231.

[0347] Step 1210: Obtain environmental images captured by at least one camera via a communication connection.

[0348] Step 1220: Construct a virtual environment based on environmental images captured by at least one camera.

[0349] Step 1230: Acquire at least one external environment scene from the virtual environment.

[0350] In some embodiments, step 1230 may be implemented as steps 1231 and 1232.

[0351] Step 1231: Obtain the camera parameters of the electronic glasses through the communication connection.

[0352] Step 1232: Based on the camera parameters of the electronic glasses, acquire at least one external environment image from the virtual environment.

[0353] Steps 1210 to 1230 are similar to steps 910 to 930 above, and will not be described again here.

[0354] Step 1240: Obtain a glasses image with perspective content based on at least one external environment image.

[0355] In some embodiments, step 1240 may be implemented as steps 1241 and 1242.

[0356] Step 1241: Acquire the images captured by the electronic glasses via a communication connection.

[0357] Step 1242: Based on at least one external environment image, perform perspective display on the obstructions in the image captured by the electronic glasses to obtain a glasses image with perspective content.

[0358] In some embodiments, based on at least one external environment image, a perspective display is performed on an obstruction in the image captured by the electronic glasses to obtain a glasses image with perspective content, including: performing perspective display on a portion of the image captured by the electronic glasses corresponding to the obstruction to obtain a processed image; and superimposing at least one external environment image and the processed image according to the position of the obstruction to obtain a glasses image with perspective content.

[0359] Step 1240 is similar to steps 820 and 830 above, except that the executing entity is different, so it will not be described again here.

[0360] Step 1250: Send the image with see-through content from the glasses to the electronic glasses via the communication connection.

[0361] In summary, the method provided in this application embodiment, which uses a carrier device to synthesize a glasses image with see-through content, can reduce the performance requirements of electronic glasses. Furthermore, since the processing performance of the carrier device is usually high, the synthesis result can be obtained more quickly when synthesizing the glasses image. Even if there is a certain delay due to transmission speed, compared with processing by electronic glasses with poor performance, it can ensure a lower total delay, improve the wearer's viewing experience, and ensure the real-time performance of the glasses image.

[0362] Next, let's take a car as an example and the wearer as the driver. Users can eliminate blind spots caused by the car's structure or road conditions while driving by wearing electronic glasses.

[0363] 1. Activate the X-ray vision feature on the electronic glasses (activate driver assistance mode)

[0364] Users can set the assisted driving mode to be turned on or off on the electronic glasses. When the assisted driving mode is turned on, the view observed by the user through the electronic glasses is a perspective view, also known as a perspective view or a glasses view with perspective content. The perspective view shows the exterior of the vehicle that is physically obscured by the vehicle body, which is the view observed by the user when not wearing the electronic glasses or when the assisted driving mode of the electronic glasses is not turned on.

[0365] For example, when the driver assistance mode is not activated, or rather, when the driver assistance mode is deactivated, the view observed by the user through the electronic glasses is as follows: Figure 9 As shown in section (1), there is a blind spot for the vehicle in this scenario; after activating the assisted driving mode, the view observed by the user through the electronic glasses is as follows. Figure 9 As shown in parts (2), (3), or (4) of the document, in this scenario, the vehicle is displayed as fully transparent or semi-transparent, allowing the user to directly observe the scene within the vehicle's blind spot, or in other words, allowing the user to directly observe the scene outside the vehicle, which is equivalent to eliminating the vehicle's blind spot.

[0366] For the entire vehicle, the perspective view after activating the assisted driving mode is a perspective view of the vehicle body, allowing the user to directly observe the scene that is obscured by the vehicle body, such as... Figure 9 As shown in parts (2), (3), or (4) of the diagram; for exterior rearview mirrors (including the left and right rearview mirrors), the perspective view is the vehicle body displayed in the exterior rearview mirror. The perspective view when the driver assistance mode is not activated is as follows: Figure 15 As shown in section (1), the perspective view when the assisted driving mode is activated is as follows: Figure 15 As shown in section (2); for the interior rearview mirror, if the interior rearview mirror displays a vehicle image, then the perspective image is the vehicle image displayed in the interior rearview mirror.

[0367] 2. Complete the viewpoint (expand the view, extend the view)

[0368] To address the blind spots at the rear of the vehicle caused by the limited viewing range of rearview mirrors, the electronic glasses, when the assisted driving mode is activated, can display an extended view. This extended view can be described as a glasses screen with expanded rearview mirror content. When the electronic glasses detect that the user is looking at the rearview mirror (usually using eye-tracking technology; when the user's eyes are directed towards the rearview mirror area, it is determined that the user is looking at the rearview mirror), the extended view is displayed around the rearview mirror. This extended view expands the image displayed in the rearview mirror, which can be understood as increasing the range of the image displayed by the rearview mirror. For example... Figure 15 and Figure 17 As shown, Figure 15 To extend the view of the exterior rearview mirrors, when the driver assistance mode is not activated, such as Figure 15 As shown in section (1), the area of ​​view that the user can observe through the exterior rearview mirror is relatively small; after the assisted driving mode is turned on and the extended display screen is set, as shown in section (1), the area of ​​view that the user can observe through the exterior rearview mirror is relatively small. Figure 15 As shown in section (3) or (4), an extended view is displayed around the rearview mirror. Optionally, the view in the exterior rearview mirror is displayed as a perspective view, such as... Figure 15 As shown in section (3); or, the view in the exterior rearview mirror is a non-perspective view, such asFigure 15 As shown in part (4) of the document. Figure 17 This refers to the extended view of the interior rearview mirror, which will not be elaborated upon here.

[0369] Optionally, the display area of ​​the extended screen can be manually set by the user; or the display area of ​​the extended screen can be fixed. For example, the user can set the display area of ​​the extended screen through pre-agreed gestures; or the user can set the display area of ​​the extended screen through voice commands.

[0370] 3. Obstacle / Danger / Abnormality Warning

[0371] The electronic glasses will display a warning image when an obstacle or abnormal situation exists within the vehicle's warning area; or, when an obstacle or abnormal situation exists in the vehicle's predicted driving route, the electronic glasses will display a warning image. Abnormal situations include unauthorized pedestrians and vehicles. Unauthorized pedestrians include pedestrians suddenly appearing within the warning area, pedestrians located within the warning area when the vehicle starts moving, and pedestrians on the predicted driving route; abnormal vehicles include vehicles suddenly appearing within the warning area, vehicles located within the warning area when the vehicle starts moving, and vehicles on the predicted driving route. The predicted driving route refers to the vehicle's predicted route within a predicted distance or time, calculated by the vehicle or the electronic glasses based on at least one of the vehicle's recorded driving habits and current vehicle speed and steering wheel deflection angle.

[0372] The warning screen is used to indicate at least one of the following: obstacle location, abnormal situation location, obstacle type, abnormal situation type, distance to the obstacle, and distance to the abnormal situation. Optionally, the warning screen is a screen that marks an obstacle, and / or, a screen that marks an abnormal situation, and / or, a screen that marks an obstacle type, and / or, a screen that marks an abnormal situation type, and / or, a screen that marks the distance to the obstacle, and / or, a screen that marks the distance to the abnormal situation. The warning screen includes at least one of the following: virtual warning elements and virtual obstacle avoidance elements.

[0373] Optionally, the electronic glasses will display an obstacle avoidance screen when there is an obstacle or abnormality within the vehicle's warning area; or, the electronic glasses will display an obstacle avoidance screen when there is an obstacle or abnormality on the vehicle's predicted driving route. The obstacle avoidance screen is used to instruct the user to adjust the driving route, and / or, the obstacle avoidance screen is used to instruct the user on driving methods to avoid obstacles, and / or, the obstacle avoidance screen is used to instruct the user on driving methods to avoid abnormalities.

[0374] Optionally, the vehicle's warning area is not obscured in the electronic glasses; or, the vehicle's warning area is obscured in the electronic glasses. The vehicle's predicted driving route is not obscured in the electronic glasses; or, the vehicle's predicted driving route is obscured in the electronic glasses. If there is an obstacle or abnormality in the vehicle's warning area, and the area where the obstacle or abnormality is located is obscured in the image displayed by the electronic glasses, a partial perspective view of the area corresponding to the obstacle or abnormality is displayed; or, if there is an obstacle or abnormality in the vehicle's predicted driving route, and the area where the obstacle or abnormality is located is obscured in the image displayed by the electronic glasses, a partial perspective view of the area corresponding to the obstacle or abnormality is displayed. That is, for areas with obscured obstacles or abnormalities, perspective processing is performed on the obscured areas.

[0375] Optionally, when the electronic glasses display a warning screen or obstacle avoidance screen, the electronic glasses provide voice prompts for obstacles or abnormal situations. The voice prompts include the location of the obstacle, the location of the abnormal situation, the type of obstacle, the type of abnormal situation, the distance to the obstacle, and the distance to the abnormal situation.

[0376] like Figure 19 As shown, a child suddenly appears in the vehicle's A-pillar blind spot. The vehicle detects the child within its warning area (i.e., the A-pillar blind spot) and sends this warning information to the electronic glasses. The electronic glasses use a perspective-based processing method to display a warning image within the warning area, including a highlighted warning icon. Simultaneously, the electronic glasses also issue a voice prompt: "Child is approaching from the left front; please pay attention to your speed and distance."

[0377] 4. Reserved Area Settings

[0378] If a user prefers to have certain areas of the vehicle interior not transparent, they can customize the designated "reserved area." When the assisted driving mode is activated, the reserved area will be opaque; alternatively, the reserved area will have lower transparency than the non-reserved area; or, the reserved area will have higher opacity than the non-reserved area.

[0379] Optionally, the reserved area settings can be activated via a fixed gesture; or, the reserved area settings can be activated via a voice command; or, the reserved area settings can be activated via a physical button; or, the reserved area settings can be activated via a virtual button.

[0380] When a user sets a reserved area, the electronic glasses will perform image recognition on the user's selected range and automatically fit the reserved area.

[0381] After a user sets a reserved area, if an obstacle or abnormal situation appears in the warning area corresponding to the reserved area, the reserved area will be made transparent. Optionally, a warning screen will be displayed in the area corresponding to the reserved area.

[0382] like Figure 21 As shown, users can set a reserved area using gestures. After confirming the reserved area, it remains visible within the user's field of view. The visibility of the reserved area is canceled and a warning icon is displayed only when an obstacle or abnormal situation appears in the warning area corresponding to the reserved area.

[0383] Please refer to Figure 28 This diagram illustrates a structural block diagram of an assisted driving device provided in an exemplary embodiment of this application. The device has the functionality to implement the aforementioned assisted driving method example; this functionality can be implemented in hardware or by hardware executing corresponding software. The device can be the electronic glasses described above, or it can be housed within electronic glasses worn by a wearer inside a vehicle. Figure 28 As shown, the device may include: a first communication module 1310, a first acquisition module 1320, and a display module 1330.

[0384] A first communication module 1310 is used to establish a communication connection with a vehicle device, the vehicle device including at least one camera facing the blind spot of the vehicle, the blind spot being an external area that the wearer cannot observe due to obstruction by an obstruction.

[0385] The first acquisition module 1320 is used to acquire auxiliary data through the communication connection, the auxiliary data being obtained based on environmental images captured by the at least one camera.

[0386] The first display module 1330 is used to display a wearer's image with see-through content based on the auxiliary data display. The see-through content is the environmental image of the blind spot presented after the obstruction is displayed through the image captured by the electronic glasses.

[0387] In some embodiments, the first display module 1330 is further configured to display the glasses image with perspective content based on the auxiliary data when perspective conditions are met, wherein the perspective conditions are conditions that trigger the display of the perspective content.

[0388] In some embodiments, the perspective conditions include at least one of the following: the wearer's line of sight is directed toward the obstruction; the obstruction appears in the glasses' view; the wearer requests a human-computer interaction instruction to see through the obstruction; a warning message exists in the warning area corresponding to the obstruction, the warning area being an area within the perimeter of the vehicle; a warning message exists in the predicted movement route corresponding to the obstruction, the predicted movement route being the route the vehicle will move within a time threshold or a distance threshold.

[0389] In some embodiments, the perspective content further displays at least one of the following: a semi-transparent full obstruction, a semi-transparent partial obstruction, a semi-transparent full blind spot environment, and a semi-transparent partial blind spot environment.

[0390] In some embodiments, the device further includes a second display module.

[0391] The second display module is used to display a glasses screen with zoomed content in response to a zoom trigger operation on a first area of ​​the screen in the glasses; wherein the first area is a portion of the screen content in the glasses screen, and the zoomed content refers to the screen content after zooming in on the first area.

[0392] In some embodiments, the glasses display rearview mirror content, which is a partial view of the external area from the wearer's non-forward perspective; the device also includes a third display module.

[0393] The third display module is used to respond to the triggering operation of the screen extension mode of the rearview mirror content and display the glasses screen with the extended rearview mirror content, wherein the extended rearview mirror content is the screen content after extending a part of the screen in the rearview mirror content.

[0394] In some embodiments, the triggering operation of the screen extension mode includes at least one of the following: the wearer's eyes are facing the content of the rearview mirror; the wearer's head is facing the content of the rearview mirror; there is a warning message in the warning area corresponding to the extended content of the rearview mirror; the wearer requests a human-computer interaction instruction to enable the screen extension mode.

[0395] In some embodiments, the device further includes an adjustment module.

[0396] An adjustment module is used to adjust the screen range of the extended content of the rearview mirror in response to a scaling operation on the extended content of the rearview mirror.

[0397] In some embodiments, the device further includes a fourth display module.

[0398] The fourth display module is used to display virtual warning elements on the glasses screen in response to a warning trigger operation. The virtual warning elements are used to indicate the screen content that the wearer needs to pay attention to.

[0399] In some embodiments, the perspective content includes perspective content corresponding to a plurality of the obstructions, and the device further includes a fifth display module.

[0400] The fifth display module is used to display a glasses screen with retained image content in response to a retention operation for a partial obstruction. The retained image content refers to the image content with an opacity higher than the perspective content and corresponding to the partial obstruction.

[0401] In some embodiments, the auxiliary data includes at least one external environment image, which is generated based on the environment image captured by the at least one camera; the first display module includes a first acquisition submodule and a first perspective submodule.

[0402] The first acquisition submodule is used to acquire the images captured by the electronic glasses;

[0403] The first perspective submodule is used to perform perspective display on the obstructions in the image captured by the electronic glasses based on the at least one external environment image, so as to obtain a glasses image with perspective content.

[0404] In some embodiments, the auxiliary data includes environmental images captured by at least one camera; the first display module further includes a first construction submodule, a first acquisition submodule, and a first determination submodule.

[0405] The first construction submodule is used to construct a virtual environment based on the environmental images captured by the at least one camera;

[0406] The first acquisition submodule is used to acquire at least one external environment image from the virtual environment;

[0407] The first determining submodule is used to obtain the glasses image with perspective content based on the at least one external environment image.

[0408] In some embodiments, the first acquisition submodule includes a first acquisition unit and a first acquisition unit.

[0409] The first acquisition unit is used to acquire the camera parameters of the electronic glasses, wherein the camera parameters include at least one of position information, rotation attitude and zoom information;

[0410] The first acquisition unit is used to acquire at least one image of the external environment from the virtual environment based on the camera parameters of the electronic glasses.

[0411] In some embodiments, the first construction submodule is further configured to input the environmental images captured by the at least one camera into the three-dimensional reconstruction model to obtain the virtual environment.

[0412] In some embodiments, the first determining submodule includes a second obtaining unit.

[0413] The second acquisition unit is used to acquire the image captured by the electronic glasses;

[0414] The first perspective submodule is also used to perform perspective display on the obstructions in the image captured by the electronic glasses based on the at least one external environment image, so as to obtain a glasses image with perspective content.

[0415] In some embodiments, the first perspective submodule is further configured to perform perspective display on a portion of the image captured by the electronic glasses corresponding to the obstruction, to obtain a processed image; and to superimpose the at least one external environment image and the processed image according to the position of the obstruction, to obtain the glasses image with perspective content.

[0416] In some embodiments, the auxiliary data includes the view through the glasses with see-through content; the device also includes a first transmitting module.

[0417] The first transmitting module is used to transmit, via the communication connection, at least one of the camera parameters of the electronic glasses and the images captured by the electronic glasses to the vehicle device. The camera parameters are used to determine at least one external environment image, and the images captured by the electronic glasses are used to overlay the at least one external environment image to obtain the glasses image with perspective content. The camera parameters include at least one of position information, rotation attitude, and zoom information.

[0418] Please refer to Figure 29 This diagram illustrates a structural block diagram of a blind spot image acquisition device provided in an exemplary embodiment of this application. The device has the functionality to implement the aforementioned blind spot image acquisition method example; this functionality can be implemented in hardware or by hardware executing corresponding software. The device can be the vehicle device described above, or it can be installed within a vehicle device. The vehicle device has a first communication connection with at least one camera facing the blind spot of the vehicle, where the blind spot refers to an external area that the wearer cannot observe due to obstruction. Figure 29 As shown, the device may include: a second communication module 1410, a first determining module 1420, and a second transmitting module 1430.

[0419] A second communication module is used to establish a second communication connection with electronic glasses, which are worn by the wearer located inside the vehicle;

[0420] The first determining module is used to determine auxiliary data based on the environmental images captured by the at least one camera; the environmental images are obtained based on the first communication connection, and the auxiliary data is used to enable the electronic glasses to display a glasses image with see-through content, wherein the see-through content is the environmental image of the blind spot presented after the obstruction is displayed through the image captured by the electronic glasses;

[0421] The second transmitting module is used to transmit the auxiliary data to the electronic glasses via the second communication connection.

[0422] It should be noted that the above embodiments of the apparatus are only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0423] Figure 30 A structural block diagram of electronic glasses provided in an exemplary embodiment of this application is shown.

[0424] Typically, electronic glasses include one or more processors 1501 and one or more memories 1502.

[0425] Processor 1501 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1501 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1501 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1501 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1501 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0426] The memory 1502 may include one or more computer-readable storage media, which may be non-transitory. The memory 1502 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1502 are used to store at least one computer program, which is executed by the processor 1501 to implement the assisted driving method provided in the method embodiments of this application.

[0427] In some embodiments, the electronic glasses may optionally include: a peripheral device interface 1503 and at least one peripheral device. The processor 1501, memory 1502, and peripheral device interface 1503 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1503 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of: a radio frequency circuit 1504, a display screen 1505, a camera assembly 1506, an audio circuit 1507, and a power supply 1508.

[0428] Peripheral interface 1503 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1501 and memory 1502. In some embodiments, processor 1501, memory 1502 and peripheral interface 1503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1501, memory 1502 and peripheral interface 1503 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0429] The radio frequency (RF) circuit 1504 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1504 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1504 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1504 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1504 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi networks. In some embodiments, the RF circuit 1504 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0430] The optical display component 1505 is used to display screen content. This screen content can be real screen content, virtual screen content, or a combination of real and virtual screen content. In some embodiments, the optical display component 1505 is mounted on the lenses of electronic glasses, which also include a projector used to project the screen content onto the optical display component 1505 for display; that is, the electronic glasses are transmissive. In another embodiment, the optical display component 1505 replaces the lenses of the electronic glasses; that is, the optical display component 1505 is mounted on the frame of the electronic glasses; that is, the electronic glasses are non-transmissive. In some embodiments, the optical display component 1505 can be a single component, disposed on the frame or lens of the electronic glasses, with only one eye able to display the image content; or, the images of both eyes of the electronic glasses share a single optical display component 1505, i.e., the optical display component 1505 of the electronic glasses is divided into left and right areas, with image content displayed in both areas; in other embodiments, there can be at least two optical display components 1505, disposed on the two frames or lenses of the electronic glasses respectively; in still other embodiments, there can be at least four optical display components 1505, i.e., four forward-facing and four backward-facing optical display components 1505 are disposed on the two frames of the electronic glasses respectively, with the forward-facing optical display component 1505 used by others to observe the wearer's eye image, and the backward-facing optical display component 1505 used by the wearer to view the wearer's image. In some embodiments, the optical display component 1505 can be a flexible display screen, i.e., the optical display component 1505 on the electronic glasses has a certain curvature. Furthermore, the optical display component 1505 can also be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. The display screen 1505 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0431] The camera assembly 1506 is used to acquire images or videos. Optionally, the camera assembly 1506 includes a front-facing camera and a built-in camera. Typically, the front-facing camera is located at the front of the electronic glasses and is used to capture the image from the electronic glasses, which can be understood as capturing the image in front from the current wearing perspective. The image from the electronic glasses can be used to display the image in front from the wearer's perspective on the display screen 1505, and can also be used to recognize the wearer's gestures to achieve human-computer interaction with the electronic glasses. The built-in camera is located inside the electronic glasses and is used to capture the image of the wearer's eyes. The image of the eyes can be used to estimate the wearer's gaze; in addition, the image of the eyes can also be used for iris recognition to confirm the wearer's identity information. In another embodiment, the camera assembly 1506 includes a camera mounted below the glasses. This camera is used to... In some embodiments, the camera assembly 1506 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0432] Audio circuitry 1507 provides an audio interface between the user and the electronic glasses. Audio circuitry 1507 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to processor 1501 for processing, or input to radio frequency circuitry 1504 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, positioned at different locations on the electronic glasses. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from processor 1501 or radio frequency circuitry 1504 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, audio circuitry 1507 may also include a headphone jack.

[0433] The power supply 1508 is used to power the various components in the electronic glasses. The power supply 1508 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When the power supply 1508 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired connection, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology and wireless charging technology.

[0434] In some embodiments, the electronic glasses also include one or more sensors 1509. The one or more sensors 1509 include, but are not limited to: a gesture sensor 1510, a gyroscope sensor 1511, a pressure sensor 1512, an optical sensor 1513, and a proximity sensor 1514.

[0435] The gesture sensor 1510 can detect the wearer's hand gestures to enable human-computer interaction between the electronic glasses and the wearer.

[0436] The gyroscope sensor 1511 can detect the orientation and rotation angle of the electronic glasses, and can be used to collect the user's 3D movements of the electronic glasses. For example, the gyroscope sensor 1511 can be used to recognize the wearer's head posture.

[0437] The pressure sensor 1512 can be disposed on the side bezel of the electronic glasses and / or on the lower layer of the display screen 1505. When the pressure sensor 1512 is disposed on the side bezel of the electronic glasses, it can detect the user's touch signal on the electronic glasses, and the processor 1501 can perform quick operations based on the touch signal collected by the pressure sensor 1512.

[0438] Optical sensor 1513 is used to collect ambient light intensity. In one embodiment, processor 1501 can control the display brightness of optical display component 1505 based on the ambient light intensity collected by optical sensor 1513. Specifically, when the ambient light intensity is high, the display brightness of optical display component 1505 is increased; when the ambient light intensity is low, the display brightness of optical display component 1505 is decreased. In another embodiment, processor 1501 can also dynamically adjust the shooting parameters of camera component 1506 based on the ambient light intensity collected by optical sensor 1513.

[0439] The proximity sensor 1514, also known as a distance sensor, is typically located on the front of the electronic glasses. The proximity sensor 1514 is used to detect the distance between the user and the front of the electronic glasses.

[0440] Those skilled in the art will understand that Figure 30 The structure shown does not constitute a limitation on electronic glasses and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0441] Figure 31 A structural block diagram of a vehicle device provided in an exemplary embodiment of this application is shown.

[0442] Typically, the vehicle equipment includes one or more processors 1601 and one or more memories 1602.

[0443] Processor 1601 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1601 may be implemented using at least one hardware form selected from DSP, FPGA, and PLA. Processor 1601 may also include a main processor and a coprocessor. The main processor, also known as a CPU, is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1601 may integrate a GPU, which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1601 may also include an AI processor, which is used to handle computational operations related to machine learning.

[0444] The memory 1602 may include one or more computer-readable storage media, which may be non-transitory. The memory 1602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1602 are used to store at least one computer program, which is executed by the processor 1601 to implement the blind spot image acquisition method provided in the method embodiments of this application.

[0445] In some embodiments, the vehicle device may optionally include: a peripheral device interface 1603 and at least one peripheral device. The processor 1601, memory 1602, and peripheral device interface 1603 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1603 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of: a radio frequency circuit 1604, a display screen 1605, a camera assembly 1606, an audio circuit 1607, and a power supply 1608.

[0446] Peripheral interface 1603 can be used to connect at least one I / O-related peripheral device to processor 1601 and memory 1602. In some embodiments, processor 1601, memory 1602 and peripheral interface 1603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1601, memory 1602 and peripheral interface 1603 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0447] The radio frequency (RF) circuit 1604 is used to receive and transmit RF signals, also known as electromagnetic signals. The RF circuit 1604 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1604 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1604 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi networks. In some embodiments, the RF circuit 1604 may also include NFC-related circuitry, which is not limited in this application.

[0448] Display screen 1605 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1605 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1601 for processing. In this case, display screen 1605 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1605, disposed on the front panel of the vehicle device; in other embodiments, there may be at least two display screens, disposed on different surfaces of the vehicle device or in a folded design; in some embodiments, display screen 1605 may be a flexible display screen, disposed on a curved or folded surface of the vehicle device. Furthermore, display screen 1605 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. Display screen 1605 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0449] Camera assembly 1606 is used to acquire images or videos. Optionally, camera assembly 1606 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the vehicle equipment, and the rear-facing camera is located on the rear of the vehicle equipment. In other embodiments, the camera assembly may be a camera mounted on the vehicle and having a communicative connection with the vehicle equipment. In some embodiments, camera assembly 1606 may also include a flash. The flash may be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0450] Audio circuit 1607 provides an audio interface between the user and the vehicle equipment. Audio circuit 1607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to processor 1601 for processing, or input to radio frequency circuit 1604 for voice communication. For stereo acquisition or noise reduction purposes, multiple microphones may be used, positioned at different locations on the vehicle equipment. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from processor 1601 or radio frequency circuit 1604 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, audio circuit 1607 may also include a headphone jack.

[0451] Power supply 1608 is used to supply power to various components in the vehicle equipment. Power supply 1608 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1608 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology and wireless charging technology.

[0452] In some embodiments, the vehicle device further includes one or more sensors 1609. The one or more sensors 1609 include, but are not limited to: an acceleration sensor 1610, a gyroscope sensor 1611, a pressure sensor 1612, an optical sensor 1613, and a proximity sensor 1614.

[0453] The accelerometer 1610 can detect the magnitude of acceleration on the three coordinate axes of a coordinate system established by the vehicle equipment.

[0454] The gyroscope sensor 1611 can measure the orientation and rotation angle of the vehicle equipment. The gyroscope sensor 1611 can work in conjunction with the accelerometer sensor 1610 to collect the user's 3D movements of the vehicle equipment.

[0455] The pressure sensor 1612 can be installed on the side frame of the vehicle device and / or on the lower layer of the display screen 1605. When the pressure sensor 1612 is installed on the side frame of the vehicle device, it can detect the user's grip signal on the vehicle device, and the processor 1601 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1612. When the pressure sensor 1612 is installed on the lower layer of the display screen 1605, the processor 1601 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1605. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0456] Optical sensor 1613 is used to collect ambient light intensity. In one embodiment, processor 1601 can control the display brightness of display screen 1605 based on the ambient light intensity collected by optical sensor 1613. Specifically, when the ambient light intensity is high, the display brightness of display screen 1605 is increased; when the ambient light intensity is low, the display brightness of display screen 1605 is decreased. In another embodiment, processor 1601 can also dynamically adjust the shooting parameters of camera assembly 1606 based on the ambient light intensity collected by optical sensor 1613.

[0457] The proximity sensor 1614, also known as a distance sensor, is typically mounted on the front of the vehicle equipment. The proximity sensor 1614 is used to detect the distance between the user and the front of the vehicle equipment.

[0458] Those skilled in the art will understand that Figure 31 The structure shown does not constitute a limitation on the vehicle equipment and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0459] In an exemplary embodiment, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored. When executed by a processor, the computer program implements the aforementioned assisted driving method and / or the blind spot image acquisition method. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0460] In an exemplary embodiment, a computer program product is also provided, which, when executed by a processor, is used to implement the above-described assisted driving method and / or the blind spot image acquisition method.

[0461] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0462] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A driving assistance method, characterized in that, The method is performed by electronic glasses worn by a wearer inside a vehicle, and the method includes: Establish a communication connection with a vehicle device, the vehicle device including at least one camera facing the blind spot of the vehicle, the blind spot being an external area that the wearer cannot observe due to obstruction; Auxiliary data is acquired through the communication connection, and the auxiliary data is obtained based on environmental images captured by the at least one camera. The auxiliary data display shows the glasses image with perspective content, which is the environmental image of the blind spot presented after the obstruction is displayed through the screen in the image captured by the electronic glasses.

2. The method according to claim 1, characterized in that, The display of the glasses image with perspective content based on the auxiliary data includes: When the perspective condition is met, the auxiliary data displays the image of the glasses with perspective content, and the perspective condition is the condition that triggers the display of the perspective content.

3. The method according to claim 2, characterized in that, The perspective conditions include at least one of the following: The wearer's gaze is directed toward the obstruction; The obstruction appears in the viewfinder; The wearer requests a human-computer interaction instruction that allows them to see through the obstruction; The warning information is present in the warning area corresponding to the obstruction, and the warning area is the area around the vehicle. The warning information is present in the predicted movement route corresponding to the obstruction, and the predicted movement route is the route that the vehicle will move within a time threshold or distance threshold.

4. The method according to any one of claims 1 to 3, characterized in that, The perspective view also shows semi-transparent, wholly or partially obscured objects.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: In response to a zoom trigger operation on the first area of ​​the screen in the glasses, a screen with zoomed content is displayed. The first area of ​​the image refers to a portion of the image content in the glasses, and the scaled content refers to the image content after scaling the first area of ​​the image.

6. The method according to any one of claims 1 to 5, characterized in that, The glasses display a rearview mirror image, which is a partial view of the external area from the wearer's non-forward perspective. The method further includes: In response to a trigger operation on the screen extension mode of the rearview mirror content, a glasses screen with extended rearview mirror content is displayed, wherein the extended rearview mirror content is the screen content after extending a portion of the screen in the rearview mirror content.

7. The method according to claim 6, characterized in that, The triggering operation for the screen expansion mode includes at least one of the following: The wearer's eyes are directed toward the contents of the rearview mirror; The wearer's head is facing the contents of the rearview mirror; The warning area corresponding to the extended content of the rearview mirror contains warning information; The wearer requests the activation of the expanded screen mode as a human-computer interaction instruction.

8. The method according to claim 6 or 7, characterized in that, The method further includes: In response to a scaling operation on the extended content of the rearview mirror, the screen range of the extended content of the rearview mirror is adjusted.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: In response to an alert trigger, a virtual alert element is displayed on the glasses screen, which is used to indicate the screen content that the wearer needs to pay attention to.

10. The method according to any one of claims 1 to 9, characterized in that, The perspective content includes perspective content corresponding to multiple of the occluding objects, and the method further includes: In response to the retention operation for partial obstruction, a glasses screen with retained image content is displayed, wherein the retained image content refers to the image content with an opacity higher than the perspective content and corresponding to the partial obstruction.

11. The method according to any one of claims 1 to 10, characterized in that, The auxiliary data includes at least one external environment image, which is generated based on the environmental images captured by the at least one camera; The display of the glasses image with perspective content based on the auxiliary data includes: Acquire the images captured by the electronic glasses; Based on the at least one external environment image, the obstructions in the image captured by the electronic glasses are displayed in a perspective manner to obtain the glasses image with perspective content.

12. The method according to any one of claims 1 to 10, characterized in that, The auxiliary data includes environmental images captured by at least one camera; The display of the glasses image with perspective content based on the auxiliary data includes: A virtual environment is constructed based on the environmental images captured by the at least one camera; At least one external environment image is captured from the virtual environment; The glasses image with perspective content is obtained based on the at least one external environment image.

13. The method according to claim 12, characterized in that, The acquisition of at least one external environment image from the virtual environment includes: Obtain the camera parameters of the electronic glasses, wherein the camera parameters include at least one of position information, rotation attitude, and zoom information; Based on the camera parameters of the electronic glasses, at least one image of the external environment is captured from the virtual environment.

14. The method according to claim 12, characterized in that, The construction of a virtual environment based on environmental images captured by the at least one camera includes: The environmental images captured by the at least one camera are input into the three-dimensional reconstruction model to obtain the virtual environment.

15. The method according to claim 12, characterized in that, Obtaining the glasses image with perspective content based on the at least one external environment image includes: Acquire the images captured by the electronic glasses; Based on the at least one external environment image, the obstructions in the image captured by the electronic glasses are displayed in a perspective manner to obtain the glasses image with perspective content.

16. The method according to claim 11 or 15, characterized in that, The step of performing perspective-enhancing display on obstructions in the image captured by the electronic glasses based on the at least one external environment image to obtain the glasses image with perspective content includes: The portion of the image captured by the electronic glasses corresponding to the obstruction is rendered with perspective to obtain the processed image. Based on the position of the obstruction, the at least one external environment image is superimposed with the processed image to obtain the glasses image with perspective content.

17. The method according to any one of claims 1 to 16, characterized in that, The auxiliary data includes the view through the glasses; The method further includes: Through the communication connection, at least one of the camera parameters of the electronic glasses and the images captured by the electronic glasses is sent to the vehicle device. The camera parameters are used to determine at least one external environment image. The images captured by the electronic glasses are used to overlay the at least one external environment image to obtain the glasses image with perspective content. The camera parameters include at least one of position information, rotation attitude, and zoom information.

18. A method for acquiring images in blind spots, characterized in that, The method is performed by a vehicle device having a first communication connection with at least one camera facing a blind spot of the vehicle, the blind spot being an external area that cannot be observed by a wearer due to obstruction, the method comprising: A second communication connection is established with electronic glasses, which are worn by the wearer located inside the vehicle; Based on the environmental images captured by the at least one camera, auxiliary data is determined; the environmental images are obtained based on the first communication connection, and the auxiliary data is used to enable the electronic glasses to display a glasses image with see-through content, wherein the see-through content is the environmental image of the blind spot presented after the obstruction is displayed through the image captured by the electronic glasses; The auxiliary data is sent to the electronic glasses via the second communication connection.

19. The method according to claim 18, characterized in that, The auxiliary data is used to enable the electronic glasses to display the image with see-through content when the see-through conditions are met.

20. A driver assistance device, characterized in that, The device is operated by electronic glasses worn by a wearer inside a vehicle, and the device includes: A first communication module is used to establish a communication connection with a vehicle device, the vehicle device including at least one camera facing the blind spot of the vehicle, the blind spot being an external area that the wearer cannot observe due to obstruction by an obstruction. The first acquisition module is used to acquire auxiliary data through the communication connection, the auxiliary data being obtained based on environmental images captured by the at least one camera; The first display module is used to display a view of glasses with perspective content based on the auxiliary data, wherein the perspective content is displayed in the image captured by the electronic glasses as the environment of the blind spot after the obstruction is made transparent.

21. A device for acquiring images in blind spots, characterized in that, The device is executed by a vehicle device and has a first communication connection with at least one camera facing the blind spot of the vehicle, the blind spot being an external area that cannot be observed by the wearer due to obstruction. The device includes: A second communication module is used to establish a second communication connection with electronic glasses, which are worn by the wearer located inside the vehicle; The first determining module is used to determine auxiliary data based on the environmental images captured by the at least one camera; the environmental images are obtained based on the first communication connection, and the auxiliary data is used to enable the electronic glasses to display a glasses image with see-through content, wherein the see-through content is the environmental image of the blind spot presented after the obstruction is displayed through the image captured by the electronic glasses; The second transmitting module is used to transmit the auxiliary data to the electronic glasses via the second communication connection.

22. An electronic pair of glasses, characterized in that, The electronic glasses include: a processor and a memory, wherein the memory stores at least one program; the processor is configured to execute the at least one program in the memory to implement the assisted driving method as described in any one of claims 1 to 17.

23. A vehicle device, characterized in that, The vehicle device includes: a processor and a memory, wherein the memory stores at least one program; the processor is configured to execute the at least one program in the memory to implement the blind spot image acquisition method as described in claim 18 or 19.

24. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores executable instructions, which are loaded and executed by a processor to implement the assisted driving method as described in any one of claims 1 to 17, and / or the blind spot image acquisition method as described in claim 18 or 19.

25. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium. The processor reads and executes the computer instructions from the computer-readable storage medium to implement the assisted driving method as described in any one of claims 1 to 17, and / or the blind spot image acquisition method as described in claim 18 or 19.

Images