Method and device for self-adapting virtual window of smart glasses and smart glasses

Abstract

The present disclosure relates to a method and device for adaptively adjusting a virtual window of smart glasses. The method includes: obtaining a real scene image, determining an initial virtual window based on a target object in the real scene image; obtaining a relative position relationship between a display screen and a camera of the smart glasses, adjusting the initial virtual window according to the relative position relationship to obtain a first virtual window; determining a relative angle relationship between a display screen installation plane and a camera installation plane based on a camera installation plane, adjusting the first virtual window according to the relative angle relationship to obtain a second virtual window; obtaining a real-time distance between a target object and a camera in an actual scene, adjusting the second virtual window according to the real-time distance between the target object and the camera to obtain a target virtual window, and displaying the target virtual window on the display screen. The target virtual window can be accurately superimposed with a target object in the real world, which helps to improve the recognition accuracy of the target object and enhance the user experience.

Description

Technical Field

[0001] The present disclosure relates to the technical field of smart glasses, and in particular to a method and device for adaptively adjusting a virtual window of smart glasses, smart glasses, and a computer-readable storage medium. Background Art

[0002] Augmented Reality (AR), also known as augmented reality, is a relatively new technology that integrates real-world information and virtual-world information. It simulates and processes physical information that is difficult to experience in the real world based on computer and other scientific technologies, and superimposes virtual information content to effectively apply it in the real world. In this process, it can be perceived by human senses, thus realizing the sensory experience of augmented reality. In the prior art, when users wearing AR glasses experience AR application recognition scenes, such as face recognition, license plate recognition, and object recognition, the virtual-real superposition window viewed through the optical lens overlaps with the real object and is misaligned, resulting in a poor user experience. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a method and device for adaptively adjusting the virtual window of smart glasses, smart glasses and a computer-readable storage medium. When there is an installation angle between the camera and the display screen of the smart glasses, the method for adaptively adjusting the virtual window of the smart glasses can adaptively adjust the virtual window according to the installation angle between the camera and the display screen and the real-time distance between the camera and the target object, so that the virtual window is accurately superimposed with the target object in the real world, which helps to improve the recognition accuracy of the target object and enhance the user experience.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for adaptively adjusting a virtual window of smart glasses is provided, which is applied to the smart glasses, wherein the smart glasses include a camera and a display screen, the camera and the display screen are mounted on different planes, and the display screen mounting plane and the camera mounting plane form a preset angle;

[0005] The virtual window adaptive adjustment method comprises the following steps:

[0006] Taking a photo of a real scene to obtain a real scene image, and determining an initial virtual window based on a target object in the real scene image;

[0007] Acquire a relative position relationship between a display screen and a camera of the smart glasses, and adjust the initial virtual window according to the relative position relationship to obtain a first virtual window;

[0008] Determining a relative angle relationship between a display screen installation plane and a camera installation plane based on a camera installation plane, and adjusting the first virtual window according to the relative angle relationship to obtain a second virtual window;

[0009] The real-time distance between the target object and the camera in the actual scene is obtained, the second virtual window is adjusted according to the real-time distance between the target object and the camera to obtain a target virtual window, and the target virtual window is displayed on the display screen.

[0010] According to a second aspect of an embodiment of the present disclosure, a virtual window adaptive adjustment device is provided, comprising:

[0011] An initial virtual window determination module, used to take a photo of a real scene to obtain a real scene image, and determine an initial virtual window based on a target object in the real scene image;

[0012] A first adjustment module, configured to obtain a relative position relationship between a display screen and a camera of the smart glasses, and to adjust the initial virtual window according to the relative position relationship to obtain a first virtual window;

[0013] A second adjustment module, configured to determine a relative angle relationship between a display screen installation plane and a camera installation plane based on a camera installation plane, and adjust the first virtual window according to the relative angle relationship to obtain a second virtual window;

[0014] The third adjustment module is used to obtain the real-time distance between the target object and the camera in the actual scene, adjust the second virtual window according to the real-time distance between the target object and the camera to obtain a target virtual window, and display the target virtual window on the display screen.

[0015] According to a third aspect of an embodiment of the present disclosure, there is provided a pair of smart glasses, the smart glasses comprising a camera and a display screen, the camera and the display screen being mounted on different planes, and the display screen mounting plane and the camera mounting plane forming a preset angle;

[0016] The smart glasses also include at least one processor and at least one memory;

[0017] The memory is electrically connected to the processor;

[0018] The memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method for adaptively adjusting the virtual window as described in any one of the above embodiments.

[0019] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for adaptively adjusting a virtual window as described in any one of the above embodiments is implemented.

[0020] By applying the above-mentioned technical solution of the present application, when there is an installation angle between the camera and the display screen of the smart glasses, the virtual window adaptive adjustment method determines the initial virtual window according to the target object in the real scene image obtained by shooting, and then adjusts the initial virtual window multiple times according to the relative position relationship between the display screen and the camera, and the relative angle relationship between the display screen installation plane and the camera installation plane, and then adaptively adjusts the virtual window according to the real-time distance between the target object and the camera in the actual scene, so that the virtual window is accurately superimposed on the target object in the real world, which helps to improve the recognition accuracy of the target object and enhance the user experience.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

[0022] For better understanding and implementation, the present disclosure is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram of the structure of smart glasses shown in an embodiment of the present disclosure;

[0024] Figure 2 A schematic diagram of a flow chart of a method for adaptively adjusting a virtual window according to an embodiment of the present disclosure;

[0025] Figure 3 This is a flow chart of step S2 of the method for adaptively adjusting a virtual window according to an embodiment of the present disclosure;

[0026] Figure 4 This is a flow chart of step S3 of the method for adaptively adjusting a virtual window according to an embodiment of the present disclosure;

[0027] Figure 5 This is a flow chart of step S4 of the method for adaptively adjusting a virtual window according to an embodiment of the present disclosure;

[0028] Figure 6 This is a schematic diagram of a first application of a face recognition scenario shown in an embodiment of the present disclosure;

[0029] Figure 7 A schematic diagram of a second application of a face recognition scenario shown in an embodiment of the present disclosure;

[0030] Figure 8 It is a structural schematic diagram of a virtual window adaptive adjustment device shown in an embodiment of the present disclosure;

[0031] Fig. 9 A schematic diagram of smart glasses showing an optional embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0033] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0034] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" / "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0035] The virtual window adaptive adjustment method in the embodiment of the present disclosure is applied to smart glasses, especially to AR glasses. Figure 1 , Figure 1 The figure is a schematic diagram of the structure of the smart glasses shown in an embodiment of the present disclosure.

[0036] The smart glasses include a frame 101, and a camera 102 and a display screen 103 installed on the frame 101, wherein the display screen 103 is the lens 103 of the smart glasses, and the lens 103 and the camera 102 are installed on different planes, that is, the display screen 103 and the camera 102 are installed on different planes, and the display screen installation plane and the camera installation plane form a certain preset angle, and the method for determining the preset angle and the specific value are not limited, and can be pre-set based on empirical values.

[0037] Smart glasses can be binocular or monocular. Figure 1 In the embodiment, the smart glasses are binocular glasses with two lenses, which are transparent lenses, and a virtual window can be displayed on the lenses, and the user can see the real world through the lenses when wearing them. In other embodiments, the smart glasses can also be monocular glasses, that is, only one lens is used as a display screen.

[0038] The smart glasses may be VR (Virtual Reality, VR) glasses, AR (Augmented Reality, AR, augmented reality) glasses, or MR (Mixed Reality, mixed reality) glasses and other smart glasses with perspective functions.

[0039] AR glasses are AR headsets, augmented reality head-mounted display devices. Augmented Reality (AR) technology is a technology that cleverly integrates virtual information with the real world. It widely uses a variety of technical means such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, and sensing. It simulates computer-generated text, images, three-dimensional models, music, videos and other virtual information and applies them to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0040] Among them, the virtual-reality overlay window application is an important application in AR glasses. When users view the real world through the lenses of smart glasses, a virtual window will be drawn on the lenses. From the user's perspective, the virtual window is superimposed on the target object in the real world without misalignment. The virtual-reality overlay window can be applied to a variety of recognition scenarios, such as recognizing other people's faces or license plates through smart glasses. Since the camera and lenses of general smart glasses are installed on the same plane, when drawing and adjusting the virtual window, the virtual window can be adjusted according to the positional relationship between the camera and lenses of the smart glasses, so that the virtual window coincides with the target object in the real world and overlaps.

[0041] During the research process, the applicant found that if there is a certain installation angle between the camera and the lens of the smart glasses, that is, the camera and the lens are not located on the same plane, on the one hand, it is beneficial to the appearance design of the smart glasses, and secondly, the user can get a more comfortable wearing experience. However, when there is an installation angle between the camera and the lens of the smart glasses, if the above-mentioned virtual window drawing and adjustment method is still used, there may be a problem that the virtual window and the target object in the real world cannot be accurately superimposed, resulting in a relative linear offset of the virtual window after imaging. In the recognition scene, the position of the user and the target object changes in real time. If the virtual window and the target object cannot be accurately superimposed, it will bring a poor experience to the user and will have a certain impact on the accuracy of the recognition information.

[0042] The virtual window adaptive adjustment method is described in detail below through specific embodiments.

[0043] According to a first aspect of the embodiments of the present disclosure, a method for adaptively adjusting a virtual window of smart glasses is disclosed, which is applied to smart glasses. Figure 2 , Figure 2The present invention is a flowchart of a method for adaptively adjusting a virtual window of smart glasses according to an embodiment of the present invention.

[0044] The method for adaptively adjusting a virtual window of smart glasses comprises the following steps:

[0045] Step S1: photographing a real scene to obtain a real scene image, and determining an initial virtual window based on a target object in the real scene image.

[0046] The real scene image may be an image obtained by photographing the current actual scene through a camera on the smart glasses, and the resolution of the real scene image is the imaging resolution of the camera.

[0047] If the smart glasses are used in recognition scenarios, the target object may be an object to be recognized, such as a face to be recognized or a license plate to be recognized, etc., which are target entities existing in the real world.

[0048] In this embodiment, step S1 can be implemented by the following method: taking a photo of the real scene to obtain a real scene image, determining the target object in the real scene image, and determining the initial virtual window based on the edge of the target object. If the target object does not exist in the real scene image, the image is deleted to prevent the captured image from occupying too much memory of the smart glasses.

[0049] Alternatively, step S1 can be implemented by the following method: based on the characteristics of the object to be identified, the object to be identified in the real scene is searched and judged in advance, the real scene where the target object exists is photographed to obtain a real scene image, and the initial virtual window is determined based on the edge of the target object in the real scene image.

[0050] Step S2: obtaining a relative position relationship between a display screen and a camera of the smart glasses, and adjusting the initial virtual window according to the relative position relationship to obtain a first virtual window.

[0051] Since the initial virtual window is determined based on the target object on the real scene image, and the display screen and camera imaging are located in different coordinate systems, the coordinate systems need to be unified and the initial virtual window needs to be transformed according to the relative position relationship between the display screen and the camera.

[0052] For example, a set of camera imaging data can be collected, the face model is fixed in the same position, and the camera is moved back and forth at distances of 1m, 2m, 3m, etc. to take pictures; then the horizontal position offset of the face frame in the photo is calculated; assuming that the forward and backward translation distance is x, then x1=offset_1; x2=offset_2; x3=offset_3, etc., and a quantitative linear relationship can be obtained. In this way, this linear relationship can be added in the process of coordinate transformation for correction.

[0053] The relative position relationship between the display screen and the camera includes: the physical position relationship between the display screen and the camera and the scaling relationship between the display screen resolution and the camera imaging resolution.

[0054] Optional, see Figure 3 , Figure 3 Schematic diagram of the process of step S2 in the embodiment of the present disclosure.

[0055] Step S2 can be implemented by the following steps:

[0056] S21: Acquire the physical position relationship between the display screen and the camera of the smart glasses, and perform plane coordinate transformation on the initial virtual window based on the physical position relationship.

[0057] In this embodiment, step S21 includes: obtaining camera external parameters and the installation position of the display screen relative to the center of the smart glasses; determining the physical position relationship between the display screen and the camera based on the camera external parameters and the installation position of the display screen relative to the center of the smart glasses; and performing plane coordinate transformation on the initial virtual window based on the physical position relationship between the display screen and the camera.

[0058] The camera extrinsic parameters are parameters in the world coordinate system, such as the position and rotation direction of the camera determined based on the center of the smart glasses. The installation parameters of the display screen are also determined based on the center of the smart glasses, so that the physical position relationship between the display screen and the camera can be determined.

[0059] S22: Acquire a scaling relationship between a display screen resolution and a camera imaging resolution, and scale the initial virtual window after the plane coordinates are converted according to the scaling relationship to obtain a first virtual window.

[0060] In this embodiment, step S22 includes: obtaining a display screen resolution and camera internal parameters; determining a scaling relationship between a virtual scene and a real scene image according to the display screen resolution and the camera internal parameters; and scaling an initial virtual window after plane coordinate conversion according to the scaling relationship to obtain a first virtual window.

[0061] The intrinsic parameters of the camera are parameters related to the camera's own characteristics, such as the camera's focal length, pixel size, etc.

[0062] If the smart glasses have a left camera and a right camera, multiple camera intrinsic parameters form the intrinsic parameter matrices of the left camera and the right camera. The intrinsic parameters of the left camera and the right camera can be the same or adjusted according to actual conditions.

[0063] Step S3, determining a relative angle relationship between the display screen installation plane and the camera installation plane based on the camera installation plane, and adjusting the first virtual window according to the relative angle relationship to obtain a second virtual window.

[0064] In an alternative embodiment, see Figure 4 , Figure 4 Schematic diagram of the process of step S3 in the embodiment of the present disclosure.

[0065] Step S3 can be implemented by the following steps, and step S3 includes:

[0066] S31: Determine a camera installation plane and a display screen installation plane;

[0067] S32: Based on the camera installation plane, determine the relative angle relationship between the display screen installation plane and the camera installation plane.

[0068] Generally, the display screen can determine its installation plane based on the center of the smart glasses, and the camera can determine its installation plane based on the center of the smart glasses. Since the virtual window is displayed on the display screen, the relative angle relationship between the two installation planes can be determined based on the display screen installation plane.

[0069] S33: According to the relative angle relationship, performing spatial coordinate transformation on the first virtual window to obtain a second virtual window.

[0070] If there is an angle between the camera and the display screen, the camera installation plane and the display screen installation plane are different planes. Therefore, it is necessary to obtain the relative angle relationship between the two planes and adjust the virtual window according to the relative angle relationship to unify the virtual window to the display screen installation plane.

[0071] Step S4, obtaining the real-time distance between the target object and the camera in the actual scene, adjusting the second virtual window according to the real-time distance between the target object and the camera to obtain a target virtual window, and displaying the target virtual window on the display screen.

[0072] In an alternative embodiment, see Figure 5 , Figure 5 Schematic diagram of the process of step S4 in the embodiment of the present disclosure.

[0073] Step S4 can be implemented by the following steps, and step S4 includes:

[0074] S41: Obtaining a real-time distance between the target object and the camera according to the imaging principle of the camera.

[0075] Specifically, multiple real scene images are obtained, the face imaging size in each real scene image is measured, and the actual distance between the corresponding camera and the target object is obtained. The correlation between the face imaging size and the actual distance is estimated by analyzing the face imaging size and the corresponding actual distance, and then the actual distance is reversely calculated based on the face imaging size.

[0076] For example, when the target object is a face, when the distance between the camera and the face is 10 meters, the size of the face in the image is 3 square centimeters. When the distance between the camera and the face is 9 meters, the size of the face in the image is 2.8 square centimeters. When the distance between the camera and the face is 8 meters, the size of the face in the image is 2.5 square centimeters. Based on the above data, the correlation between the face imaging size and the actual distance is statistically calculated, and then the actual distance is inferred based on the face imaging size.

[0077] S42: performing a linearization operation based on a predetermined distance linearization conversion relationship and the real-time distance between the target object and the camera to obtain a linear distance between the target object and the camera.

[0078] Among them, the predetermined distance linearization conversion relationship includes:

[0079] Pre-acquire several distance values ​​between the target object and the camera;

[0080] Based on the distance values, counting the target object size and the virtual window size on the corresponding real scene image;

[0081] A distance linearization conversion relationship is determined according to the distance values, the corresponding target object dimensions and the virtual window size.

[0082] S43: adjusting the second virtual window according to the linear distance between the target object and the camera to obtain a target virtual window.

[0083] By adjusting the second virtual window according to the linear distance between the target object and the camera, the amount of calculation can be reduced and the delay can be reduced.

[0084] S44: Draw the target virtual window and display it on the display screen.

[0085] See also Figure 6 and Figure 7 , Figure 6 This is a schematic diagram of a first application of a face recognition scenario shown in an embodiment of the present disclosure; Figure 7 This is a schematic diagram of a second application of a face recognition scenario shown in an embodiment of the present disclosure.

[0086] The virtual window adaptive adjustment method can be applied to face recognition scenarios or license plate recognition scenarios, as well as other recognition scenarios, and has a wide range of applications. Optionally, face recognition can be implemented by the smart glasses themselves or by a face recognition system.

[0087] In an optional embodiment, the smart glasses can be connected to a face recognition system. When a face in a real scene is photographed, the photo can be transmitted to the face recognition system, and the face recognition system then returns the face recognition information.

[0088] Figure 6 In the embodiment, the method for adaptively adjusting the virtual window is applied to the face recognition scenario of smart glasses. The circle is the face to be recognized, the dotted box can be the target virtual window in this embodiment, and the information in the solid box is the information obtained by recognizing the face in the virtual box, such as the gender corresponding to the face, and / or the name, and / or the ID number, and / or the geographical location, and other relevant information.

[0089] Figure 7 In the video, an image is displayed on the computer. The image is virtual. In reality, the face recognition system does not include relevant information about the person in the image. Therefore, the face in the image cannot be recognized. The word "Unknown" is displayed below, along with some image information.

[0090] By applying the above-mentioned technical solution of the present application, when there is an installation angle between the camera and the display screen of the smart glasses, the virtual window adaptive adjustment method determines the initial virtual window according to the target object in the real scene image obtained by shooting, and then adjusts the initial virtual window multiple times according to the relative position relationship between the display screen and the camera, and the relative angle relationship between the display screen installation plane and the camera installation plane, and then adaptively adjusts the virtual window according to the real-time distance between the target object and the camera in the actual scene, so that the virtual window is accurately superimposed on the target object in the real world, which helps to improve the recognition accuracy of the target object and enhance the user experience.

[0091] According to the second aspect of the embodiment of the present disclosure, a virtual window adaptive adjustment device is disclosed, which can be used to execute the virtual window adaptive adjustment method of the smart glasses in the corresponding embodiment of the present application, and has corresponding functions and beneficial effects. For details not disclosed in the embodiment of the virtual window adaptive adjustment device of the present application, please refer to the content of the virtual window adaptive adjustment method in the above embodiment of the present application.

[0092] See also Figure 8 , Figure 8 The figure is a schematic diagram of the structure of a virtual window adaptive adjustment device according to an embodiment of the present disclosure.

[0093] A virtual window adaptive adjustment device, comprising:

[0094] An initial virtual window determination module 801 is used to take a photo of a real scene to obtain a real scene image, and determine an initial virtual window based on a target object in the real scene image;

[0095] A first adjustment module 802 is used to obtain a relative position relationship between a display screen and a camera of the smart glasses, and adjust the initial virtual window according to the relative position relationship to obtain a first virtual window;

[0096] A second adjustment module 803 is used to determine the relative angle relationship between the display screen installation plane and the camera installation plane based on the camera installation plane, and adjust the first virtual window according to the relative angle relationship to obtain a second virtual window;

[0097] The third adjustment module 804 is used to obtain the real-time distance between the target object and the camera in the actual scene, adjust the second virtual window according to the real-time distance between the target object and the camera to obtain a target virtual window, and display the target virtual window on the display screen.

[0098] The virtual window adaptive adjustment device of this embodiment, when there is an installation angle between the camera and the display screen of the smart glasses, the virtual window adaptive adjustment method determines the initial virtual window according to the target object in the real scene image obtained by shooting, and then adjusts the initial virtual window multiple times according to the relative position relationship between the display screen and the camera, and the relative angle relationship between the display screen installation plane and the camera installation plane, and then adaptively adjusts the virtual window according to the real-time distance between the target object and the camera in the actual scene, so that the virtual window is accurately superimposed on the target object in the real world, which helps to improve the recognition accuracy of the target object and enhance the user experience.

[0099] It is worth noting that the virtual window adaptive adjustment device provided in the above embodiment only uses the division of the above functional modules as an example when executing the virtual window adaptive adjustment method of the above smart glasses. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the virtual window adaptive adjustment method device provided in the above embodiment and the virtual window adaptive adjustment method embodiment belong to the same concept, and the embodiment process is detailed in the embodiment, which will not be repeated here.

[0100] According to a third aspect of an embodiment of the present disclosure, a pair of smart glasses is provided, which may be VR glasses, AR glasses, or MR glasses and other smart glasses, which are not limited in this embodiment.

[0101] See also Figure 1 and Fig. 9 , Figure 1 is a schematic diagram of the structure of smart glasses shown in an embodiment of the present disclosure, Fig. 9 A functional block diagram of the smart glasses according to an embodiment of the present disclosure.

[0102] The smart glasses are AR glasses, which include a camera and a display screen. The camera and the display screen are installed on different planes, and the display screen installation plane and the camera installation plane form a preset angle.

[0103] The smart glasses include at least one processor 110 , at least one network interface 120 , a user interface 130 , a memory 140 , and at least one communication bus 150 .

[0104] The communication bus 150 is used to realize the connection and communication between these components.

[0105] The user interface 130 may include an interface for connecting to a display screen, and the optional user interface 130 may also include a standard wired interface or a wireless interface.

[0106] The network interface 120 may optionally include a standard wired interface or a wireless interface (such as a WIFI interface).

[0107] Among them, the processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts within the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the processor 110, and calling data stored in the memory 140. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic arrays (Programmable logic arrays, PLA). The processor 110 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, and it can be implemented separately through a chip.

[0108] The memory 140 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). Optionally, the memory 140 includes a non-transitory computer-readable storage medium. The memory 140 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 140 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area may store data involved in the above-mentioned method embodiments, etc. The memory 140 may optionally be at least one storage device located away from the aforementioned processor 110. As Fig. 9 As shown, the memory 140 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an operating application program of a smart device.

[0109] The memory 140 stores a computer program, and the computer program is suitable for being loaded by the processor 110 and executing the virtual window adaptive adjustment method as described in any one of the above embodiments.

[0110] The above-mentioned smart glasses can be used to execute the contents of the virtual window adaptive adjustment method of the corresponding embodiment of the present application, and have corresponding functions and beneficial effects.

[0111] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. The computer program is suitable for being loaded by a processor and executing relevant operations of the virtual window adaptive adjustment method as described in the above embodiment, and has corresponding functions and beneficial effects.

[0112] Among them, computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0113] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0114] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A method for adaptively adjusting a virtual window of smart glasses, applied to smart glasses, wherein the smart glasses include a camera and a display screen, the camera and the display screen are mounted on different planes, and the display screen mounting plane and the camera mounting plane form a preset angle; characterized in that: The virtual window adaptive adjustment method comprises the following steps: Taking a photo of a real scene to obtain a real scene image, and determining an initial virtual window based on a target object in the real scene image; Acquiring a relative position relationship between a display screen and a camera of the smart glasses, and adjusting the initial virtual window according to the relative position relationship to obtain a first virtual window, comprising: acquiring a physical position relationship between the display screen and the camera of the smart glasses, and performing a plane coordinate transformation on the initial virtual window based on the physical position relationship; Acquire a scaling relationship between a display screen resolution and a camera imaging resolution, and scale the initial virtual window after the plane coordinates are converted according to the scaling relationship to obtain a first virtual window; Determining a relative angle relationship between a display screen installation plane and a camera installation plane based on a camera installation plane, and adjusting the first virtual window according to the relative angle relationship to obtain a second virtual window; The real-time distance between the target object and the camera in the actual scene is obtained, the second virtual window is adjusted according to the real-time distance between the target object and the camera to obtain a target virtual window, and the target virtual window is displayed on the display screen.

2. The method for adaptively adjusting the virtual window of smart glasses according to claim 1, characterized in that: The relative position relationship between the display screen and the camera includes: the physical position relationship between the display screen and the camera and the proportional scaling relationship between the display screen resolution and the camera imaging resolution.

3. The method for adaptively adjusting the virtual window of smart glasses according to claim 2, characterized in that: The obtaining of the physical position relationship between the display screen and the camera of the smart glasses, and performing plane coordinate transformation on the initial virtual window based on the physical position relationship, comprises: Obtain the external parameters of the camera and the installation position of the display screen relative to the center of the smart glasses; Determining a physical position relationship between the display screen and the camera according to the camera external parameters and the installation position of the display screen relative to the center of the smart glasses; The initial virtual window is subjected to a plane coordinate transformation based on a physical position relationship between the display screen and the camera.

4. The method for adaptively adjusting the virtual window of smart glasses according to claim 2, characterized in that: The step of acquiring a scaling relationship between a display screen resolution and a camera imaging resolution, and scaling an initial virtual window after plane coordinate conversion according to the scaling relationship to obtain a first virtual window, includes: Get the display resolution and camera parameters; Determining a scaling relationship between a virtual scene and a real scene image according to the display screen resolution and the camera internal parameters; The initial virtual window after the plane coordinate conversion is scaled according to the scaling relationship to obtain a first virtual window.

5. The method for adaptively adjusting the virtual window of smart glasses according to claim 1, characterized in that: The determining the relative angle relationship between the display screen installation plane and the camera installation plane based on the camera installation plane, and adjusting the first virtual window according to the relative angle relationship to obtain the second virtual window, comprises: Determine the camera installation plane and the display screen installation plane; Based on the camera installation plane, determine the relative angle relationship between the display installation plane and the camera installation plane; According to the relative angle relationship, the first virtual window is transformed into a space coordinate to obtain a second virtual window.

6. The method for adaptively adjusting the virtual window of smart glasses according to claim 1, characterized in that: The step of acquiring a real-time distance between a target object and a camera in an actual scene, adjusting the second virtual window according to the real-time distance between the target object and the camera to obtain a target virtual window, and displaying the target virtual window on a display screen includes: The real-time distance between the target object and the camera is obtained according to the imaging principle of the camera; Performing a linearization operation based on a predetermined distance linearization conversion relationship and the real-time distance between the target object and the camera to obtain a linear distance between the target object and the camera; Adjusting the second virtual window according to the linear distance between the target object and the camera to obtain a target virtual window; The target virtual window is drawn and displayed on a display screen.

7. The method for adaptively adjusting the virtual window of smart glasses according to claim 6, characterized in that: The predetermined distance linearization conversion relationship includes: Pre-acquire several distance values ​​between the target object and the camera; Based on the distance values, counting the target object size and the virtual window size on the corresponding real scene image; A distance linearization conversion relationship is determined according to the distance values, the corresponding target object dimensions and the virtual window size.

8. A virtual window adaptive adjustment device, characterized in that: include: An initial virtual window determination module, used to take a photo of a real scene to obtain a real scene image, and determine an initial virtual window based on a target object in the real scene image; The first adjustment module is used to obtain the relative position relationship between the display screen and the camera of the smart glasses, and adjust the initial virtual window according to the relative position relationship to obtain the first virtual window, including: obtaining the physical position relationship between the display screen and the camera of the smart glasses, and performing plane coordinate conversion on the initial virtual window based on the physical position relationship; obtaining the scaling relationship between the display screen resolution and the camera imaging resolution, and scaling the initial virtual window after the plane coordinate conversion according to the scaling relationship to obtain the first virtual window; A second adjustment module, configured to determine a relative angle relationship between a display screen installation plane and a camera installation plane based on a camera installation plane, and adjust the first virtual window according to the relative angle relationship to obtain a second virtual window; The third adjustment module is used to obtain the real-time distance between the target object and the camera in the actual scene, adjust the second virtual window according to the real-time distance between the target object and the camera to obtain a target virtual window, and display the target virtual window on the display screen.

9. A smart glasses, characterized in that: The smart glasses include a camera and a display screen, the camera and the display screen are mounted on different planes, and the display screen mounting plane and the camera mounting plane form a preset angle; The smart glasses also include at least one processor and at least one memory; The memory is electrically connected to the processor; The memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the virtual window adaptive adjustment method according to any one of claims 1 to 7.

10. The smart glasses according to claim 9, characterized in that: The smart glasses are AR glasses.

11. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for adaptively adjusting a virtual window as claimed in any one of claims 1 to 7 is implemented.

Images