Visual map generation method, device, terminal and storage medium

Abstract

The present application belongs to the field of terminal technology, and specifically, relates to a visual map generation method, device, terminal and storage medium. A visual map generation method is applied to a terminal with a binocular camera, and the method comprises: obtaining an image of a target area captured by the binocular camera; obtaining key image frames in the image, and obtaining feature points of each key image frame; based on the constraint information of the key image frame and the parameter expression information of the feature points, generating a visual map corresponding to the target area. The use of this application can improve the accuracy of visual map generation.

Description

Technical Field

[0001] The present application belongs to the field of terminal technology, and specifically, relates to a visual map generation method, device, terminal and storage medium. Background Art

[0002] With the continuous development of terminal technology, the functions supported by terminals are increasing. In the field of building visual maps, how to improve the accuracy of visual map construction has become the focus of users. The terminal can use a monocular or binocular camera to collect image data of the target scene. The terminal can use the collected video stream or picture as input data, and use visual SLAM (Simultaneous Localization and Mapping) technology or SFM (Structure From Motion) algorithm to build a visual map of the target scene. Summary of the invention

[0003] The embodiment of the present application provides a visual map generation method, device, terminal and storage medium, which can improve the accuracy of visual map generation. The technical solution of the embodiment of the present application is as follows:

[0004] In a first aspect, an embodiment of the present application provides a visual map generation method, which is applied to a terminal with a binocular camera, and the method includes:

[0005] Acquire an image of the target area captured by the binocular camera;

[0006] Acquire key image frames in the image, and acquire feature points of each key image frame;

[0007] Based on the constraint information of the key image frame and the parameter expression information of the feature points, a visual map corresponding to the target area is generated.

[0008] In a second aspect, an embodiment of the present application provides a visual map generating device, the device comprising:

[0009] An image acquisition unit, used to acquire an image of a target area captured by the binocular camera;

[0010] A feature point acquisition unit, used to acquire key image frames in the image, and acquire feature points of each key image frame;

[0011] A map generating unit is used to generate a visual map corresponding to the target area based on the constraint information of the key image frame and the parameter expression information of the feature point.

[0012] In a third aspect, a terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the methods described in the first aspect when executing the computer program.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements any of the methods described above when executed by a processor.

[0014] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in the first aspect of the embodiment of the present application. The computer program product may be a software installation package.

[0015] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least:

[0016] In one or more embodiments of the present application, by acquiring an image of the target area captured by a binocular camera, acquiring key image frames in the image and acquiring feature points of each key image frame, and based on the constraint information of the key image frames and the parameter expression information of the feature points, a visual map corresponding to the target area is generated. Therefore, when the texture differentiation of the target area is low, the terminal can generate a visual map of the target area based on the constraint information of the key image frames and the parameter expression information of the feature points, thereby reducing the situation where the accuracy of the visual map of the target area generated based only on visual information is low, and the accuracy of the visual map generation can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A background schematic diagram showing a visual map generation method applied to an embodiment of the present application;

[0019] Figure 2 A schematic diagram showing an effect diagram of a light beam adjustment factor of a visual map generation method applied to an embodiment of the present application;

[0020] Figure 3 A system architecture diagram showing a visual map generation method applied to an embodiment of the present application;

[0021] Figure 4 A schematic diagram showing a flow chart of a visual map generation method according to an embodiment of the present application;

[0022] Figure 5 A schematic diagram showing a flow chart of a visual map generation method according to an embodiment of the present application;

[0023] Figure 6 A schematic diagram showing a flow chart of a visual map generation method according to an embodiment of the present application;

[0024] Figure 7 A schematic diagram showing a flow chart of a visual map generation method according to an embodiment of the present application;

[0025] Figure 8 A schematic diagram showing the effect of a light beam adjustment factor of a visual map generation method according to an embodiment of the present application;

[0026] Fig. 9 An example schematic diagram showing an image captured by a binocular camera according to an embodiment of the present application;

[0027] Fig.10 An example schematic diagram showing a visual map according to an embodiment of the present application;

[0028] Fig.11 An example schematic diagram showing a visual map according to an embodiment of the present application;

[0029] Fig.12 An example schematic diagram showing an image captured by a binocular camera according to an embodiment of the present application;

[0030] Fig.13 An example schematic diagram showing a visual map according to an embodiment of the present application;

[0031] Fig.14 An example schematic diagram showing a visual map according to an embodiment of the present application;

[0032] Fig.15 A schematic diagram showing the structure of a visual map generating device according to an embodiment of the present application;

[0033] Fig.16 A schematic diagram showing the structure of a visual map generating device according to an embodiment of the present application;

[0034] Fig.17 A schematic diagram showing the structure of a visual map generating device according to an embodiment of the present application;

[0035] Fig.18 A schematic diagram showing the structure of a visual map generating device according to an embodiment of the present application;

[0036] Fig.19A schematic diagram showing the structure of a visual map generating device according to an embodiment of the present application;

[0037] Fig. 20 A schematic diagram showing the structure of a terminal according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0038] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of them. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0039] In the description of the present application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present application, it should be noted that, unless otherwise clearly specified and limited, "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood in specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are an "or" relationship.

[0040] With the development of science and technology, the construction of visual maps has received more and more attention. Figure 1 A background diagram of a visual map generation method applied to an embodiment of the present application is shown. Figure 1As shown, in the field of map construction, the terminal can obtain a large number of feature points in different image frames. By matching the same feature points in different image frames, the terminal can obtain the feature trajectory corresponding to each image frame. For example, the terminal can obtain the feature trajectory corresponding to each image frame through 2D-2D matching of adjacent image frames. The motion state of the camera is inferred through the feature trajectory, and then based on the position information of the feature points and the motion state of the camera, the terminal can obtain the three-dimensional coordinates of the feature points, and then obtain a corresponding series of three-dimensional points. Based on the obtained three-dimensional points, the terminal can construct a three-dimensional space image, that is, the terminal can construct the prototype of the visual map of the target area. When the terminal can construct the prototype of the visual map of the target area, the terminal can also use the vocabulary tree to detect whether a closed loop appears. When the terminal obtains the three-dimensional space image, the terminal can optimize the posture and merge similar three-dimensional points through a closed loop, and then perform global posture and three-dimensional point joint optimization to obtain a visual map of the target area. Figure 1 The beam adjustment factor effect diagram obtained by the technical solution shown in the figure can be shown as Figure 2 shown.

[0041] It is easy to understand that Figure 1 The technical solution shown in the figure only relies on visual information to build a visual map. When there is a scene with low texture differentiation in the target area, it is impossible to extract too many feature points. Figure 1 The technical solution shown is to build a visual map based only on visual information, which may result in a low accuracy of the built visual map. For example, when the target area is indoors, when building a visual map for the indoor area, it is impossible to extract many feature points from the white wall or glass due to the low texture differentiation of the white wall or glass. Therefore, the accuracy of the visual map built by the terminal for the indoor area is low.

[0042] According to some embodiments, Figure 3 A system architecture diagram of a visual map generation method applied to an embodiment of the present application is shown. Figure 3As shown, the execution subject of the embodiment of the present application is a terminal, which has a binocular camera terminal, and the terminal includes but is not limited to: a robot, a wearable device, a handheld device, a personal computer, a tablet computer, a vehicle-mounted device, a smart phone, a computing device or other processing devices connected to a wireless modem. In different networks, terminal devices can be called different names, such as: electronic equipment, user equipment, access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication device, user agent or user device, cellular phone, cordless phone, personal digital assistant (PDA), terminal equipment in the fifth generation mobile communication technology (5th generation mobile networks, 5G) network or future evolution network, etc. The terminal is set with an operating system that can run on the terminal, which is a program for managing and controlling terminal hardware and terminal applications, and is an indispensable system application for the terminal. The operating system includes but is not limited to Android system, IOS system, Windows phone (WP) system and Ubuntu mobile version operating system, etc.

[0043] The terminal 10 in the present application includes a binocular camera 100 and a processor 200. The terminal 10 obtains the image of the target area captured by the binocular camera 100, and the processor 200 of the terminal can obtain the key image frames in the image and obtain the feature points of each key image frame. When the key image frames in the image are obtained and the feature points of each key image frame are obtained, the processor 200 of the terminal can obtain the constraint information of the key image frames and the parameter expression information of the feature points. The processor 200 of the terminal can generate a visual map corresponding to the target area based on the constraint information of the key image frames and the parameter expression information of the feature points. Therefore, when the texture distinction of the target area is low, the terminal can generate a visual map of the target area based on a small number of feature points, the constraint information of the key image frames and the parameter expression information of the feature points, thereby reducing the situation where the accuracy of the visual map of the target area generated based only on visual information is low, and the accuracy of visual map generation can be improved.

[0044] The present application is described in detail below with reference to specific embodiments.

[0045] In one embodiment, Figure 4 As shown, a visual map generation method is proposed, which can be implemented by a computer program and can be run on a terminal based on a binocular camera. The computer program can be integrated into an application or run as an independent tool application.

[0046] Specifically, the visual map generation method includes:

[0047] S101, obtaining an image of a target area captured by a binocular camera;

[0048] According to some embodiments, the execution subject of the embodiments of the present application is a terminal, which is a terminal with a binocular camera. A binocular camera refers to two parallel cameras that can simulate the operation of human eyes, so the images captured by the two parallel cameras can be considered as the scenes within the field of view seen by the human eye.

[0049] It is easy to understand that the target area refers to the area corresponding to the image, and the target area does not specifically refer to a fixed area. The target area can be changed based on the area change instruction input by the user, or it can be changed based on the area change method set by the terminal. The area change instructions input by the user include but are not limited to voice change instructions, click change instructions, timed change instructions, etc. The area change method set by the terminal includes but is not limited to a timed change method, a method of changing the target area when the visual map of the previous area is generated, and a method of determining the unknown area as the target area when entering an unknown area is detected, etc.

[0050] According to some embodiments, when the terminal generates a visual map of the target area, the terminal may obtain an image of the target area captured by the binocular camera. The image may be captured by the terminal directly controlling the camera, or may be captured by the terminal in advance controlling the camera, and the image may be stored in a memory of the terminal, for example.

[0051] It is easy to understand that the image may include multiple image frames. For example, the multiple image frames may be image frames including different feature points in the target area. The multiple image frames may also be image frames with repeated feature points. The image captured by the binocular camera acquired by the terminal may be, for example, an A image frame, a B image frame, a C image frame, a D image frame, an E image frame, and an F image frame.

[0052] S102, obtaining key image frames in the image, and obtaining feature points of each key image frame;

[0053] According to some embodiments, a key image frame refers to an image frame in an image used to generate a visual map. The key image frame does not specifically refer to a fixed key image frame. For example, when the target area changes, the key image frame will also change accordingly. When the method by which the terminal obtains the key image frame in the image changes, the key image frame will also change accordingly.

[0054] It is easy to understand that the feature point of the key image frame refers to the feature point on each key image frame, and the feature point can be used to represent the object included in each key image frame. The object is an object existing in the target area. Among them, the number of feature points of each key image frame is at least one. The feature point does not specifically refer to a fixed feature point. When the key image frame changes, the feature point will also change accordingly. When the image acquired by the terminal changes, the feature point will also change accordingly. When the target area changes, the feature point will also change accordingly.

[0055] According to some embodiments, when the terminal acquires an image of a target area captured by a binocular camera, the terminal may acquire a key image frame in the image. For example, the terminal may use an image recognition algorithm to identify the image of the target area captured by the binocular camera to obtain a key image frame. The number of the key image frames is at least one. For example, the image of the target area Q captured by the binocular camera acquired by the terminal may be, for example, an A image frame, a B image frame, a C image frame, a D image frame, an E image frame, and an F image frame. The key image frames in the image obtained by the terminal using the image recognition algorithm may be, for example, an A image frame, a B image frame, a C image frame, and a D image frame.

[0056] It is easy to understand that when the terminal obtains the key image frames in the image, the terminal can obtain the feature points of each key image frame. For example, the key image frames in the image obtained by the terminal may be, for example, image frames A, B, C, and D. The feature points of image frame A obtained by the terminal may be, for example, feature points a1, a2, and a3; the feature points of image frame B may be, for example, feature points a1, a4, and a5; the feature points of image frame C may be, for example, feature points a3, a2, and a6; and the feature points of image frame D may be, for example, feature points a6, a4, and a5.

[0057] S103, generating a visual map corresponding to the target area based on the constraint information of the key image frame and the parameter expression information of the feature points.

[0058] According to some embodiments, the constraint information refers to the constraint information of the key image frame. The constraint information does not specifically refer to a fixed constraint information, and the constraint information can be determined based on the number information of the key image frame, and the constraint information can also be obtained based on the terminal based on a pre-set constraint algorithm. When the key image frame changes, the constraint information will also change accordingly. The parameter expression information is used to represent the parameterized expression form of the feature point.

[0059] Optionally, the visual map refers to a map of the target area obtained by the terminal by restoring the scene structure from the image. The visual map does not specifically refer to a fixed visual map. When the target area changes, the visual map will also change accordingly.

[0060] According to some embodiments, when the terminal obtains the key image frames in the image and obtains the feature points of each key image frame, the terminal can obtain the constraint information of the key image frames and the parameter expression information of the feature points. When the terminal obtains the constraint information of the key image frames and the parameter expression information of the feature points, the terminal can generate a visual map corresponding to the target area.

[0061] It is easy to understand that, for example, the key image frames in the image of the Q target area obtained by the terminal may be, for example, the A image frame, the B image frame, the C image frame, and the D image frame. The feature points of the A image frame obtained by the terminal may be, for example, the a1 feature point, the a2 feature point, and the a3 feature point; the feature points of the B image frame may be, for example, the a1 feature point, the a4 feature point, and the a5 feature point; the feature points of the C image frame may be, for example, the a3 feature point, the a2 feature point, and the a6 feature point; and the feature points of the D image frame may be, for example, the a6 feature point, the a4 feature point, and the a5 feature point. The terminal may obtain the constraint information of the A image frame, the B image frame, the C image frame, and the D image frame and the parameter expression information of the feature points on each key image frame. Based on the constraint information of the A image frame, the B image frame, the C image frame, and the D image frame and the parameter expression information of the feature points on each key image frame, the terminal may generate a visual map of the Q target area.

[0062] In one or more embodiments of the present application, by acquiring an image of the target area captured by a binocular camera, acquiring key image frames in the image and acquiring feature points of each key image frame, and based on the constraint information of the key image frames and the parameter expression information of the feature points, a visual map corresponding to the target area is generated. Therefore, when the texture differentiation of the target area is low, the terminal can generate a visual map of the target area based on the constraint information of the key image frames and the parameter expression information of the feature points, thereby reducing the situation where the accuracy of the visual map of the target area generated based only on visual information is low, and the accuracy of the visual map generation can be improved.

[0063] See also Figure 5 , which is a flow chart of a method for generating a visual map according to an embodiment of the present application. Figure 5 As shown, the method includes the following steps S201-S209.

[0064] S201, calibrating and synchronizing the time information of the binocular camera;

[0065] According to some embodiments, the time information refers to the time of the binocular camera, and the time information can be used to indicate the acquisition time of the image of the target area. Before the terminal acquires the image of the target area acquired by the binocular camera, the terminal can calibrate and synchronize the time information of the binocular camera so that the time information of the binocular camera remains consistent, which can improve the accuracy of acquiring the image of the target area, and further improve the accuracy of generating the visual map.

[0066] It is easy to understand that when the terminal calibrates and synchronizes the time information of the binocular camera, the terminal can obtain the current absolute time information, which can be, for example, a UNIX (Unix epoch) timestamp, that is, the relative time of the current moment to the UNIX time base 00:00:00 on July 1, 1970, accurate to seconds. The time information of the first camera in the binocular camera obtained by the terminal can be, for example, 12:00:00 on May 15, 2020, and the time information of the second camera can be, for example, 12:00:05 on May 15, 2020. When the current absolute time obtained by the terminal is 12:00:02 on May 15, 2020, the time information of the first camera and the time information of the second camera can be calibrated, and the time information of the first camera and the time information of the second camera can be synchronized to 12:00:02 on May 15, 2020.

[0067] Optionally, when the terminal calibrates and synchronizes the time information of the binocular camera, the terminal may calibrate the time information of the second camera based on the time information of the first camera in the binocular camera, and synchronize the time information of the second camera with the time information of the first camera. For example, the time information of the first camera in the binocular camera acquired by the terminal may be, for example, 12:00:00 on May 15, 2020, and the time information of the second camera may be, for example, 12:00:05 on May 15, 2020. The terminal may synchronize the time information of the second camera, 12:00:05 on May 15, 2020, with the time information of the first camera, 12:00:00 on May 15, 2020.

[0068] S202, acquiring an image of the target area captured by a binocular camera;

[0069] S203, obtaining the sampling interval duration;

[0070] According to some embodiments, the sampling interval duration refers to the interval duration for the terminal to obtain a key image frame. The sampling interval duration does not specifically refer to a fixed duration. The sampling interval duration can be modified based on the user's duration modification instruction, for example. The duration modification instruction includes but is not limited to a voice duration modification instruction, a click duration modification instruction, a timing duration modification instruction, a regional duration modification instruction, and the like. The regional duration modification instruction means that different regions correspond to different sampling interval durations.

[0071] It is easy to understand that when the terminal obtains the image of the target area captured by the binocular camera, the terminal can obtain the sampling interval duration. The sampling interval duration obtained by the terminal can be, for example, 1 second.

[0072] S204, acquiring a key image frame in the image based on the sampling interval duration;

[0073] According to some embodiments, when the terminal obtains the image of the target area captured by the binocular camera and the sampling interval duration, the terminal can obtain the key image frame in the image based on the sampling interval duration. The terminal obtains the key image frame in the image based on the sampling interval duration, which can reduce the image occupancy rate in the terminal storage space and save storage space.

[0074] It is easy to understand that the sampling interval duration obtained by the terminal may be, for example, 1 second. When the terminal obtains an image of the target area captured by the binocular camera, the terminal may obtain key image frames in the image based on the sampling interval duration of 1 second. The key image frames obtained by the terminal may be, for example, image frames A, B, C, and D.

[0075] S205, obtaining key image frames in the image, and obtaining feature points of each key image frame;

[0076] S206, obtaining constraint information of the key image frame;

[0077] According to some embodiments, the constraint information is used to constrain the key image frame. The constraint information includes first constraint information and second constraint information, the first constraint information is relative posture constraint information of two adjacent key image frames, and the second constraint information is roll-pitch constraint information of a key image frame.

[0078] It is easy to understand that after the terminal obtains the key image frame in the image, the terminal can obtain the constraint information of the key image frame. For example, the terminal can obtain the relative posture constraint information of two adjacent key image frames and the roll-pitch constraint information of a key image frame. The constraint information of the key image frame obtained by the terminal can be, for example, formula (1).

[0079]

[0080] Among them, E fs Represents constraint information of key image frames;

[0081] r ij Represents the relative posture constraint between the posture at time i and time j;

[0082] r i A roll-pitch constraint representing the keyframe pose;

[0083] Handwritten Indicates a combination of two keyframes that are related in observations;

[0084] Represents a collection of single keyframes;

[0085] r ij Represents the relative pose constraint r between relative keyframes rel ;

[0086] r i Represents the roll-pitch constraint r of a single keyframe rp ;

[0087] H ij and H i The covariance matrix representing these constraints. .

[0088] S207, obtaining visual factors of feature points;

[0089] According to some embodiments, when the terminal obtains the feature point of each key image frame, the terminal may obtain a visual factor of the feature point, which is used to obtain parameter expression information of the feature point.

[0090] According to some embodiments, Figure 6 A schematic diagram showing a flow chart of a visual map generation method according to an embodiment of the present application is shown. Figure 6 As shown, the method of the embodiment of the present application for obtaining the visual factor of the feature point may include the following steps S301-S304. S301, obtaining the coordinate information of the feature point on the current key image frame; S302, obtaining the depth information of the feature point relative to the initial key image frame; S303, obtaining the first posture information of the initial key image frame and the second posture information of the current key image frame; S304, generating the visual factor of the feature point based on the coordinate information, the depth information, the first posture information and the second posture information.

[0091] According to some embodiments, when the terminal obtains the visual factor of the feature point, the terminal may obtain the coordinate information of the feature point on the current key frame. The terminal may also obtain the depth information of the feature point relative to the initial key image frame. The terminal may also obtain the first posture information of the initial key image frame and the second posture information of the current key image frame. The coordinate information is the plane coordinate information of the feature point on the initial key image frame, and the initial key image frame is the image frame in which the feature point is observed for the first time.

[0092] Optionally, the first posture information is used to represent the posture of the initial key image frame, and the first posture information does not specifically refer to a fixed posture information. When the initial key image frame changes or the posture of the initial key frame changes, the first posture information will change accordingly. The second posture information does not specifically refer to a fixed posture information. When the current key image frame changes or the posture of the current key frame changes, the second posture information will change accordingly.

[0093] It is easy to understand that the order in which the terminal obtains the coordinate information of the feature points on the current key image frame, obtains the depth information of the feature points relative to the initial key image frame, and obtains the first posture information of the initial key image frame and the second posture information of the current key image frame is not limited. For example, the terminal can first obtain the coordinate information of the feature points on the current key image frame, then obtain the depth information of the feature points relative to the initial key image frame, and finally obtain the first posture information of the initial key image frame and the second posture information of the current key image frame. The terminal can also first obtain the depth information of the feature points relative to the initial key image frame, then obtain the coordinate information of the feature points on the current key image frame, and finally obtain the first posture information of the initial key image frame and the second posture information of the current key image frame.

[0094] According to some embodiments, when the terminal obtains the coordinate information of the feature point on the current key image frame, the depth information of the feature point relative to the initial key image frame, the first posture information of the initial key image frame, and the second posture information of the current key image frame, the terminal can generate a visual factor of the feature point based on the coordinate information, the depth information, the first posture information, and the second posture information. The visual factor generated by the terminal can be as shown in formula (2).

[0095]

[0096] Wherein, (u, v) represents the image plane coordinate information observed by the feature point on the initial key image frame;

[0097] d represents the depth information of the feature point relative to the initial key image frame;

[0098] T h is the first posture information representing the initial key image frame;

[0099] T t Represents the second posture information of the current key image frame;

[0100] z it represents the observation calculated from the second pose information of the current key image frame.

[0101] It is easy to understand that, for example, the key image frames in the image acquired by the terminal may be, for example, the A image frame, the B image frame, the C image frame, and the D image frame. The feature points of the A image frame acquired by the terminal may be, for example, the a1 feature point, the a2 feature point, and the a3 feature point; the feature points of the B image frame may be, for example, the a1 feature point, the a4 feature point, and the a5 feature point; the feature points of the C image frame may be, for example, the a3 feature point, the a2 feature point, and the a6 feature point; and the feature points of the D image frame may be, for example, the a6 feature point, the a4 feature point, and the a5 feature point. When the terminal acquires the visual factor of the a1 feature point on the B image frame, the terminal may acquire the coordinate information of the a1 feature point on the A image frame, the depth information of the a1 feature point relative to the A image frame, the first posture information of the A image frame, and the second posture information of the B image frame. The terminal may generate the visual factor of the a1 feature point based on the coordinate information of the a1 feature point on the A image frame, the depth information of the a1 feature point relative to the A image frame, the first posture information of the A image frame, and the second posture information of the B image frame.

[0102] According to some embodiments, Figure 7 A schematic diagram showing a flow chart of a visual map generation method according to an embodiment of the present application is shown. Figure 7 As shown, the method of the embodiment of the present application for obtaining the coordinate information of the feature point on the current key image frame may include the following steps S401-S402. S401, obtaining the universal polar spherical projection information of the feature point on the current key image frame; S402, calculating the coordinate information of the feature point based on the universal polar spherical projection information.

[0103] According to some embodiments, when the terminal obtains the coordinate information of the feature point on the current key image frame, the terminal can obtain the universal polar spherical projection information of the feature point on the current key frame. When the terminal obtains the universal polar spherical projection information, the terminal can calculate the coordinate information of the feature point based on the universal polar spherical projection information. Since the parameterized form of inverse depth does not consider the error of the bearing vector, a fixed bearing vector will be selected, resulting in inaccurate parameter expression information of the feature point. Therefore, the terminal calculates the coordinate information of the feature point based on the universal polar spherical projection information, which can improve the accuracy of obtaining the parameter expression information of the feature point and improve the accuracy of visual map generation.

[0104] It is easy to understand that the terminal can use formula (3) to calculate the coordinate information of the feature point.

[0105]

[0106] in, Corresponding to a point in the physical world, the lowercase one is the bearing vector.

[0107] S208, generating parameter expression information of the feature point based on the visual factor of the feature point;

[0108] According to some embodiments, parameter expression information is used to represent the parametric expression form of the feature point. The parameter expression information may be, for example, stereographic parameterized form information. When the terminal obtains the visual factor of the feature point, the terminal may generate the parameter expression information of the feature point based on the visual factor of the feature point. For example, when the terminal obtains the visual factor of the feature point as formula (2), the parameter expression information of the feature point generated by the terminal based on the visual factor of the feature point may be as shown in formula (4).

[0109]

[0110] S209, generating a visual map corresponding to the target area based on the constraint information of the key image frame and the parameter expression information of the feature points.

[0111] The specific process is as described above and will not be repeated here.

[0112] According to some embodiments, when the terminal obtains the constraint information of the key image frame and the parameter expression information of the feature point, the terminal can generate an objective function. When the terminal generates the objective function, the terminal can generate a visual map of the target area based on the objective function. The beam adjustment factor effect diagram of the terminal using the technical solution can be as follows: Figure 8 For example, the constraint information of the key image frame acquired by the terminal may be The parameter expression information of the feature point obtained by the terminal can be, for example, The objective function of the target area generated by the terminal may be, for example, as shown in formula (5).

[0113]

[0114] It is easy to understand that when the terminal generates the visual map of office 1, the image captured by the binocular camera obtained by the terminal can be, for example, Fig. 9 The visual map of the office 1 generated by the terminal without using the technical solution can be as follows Fig.10 As shown, the visual map of office 1 generated by the terminal using the technical solution can be as follows Fig.11 As shown. Fig.10As shown in FIG. 1 , the generated visual map of Office 1 is relatively scattered, and the accuracy of the generated visual map of Office 1 is low. Fig.11 As shown, the visual map of office 1 generated by the terminal using the present technical solution is more regular, and the generated visual map of office 1 is more accurate.

[0115] Optionally, when the terminal generates the visual map of office 2, the image captured by the binocular camera obtained by the terminal can be, for example, Fig.12 The visual map of Office 2 generated by the terminal without using the technical solution can be as follows Fig.13 As shown, the visual map of office 2 generated by the terminal using the technical solution can be as follows Fig.14 As shown. Fig.13 As shown in FIG. 1 , there are multiple scattered points in the generated visual map of Office 2, and the accuracy of the generated visual map of Office 2 is low. Fig.14 As shown, the visual map of Office 2 generated by the terminal using the present technical solution is relatively regular, and the generated visual map of Office 2 is highly accurate.

[0116] In one or more embodiments of the present application, the time information of the binocular camera is calibrated and synchronized so that the time information of the binocular camera remains consistent, which can improve the accuracy of the acquisition of the target area image, and thus improve the accuracy of the visual map generation. Secondly, by obtaining the sampling interval duration, the terminal can obtain the key image frame in the image based on the sampling interval duration, which can reduce the image occupancy rate in the terminal storage space and save storage space. In addition, by obtaining the constraint information of the key image frame and generating the parameter expression information of the feature point based on the visual factor of the feature point, a visual map of the target area can be generated, reducing the situation where the accuracy of the visual map of the target area generated based only on visual information is low, and the accuracy of the visual map generation can be improved.

[0117] The following will be combined with the attached Figure 15-19 , the visual map generation device provided in the embodiment of the present application is introduced in detail. It should be noted that the attached Figure 15-19 The visual map generation device shown is used to execute the present application Figure 4-Figure 14 For the convenience of explanation, only the part related to the embodiment of the present application is shown. For the specific technical details not disclosed, please refer to the present application. Figure 4-Figure 14 The embodiment shown.

[0118] See also Fig.15 , which shows a schematic diagram of the structure of a visual map generating device according to an embodiment of the present application. The visual map generating device 1500 can be implemented as all or part of a user terminal through software, hardware, or a combination of both.

[0119] According to some embodiments, the visual map generating device 1500 includes an image acquiring unit 1501, a feature point acquiring unit 1502 and a map generating unit 1503, which are specifically used to:

[0120] An image acquisition unit 1501 is used to acquire an image of a target area captured by a binocular camera;

[0121] The feature point acquisition unit 1502 is used to acquire key image frames in the image and acquire feature points of each key image frame;

[0122] The map generation unit 1503 is used to generate a visual map corresponding to the target area based on the constraint information of the key image frame and the parameter expression information of the feature point.

[0123] According to some embodiments, Fig.16 A schematic diagram of the structure of a visual map generation device according to an embodiment of the present application is shown. The visual map generation device 1500 also includes an information acquisition unit 1504, which is used to acquire the constraint information of the key image frame before generating the visual map corresponding to the target area based on the constraint information of the key image frame and the parameter expression information of the feature point; the constraint information includes first constraint information and second constraint information, the first constraint information is the relative posture constraint information of two adjacent key image frames, and the second constraint information is the roll-pitch constraint information of a key image frame.

[0124] According to some embodiments, Fig.17 A schematic diagram of the structure of a visual map generating device according to an embodiment of the present application is shown. The visual map generating device 1500 also includes an information generating unit 1505, which is used to obtain visual factors of feature points before generating a visual map corresponding to a target area based on constraint information of a key image frame and parameter expression information of feature points;

[0125] Based on the visual factors of the feature points, the parameter expression information of the feature points is generated.

[0126] According to some embodiments, Fig.18 The schematic diagram of the structure of a visual map generating device according to an embodiment of the present application is shown. The information generating unit 1505 includes a coordinate information acquiring subunit 1515, a depth information acquiring subunit 1525, a posture information acquiring subunit 1535 and a visual factor generating subunit 1545, which are used to acquire the visual factor of the feature point, specifically for:

[0127] The coordinate information acquisition subunit 1515 is used to acquire the coordinate information of the feature point on the current key image frame; the coordinate information is the plane coordinate information of the feature point on the initial key image frame, and the initial key image frame is the image frame where the feature point is observed for the first time;

[0128] The depth information acquisition subunit 1525 is used to acquire the depth information of the feature point relative to the initial key image frame;

[0129] The posture information acquisition subunit 1535 is used to acquire the first posture information of the initial key image frame and the second posture information of the current key image frame;

[0130] The visual factor generating subunit 1545 is used to generate a visual factor of the feature point based on the coordinate information, the depth information, the first posture information and the second posture information.

[0131] According to some embodiments, the coordinate information acquisition subunit 1515 is used to acquire the coordinate information of the feature point on the current key image frame:

[0132] Get the universal polar spherical projection information of the feature points on the current key image frame;

[0133] Based on the general polar spherical projection information, the coordinate information of the feature points is calculated.

[0134] According to some embodiments, the feature point acquisition unit 1502, when used to acquire a key image frame in an image, is specifically used to:

[0135] Get the sampling interval duration;

[0136] Based on the sampling interval duration, key image frames in the image are obtained.

[0137] According to some embodiments, a schematic diagram of the structure of a visual map generating device according to an embodiment of the present application is shown in FIG19. The visual map generating device 1500 further includes an information calibration unit 1506 for calibrating and synchronizing the time information of the binocular camera before acquiring the image of the target area captured by the binocular camera.

[0138] In one or more embodiments of the present application, by acquiring an image of the target area captured by a binocular camera, acquiring key image frames in the image and acquiring feature points of each key image frame, and based on the constraint information of the key image frames and the parameter expression information of the feature points, a visual map corresponding to the target area is generated. Therefore, when the texture differentiation of the target area is low, the visual map generation device can generate a visual map of the target area based on the constraint information of the key image frames and the parameter expression information of the feature points, thereby reducing the situation where the accuracy of the visual map of the target area generated based only on visual information is low, and the accuracy of the visual map generation can be improved.

[0139] See also Fig. 20 , is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. Fig. 20As shown, the terminal 2000 may include: at least one processor 2001 , at least one network interface 2004 , a user interface 2003 , a memory 2005 , and at least one communication bus 2002 .

[0140] The communication bus 2002 is used to realize the connection and communication between these components.

[0141] The user interface 2003 may include a binocular camera, and the optional user interface 2003 may also include a standard wired interface or a wireless interface.

[0142] The network interface 2004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0143] Among them, the processor 2001 may include one or more processing cores. The processor 2001 uses various interfaces and lines to connect various parts in the entire terminal 2000, and executes various functions and processes data of the terminal 2000 by running or executing instructions, programs, code sets or instruction sets stored in the memory 2005, and calling data stored in the memory 2005. Optionally, the processor 2001 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 2001 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 2001, and it can be implemented separately through a chip.

[0144] Among them, the memory 2005 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 2005 includes a non-transitory computer-readable storage medium. The memory 2005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 2005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 2005 may also be optionally at least one storage device located away from the aforementioned processor 2001. As Fig. 20 As shown, the memory 2005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application for generating a visual map.

[0145] exist Fig. 20 In the terminal 2000 shown, the user interface 2003 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 2001 can be used to call the application for generating the visual map stored in the memory 2005, and specifically perform the following operations:

[0146] Acquire the image of the target area captured by the binocular camera;

[0147] Obtain key image frames in the image, and obtain feature points of each key image frame;

[0148] Based on the constraint information of the key image frames and the parameter expression information of the feature points, a visual map corresponding to the target area is generated.

[0149] According to some embodiments, before the processor 2001 generates a visual map corresponding to the target area based on the constraint information of the key image frame and the parameter expression information of the feature points, it further specifically performs the following operations:

[0150] Obtain constraint information of the key image frame; the constraint information includes first constraint information and second constraint information, the first constraint information is relative posture constraint information of two adjacent key image frames, and the second constraint information is roll-pitch constraint information of a key image frame.

[0151] According to some embodiments, before the processor 2001 generates a visual map corresponding to the target area based on the constraint information of the key image frame and the parameter expression information of the feature points, it further specifically performs the following operations:

[0152] Get the visual factor of the feature point;

[0153] Based on the visual factors of the feature points, the parameter expression information of the feature points is generated.

[0154] According to some embodiments, when the processor 2001 executes the process of acquiring the visual factor of the feature point, the following operations are specifically performed:

[0155] Obtaining coordinate information of feature points on the current key image frame; the coordinate information is the plane coordinate information of the feature points on the initial key image frame, and the initial key image frame is the image frame where the feature points are observed for the first time;

[0156] Obtaining the depth information of the feature points relative to the initial key image frame;

[0157] Acquire first posture information of an initial key image frame and second posture information of a current key image frame;

[0158] A visual factor of the feature point is generated based on the coordinate information, the depth information, the first posture information, and the second posture information.

[0159] According to some embodiments, when the processor 2001 executes the acquisition of coordinate information of feature points on the current key image frame, the following operations are specifically performed:

[0160] Get the universal polar spherical projection information of the feature points on the current key image frame;

[0161] Based on the general polar spherical projection information, the coordinate information of the feature points is calculated.

[0162] According to some embodiments, when the processor 2001 executes the process of acquiring a key image frame in an image, the following operations are specifically performed:

[0163] Get the sampling interval duration;

[0164] Based on the sampling interval duration, key image frames in the image are obtained.

[0165] According to some embodiments, before the processor 2001 executes the process of acquiring the image of the target area captured by the binocular camera, the processor 2001 further specifically performs the following operations:

[0166] Calibrate and synchronize the time information of the binocular camera.

[0167] In one or more embodiments of the present application, by acquiring an image of the target area captured by a binocular camera, acquiring key image frames in the image and acquiring feature points of each key image frame, and based on the constraint information of the key image frames and the parameter expression information of the feature points, a visual map corresponding to the target area is generated. Therefore, when the texture differentiation of the target area is low, the terminal can generate a visual map of the target area based on the constraint information of the key image frames and the parameter expression information of the feature points, thereby reducing the situation where the accuracy of the visual map of the target area generated based only on visual information is low, and the accuracy of the visual map generation can be improved.

[0168] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the above method are implemented. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0169] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps of any one of the visual map generation methods recorded in the above method embodiments.

[0170] Those skilled in the art can clearly understand that the technical solution of the present application can be implemented with the help of software and / or hardware. The "unit" and "module" in this specification refer to software and / or hardware that can independently complete or cooperate with other components to complete specific functions, where the hardware can be, for example, a field programmable gate array (FPGA), an integrated circuit (IC), etc.

[0171] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0172] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0173] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0174] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0175] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0176] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program code.

[0177] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by entering a program to instruct the relevant hardware. The program may be stored in a computer-readable memory, and the memory may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0178] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method for generating a visual map, characterized in that: Applied to a terminal with a binocular camera, the method comprises: Acquire an image of a target area captured by the binocular camera, wherein the target area is changed based on an instruction or setting; Acquire key image frames in the image, and acquire feature points of each key image frame, wherein the feature points are used to represent objects included in each key image frame, and the objects are objects existing in the target area; Based on the constraint information of the key image frame and the parameter expression information of the feature point, a visual map corresponding to the target area is generated, wherein the constraint information is used to constrain the key image frame, and the constraint information includes first constraint information and second constraint information, wherein the first constraint information is relative posture constraint information of two adjacent key image frames, and the second constraint information is roll-pitch constraint information of a key image frame, and the parameter expression information is used to represent the parameterized expression form of the feature point; The formula of the constraint information is: ; Among them, the represents constraint information of a key image frame, wherein fs represents a key image frame; Said Represents the relative posture constraint between the posture at time i and time j; Said Represents the roll-pitch constraint of a single keyframe pose; Said Indicates a combination of two keyframes that are related in observation; Said Represents a collection of single keyframes; Said represents the covariance matrix of the relative posture constraint, the represents the covariance matrix of the roll-pitch constraint.

2. The method according to claim 1, characterized in that Before generating the visual map corresponding to the target area based on the constraint information of the key image frame and the parameter expression information of the feature point, the method further includes: Obtain constraint information of the key image frame.

3. The method according to claim 1, characterized in that Before generating the visual map corresponding to the target area based on the constraint information of the key image frame and the parameter expression information of the feature point, the method further includes: Obtaining a visual factor of the feature point, where the visual factor is a visual observation error of the binocular camera; Based on the visual factors of the feature points, parameter expression information of the feature points is generated.

4. The method according to claim 3, characterized in that The obtaining of the visual factor of the feature point comprises: Obtain the coordinate information of the feature point on the current key image frame; the coordinate information is the plane coordinate information of the feature point on the initial key image frame, and the coordinate information is used Indicates that the initial key image frame is an image frame in which the feature point is observed for the first time; Acquire depth information of the feature point relative to the initial key image frame, where the depth information is represented by d; Acquire the first posture information of the initial key image frame and the second posture information of the current key image frame, wherein the first posture information is used Indicates that the second posture information is Indicates that h is used to represent the initial key image frame, and t is used to represent the current key image frame; Generate a visual factor of the feature point based on the coordinate information, the depth information, the first posture information and the second posture information , t in it is used to represent the current key image frame, and i is used to represent the i-th feature point in the current key image frame; Among them, the is used to represent the position of the feature point in the three-dimensional space, represents an observation calculated from the second pose information of the current key image frame.

5. The method according to claim 4, characterized in that The step of obtaining the coordinate information of the feature points on the current key image frame includes: Obtaining universal polar spherical projection information of feature points on the current key image frame; Based on the universal polar spherical projection information, coordinate information of the feature point is calculated.

6. The method according to claim 1, characterized in that The step of acquiring a key image frame in the image comprises: Get the sampling interval duration; Based on the sampling interval duration, a key image frame in the image is acquired.

7. The method according to claim 1, characterized in that Before acquiring the image of the target area captured by the binocular camera, the method further includes: The time information of the binocular camera is calibrated and synchronized.

8. A visual map generating device, characterized in that: The device comprises: An image acquisition unit, used to acquire an image of a target area captured by a binocular camera, wherein the target area is changed based on an instruction or a setting; A feature point acquisition unit, used to acquire key image frames in the image, and acquire feature points of each key image frame, wherein the feature points are used to represent objects included in each key image frame, and the objects are objects existing in the target area; a map generation unit, configured to generate a visual map corresponding to the target area based on constraint information of the key image frame and parameter expression information of the feature point, wherein the constraint information is used to constrain the key image frame, and the constraint information includes first constraint information and second constraint information, wherein the first constraint information is relative posture constraint information of two adjacent key image frames, and the second constraint information is roll-pitch constraint information of a key image frame, and the parameter expression information is used to represent a parametric expression form of the feature point; The formula of the constraint information is: ; Among them, the represents constraint information of a key image frame, wherein fs represents a key image frame; Said Represents the relative posture constraint between the posture at time i and time j; Said Represents the roll-pitch constraint of a single keyframe pose; Said Indicates a combination of two keyframes that are related in observation; Said Represents a collection of single keyframes; Said represents the covariance matrix of the relative posture constraint, the represents the covariance matrix of the roll-pitch constraint.

9. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When a processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images