Pose detection method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a pose detection method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining image data which comprises at least two image frames collected by terminal equipment; the image data is detected, a target image frame in the image data is determined, the number of first feature points corresponding to the target image frame is not larger than a preset number, and the first feature points are at least one pixel point set matched based on image features of a reference image in a map database in the image frame; and according to the first feature point in the target image frame and the second feature point in at least one other image frame in the image data, determining a first target pose when the terminal device collects the target image frame. The problems that the pose detection success rate is low and the accuracy is poor due to the fact that an individual image frame collected by the terminal equipment cannot provide enough image information are solved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of visual positioning technology, and in particular to a posture detection method, device, electronic device and storage medium. Background Art

[0002] Visual positioning technology refers to the use of image acquisition devices installed on terminal devices, such as cameras, to collect and process surrounding image information, thereby achieving accurate estimation of the terminal device's position and posture. It is often used in application scenarios such as indoor navigation and high-precision vehicle navigation.

[0003] The posture detection scheme of the terminal device in the prior art is usually implemented by comparing the feature points in the collected environment image with the feature points in the pre-collected image in the map database. However, due to the influence of the complex environment in which the terminal device is actually located, the scheme in the prior art has the problems of low posture detection success rate and poor accuracy. Summary of the invention

[0004] The embodiments of the present disclosure provide a posture detection method, device, electronic device and storage medium to overcome the problems of low posture detection success rate and poor accuracy.

[0005] In a first aspect, an embodiment of the present disclosure provides a posture detection method, including:

[0006] Acquire image data, wherein the image data includes at least two image frames captured by a terminal device; detect the image data to determine a target image frame in the image data, wherein the number of first feature points corresponding to the target image frame is not greater than a preset number, and the first feature point is at least one set of pixel points in the image frame that is matched to image features of a reference image in a map database; determine a first target posture of the terminal device when the target image frame is captured based on the first feature point in the target image frame and a second feature point in at least one other image frame in the image data.

[0007] In a second aspect, an embodiment of the present disclosure provides a posture detection device, including:

[0008] An acquisition module, used to acquire image data, wherein the image data includes at least two image frames acquired by a terminal device;

[0009] a processing module, configured to detect the image data, determine a target image frame in the image data, wherein the number of first feature points corresponding to the target image frame is not greater than a preset number, and the first feature points are at least one set of pixel points in the image frame matched to image features of a reference image in a map database;

[0010] A prediction module is used to determine a first target posture of the terminal device when capturing the target image frame based on a first feature point in the target image frame and a second feature point in at least one other image frame in the image data.

[0011] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0012] A processor, and a memory communicatively connected to the processor;

[0013] The memory stores computer-executable instructions;

[0014] The processor executes the computer-executable instructions stored in the memory to implement the posture detection method described in the first aspect and various possible designs of the first aspect.

[0015] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the posture detection method described in the first aspect and various possible designs of the first aspect is implemented.

[0016] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the posture detection method described in the first aspect and various possible designs of the first aspect.

[0017] The posture detection method, device, electronic device and storage medium provided in this embodiment obtain image data, wherein the image data includes at least two frames of image frames collected by a terminal device; detect the image data to determine a target image frame in the image data, wherein the number of first feature points corresponding to the target image frame is not greater than a preset number, and the first feature point is at least one set of pixel points in the image frame matched to the image feature of a reference image in a map database; and determine the first target posture of the terminal device when collecting the target image frame based on the first feature point in the target image frame and the second feature point in at least one other image frame in the image data. By determining a target image frame with too few feature points in the image data collected by the terminal device, and combining the feature points in the target image frame and other image frames to jointly determine the posture of the terminal device when collecting the target image frame, the problem of low posture detection success rate and poor accuracy caused by the inability of individual image frames collected by the terminal device to provide sufficient image information is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 An application scenario diagram of the posture detection method provided in the embodiment of the present disclosure;

[0020] Figure 2 Schematic diagram of the process of the posture detection method provided in the embodiment of the present disclosure Figure 1 ;

[0021] Figure 3 for Figure 2 A flowchart of a specific implementation method of step S101 in the illustrated embodiment;

[0022] Figure 4 A schematic diagram of a process for generating image data provided by an embodiment of the present disclosure;

[0023] Figure 5 for Figure 2 A flowchart of a specific implementation method of step S102 in the illustrated embodiment;

[0024] Figure 6 A schematic diagram of a target feature point provided by an embodiment of the present disclosure;

[0025] Figure 7 Schematic diagram of the process of the posture detection method provided in the embodiment of the present disclosure Figure 2 ;

[0026] Figure 8 for Figure 7 A flowchart of a specific implementation method of step S205 in the illustrated embodiment;

[0027] Fig. 9 A schematic diagram of obtaining fused feature points provided by an embodiment of the present disclosure;

[0028] Fig.10 A structural block diagram of a posture detection device provided in an embodiment of the present disclosure;

[0029] Fig.11 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure;

[0030] Fig.12 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0033] The application scenarios of the embodiments of the present disclosure are explained below:

[0034] Figure 1 This is an application scenario diagram of the posture detection method provided by the embodiment of the present disclosure. The posture detection method provided by the embodiment of the present disclosure can be applied to scenarios such as walking navigation and vehicle precision navigation. Specifically, the method provided by the embodiment of the present disclosure can be applied to terminal devices, such as handheld devices such as smart phones, or wearable devices such as smart glasses and helmets. Figure 1 As shown, the terminal device takes smart glasses as an example, and the user installs and runs the navigation software on the smart glasses 11 to achieve accurate navigation indoors or outdoors. Specifically, a rigidly connected image acquisition unit, such as a camera, is provided in the smart glasses 11. After the navigation software is run, the smart glasses 11 collect environmental images in real time through the image acquisition unit, and use the method provided in this embodiment, combined with the pre-generated map database, to process the environmental images, and use the content in the environmental images to determine the real-time position and orientation of the smart glasses 11, that is, the posture of the terminal device, and display the posture on the display screen 12 of the terminal device, so as to achieve the positioning of the target object in the terminal device / environment.

[0035] In the prior art, for the above-mentioned application scenarios such as pedestrian navigation and vehicle precise navigation, the scheme for realizing the posture detection of the terminal device is usually based on the feature points in the collected environmental image, which are compared with the feature points in the pre-collected image in the map database, and then mapped to the three-dimensional world coordinates to realize the posture detection of the terminal device. In this process, it is usually required that the collected image frame include multiple feature points (usually more than 3) to improve the accuracy of the detection result. However, in the actual application process, due to the complex and changeable environment in which the terminal device is located, when the terminal device moves or shakes significantly, or faces the wall, for example, resulting in a low amount of information in the collected environmental image, the feature points in the environmental image will be less than 3, or even unable to match the feature points, resulting in the posture detection for the frame image being invalid or inaccurate. This will lead to problems such as freezing and jumping of navigation coordinates in the user interface of the terminal device, affecting the user experience and causing safety hazards.

[0036] The disclosed embodiment provides a posture detection method to solve the above problem. The method detects the image data collected by the terminal, determines the target image frame with too few feature points, and then combines the feature points in the target image frame and other image frames in the image data to jointly determine the posture of the terminal device when collecting the target image frame, thereby solving the problem of low posture detection success rate and poor accuracy caused by the inability of individual image frames to provide sufficient image information.

[0037] refer to Figure 2 , Figure 2 Schematic diagram of the process of the posture detection method provided in the embodiment of the present disclosure Figure 1 The method of this embodiment can be applied in a terminal device or a server. When the method provided by this embodiment is executed by a server, it is necessary to obtain corresponding data by communicating with the terminal device, and after obtaining the detection result, the detection result can be sent to the terminal device for display. In the following embodiments, the terminal device is used as the execution subject of the method provided by this embodiment. Exemplarily, the posture detection method includes:

[0038] Step S101: Acquire image data, where the image data includes at least two image frames captured by a terminal device.

[0039] For example, see Figure 1In the schematic diagram of the application scenario shown in the figure, the terminal device collects images of the external environment through the set image acquisition unit, and can obtain corresponding image frames, wherein the set of at least two image frames collected is image data. In a possible implementation, the terminal device continuously collects continuous images of the external environment at equal sampling intervals within a period of time, and uses the set of all environmental images (image frames) collected within the time period as image data for subsequent processing. Among them, further, in a possible implementation, the above-mentioned acquisition process and the subsequent processing process can be performed synchronously, that is, after the above-mentioned acquisition process is completed and the determined image data is obtained, the subsequent processing steps for the image are performed; for example, the terminal device collects 10 frames of environmental images at equal time intervals in the time period from the 0th second to the 5th second: environmental image P0 to environmental image P9. Starting from the 5th second, environmental images P0 to environmental images P9 are used as image data for subsequent processing.

[0040] In another possible implementation, the acquisition process and subsequent processing process may be performed asynchronously, that is, after the terminal device obtains at least two frames of image data constituting the image data, the subsequent processing steps for the image data can be performed, and at the same time, the terminal device continues to acquire images, and continues to acquire environmental images, and uses the set of environmental images (image frames) acquired in real time as image data to perform subsequent processing steps. For example, the terminal device acquires environmental images starting from the 0th second, and continuously acquires environmental images at equal time intervals. For example, the terminal device acquires environmental image P0 at the 0.5th second, acquires environmental image P1 at the 1st second, and so on. When the terminal device collects the environmental image P1 at the 1st second, the environmental image P0 and the environmental image P1 are used as image data for subsequent processing, and at the same time, new environmental images are continuously collected; when the terminal device collects the environmental image P_N at the Nth second, the environmental image P_N and the m environmental images before the environmental image P_N are used together as image data for subsequent processing, so that the image data is always the latest collected m+1 frame environmental image, and then based on the latest collected m+1 frame environmental image, continuous posture detection for the terminal is realized. Among them, this implementation method requires at least two processing threads to cooperate and realize, and the specific implementation method of multi-threaded data asynchronous processing is a prior art known to those skilled in the art, which will not be repeated here.

[0041] In another possible implementation, after the terminal device collects the environmental image, it screens the environmental image, takes the image frames that meet the requirements, and takes a set of at least two image frames as image data to obtain the image data. Exemplarily, the environmental image that meets the requirements includes at least one of the following: the collection interval between each environmental image is not less than the first duration; the relative displacement between the collection positions corresponding to each environmental image is not less than the first length. Specifically, for example, every 0.5 second interval, a frame of the collected environmental image is extracted as an image frame constituting the image data; for another example, according to the collection position when the environmental image is collected, when the relative displacement between the collection positions corresponding to two adjacent frames is greater than 1 meter, the corresponding environmental image is used as an image frame constituting the image data, wherein the relative displacement between the collection positions when the environmental image is collected can be determined by other reference information collected by the terminal device, such as inertial sensor data collected by an inertial sensor (Inertial Measurement Unit, IMU).

[0042] like Figure 3 As shown, illustratively, the specific implementation of step S101 includes:

[0043] Step S1011: at least two frames of environment images are collected through the terminal device, and motion information corresponding to each environment image is obtained, where the motion information represents the motion state of the terminal device when shooting the environment image.

[0044] Step S1012: Determine the relative displacement corresponding to the environment image according to the motion information.

[0045] Step S1013: based on the relative displacement corresponding to each environmental image, extract at least two frames of environmental images to generate image data.

[0046] Based on the above introduction, exemplarily, in the process of continuously performing posture detection, the terminal device collects environmental images based on a specific image sampling frequency through its own image acquisition unit, such as a camera, for example, collecting environmental images once every 0.5 seconds, and setting an acquisition timestamp for each environmental image; at the same time, the terminal device collects the motion information of the terminal device at a motion information sampling frequency not less than the above image sampling frequency, wherein the motion information represents the motion state of the terminal device when taking the environmental image, such as acceleration, speed, etc. In a possible implementation, the acquisition phases of the environmental image acquisition process and the motion information acquisition process are synchronized, that is, the terminal device simultaneously collects the environmental image and motion information at the same (logical) moment, so that the environmental image and motion information have the same acquisition timestamp, and then a timestamp-based mapping is established between the two. In another possible implementation, the acquisition phases of the environmental image acquisition process and the motion information acquisition process are asynchronous, that is, the terminal device acquires environmental images and motion information at different (logical) moments, and the acquisition timestamps of the environmental images and motion information are different. In this case, the acquisition timestamp of the motion information closest to the environmental image can be determined based on the acquisition timestamp of the environmental image, and then the two can establish a timestamp-based mapping. The specific implementation method can be set as needed and will not be repeated here. Among them, the motion information can be obtained through the inertial sensor data collected by the inertial sensor, which will not be repeated here.

[0047] Furthermore, motion information corresponding to each environmental image is obtained, and based on the motion information, the relative displacement corresponding to the environmental image is determined. Specifically, for example, the motion information is such as the acceleration or velocity vector when the terminal device captures the environmental image. The relative displacement when the terminal device captures each environmental image can be obtained by integrating the motion information. Thereafter, the relative displacement is accumulated. Whenever the accumulated relative displacement is greater than or equal to a preset first length, the corresponding environmental image is used as an image frame constituting the image data.

[0048] Figure 4 A schematic diagram of a process for generating image data provided by an embodiment of the present disclosure is shown below. Figure 4 The above process is described in detail. Figure 4As shown, with reference to the acquisition time axis, the terminal device acquires environmental images at time intervals of 0.1 seconds (shown as 0.1 in the figure, the same below), and acquires motion information at time intervals of 0.5 seconds. Starting from the 0th second, environmental image acquisition and motion information acquisition are performed simultaneously. After that, the terminal device extracts and acquires a frame of environmental image every 0.5 seconds, including at least environmental image P1, environmental image P2, environmental image P3, environmental image P4, and environmental image P5 shown in the figure (shown as P1, P2, P3, P4, and P5, respectively). And obtain the motion information of the above-mentioned extracted environmental images (respectively shown as S1, S2, S3, S4, and S5 in the figure), and then determine the relative displacements between the acquisition positions of environmental images P1, environmental image P2, environmental image P3, environmental image P4, and environmental image P5 when they are acquired based on the motion information of the above-mentioned extracted environmental images, as shown in the reference acquisition position axis, wherein environmental image P1 is the initial point and the relative displacement is 0; the relative displacement of environmental image P2 is the displacement between the position when environmental image P2 is acquired and the position when environmental image P1 is acquired, for example, Dis_1; similarly, the relative displacement of environmental image P3 is Dis_2, the relative displacement of environmental image P4 is Dis_3, and the relative displacement of environmental image P5 is Dis_4. The above relative displacements are accumulated and compared with the preset first length Dis_0. Since Dis_1 is greater than Dis_0, the environment image P2 corresponding to Dis_1 is set as the image frame constituting the image data; then, starting from the position where the environment image P2 is collected, the sum of Dis_2 and Dis_3 is greater than Dis_0, so the environment image P4 corresponding to Dis_3 is set as another image frame constituting the image data; starting from the position where the environment image P5 is collected, Dis_4 is greater than Dis_0, so the environment image P5 corresponding to Dis_4 is set as another image frame constituting the image data. And so on, until all the collected environment images are detected, the environment images detected by the above steps and meeting the requirements are used as the image frames in the image data; or, the environment images detected by the above steps and meeting the requirements are continuously used as the image frames in the image data.

[0049] In the steps of this embodiment, the initial environmental image collected is extracted by collecting intervals and / or relative displacements between collection positions to obtain image data that meets the requirements in time and space. On the one hand, the amount of image data to be processed subsequently is reduced, reducing the operating load and resource consumption. On the other hand, similar image frames that are close in time and / or space are removed, thereby improving the effective information density and information validity of the image data and improving the computational efficiency of posture detection.

[0050] Step S102: Detect the image data to determine the target image frame in the image data. The number of first feature points corresponding to the target image frame is not greater than a preset number. The first feature point is at least one set of pixel points in the image frame that is matched to the image feature of the reference image in the map database.

[0051] Exemplarily, after obtaining the image data, the image data is detected to determine the target image frame, wherein the number of first feature points corresponding to the target image frame is not greater than the preset number, and the first feature point is at least one set of pixels in the image frame matched to the image feature of the reference image in the map database. In short, the target image frame is an image frame that does not meet the requirement of the number of feature points. The first feature point corresponding to the target image frame is the feature point in the target image frame, and the feature point refers to the pixel point in the image frame with similar image features after comparing the image frame with the reference image pre-collected in the map database. In the visual positioning technology based on the prior map database, the map database contains a three-dimensional image model and a two-dimensional image model. By querying the map database, the two-dimensional image in the two-dimensional image model is first matched, and then the two-dimensional image in the map database is mapped to the three-dimensional image model, so as to obtain the position and posture of the image acquisition point. In the above process, by querying the map database, the step of matching the two-dimensional image in the two-dimensional image model, that is, the process of determining the feature points, and then counting the feature points, the number of feature points in the image frame can be obtained. Among them, the specific method for determining the feature points can be implemented by a random consensus sampling (RANSAC) algorithm, and its implementation process is a prior art known in the art and will not be repeated here.

[0052] Furthermore, by detecting the number of feature points corresponding to each image frame in the image data, an image frame with a number of feature points less than a preset number is determined as a target image frame, and the feature points in the target image frame are first feature points, that is, the number of first feature points is less than the preset number; correspondingly, an image frame with a number of feature points greater than or equal to the preset number is determined as a non-target image frame, and the feature points in the non-target image frame are second feature points, that is, the number of second feature points is greater than or equal to the preset number.

[0053] For example, Figure 5 As shown, the specific implementation of step S102 includes:

[0054] Step S1021: extracting features from image frames in the image data to obtain first image features.

[0055] Step S1022: performing feature comparison between the first image feature and the reference image in the map database to determine the target feature point in the image frame, wherein the feature similarity between the target feature point and the corresponding reference feature point in the reference image is greater than a similarity threshold.

[0056] Step S1023: Determine the target image frame according to the number of target feature points.

[0057] Exemplarily, first, after obtaining the image data, based on the acquisition timestamp of each image frame in the image data, feature extraction is performed in sequence to obtain the corresponding pixel feature matrix, i.e., the first image feature. Then, based on the first image feature, feature comparison is performed on the reference image in the map database to retrieve the reference image with similar image features to the image frame. Since the reference image is an image acquired in a real environment, when the image frame in the image data has features (pixel point set) similar to those in the reference image, it means that the object in the image frame can be mapped to the object in the real environment to a certain extent, that is, the pose of the acquired image frame is similar to the pose when the reference image is acquired (in order to capture similar content). The higher the feature similarity between the feature point in the image frame and the reference feature point in the reference frame, the more likely the feature point is to be a target feature point. At the same time, the more target feature points in the image frame, the higher the possibility that the image frame and the reference image are acquired in the same or similar pose. The method for determining the target feature point in the image frame can be implemented by a random consistency sampling algorithm, and its implementation process is a prior art known in the art and will not be repeated here.

[0058] Figure 6 A schematic diagram of a target feature point provided by an embodiment of the present disclosure, such as Figure 6 As shown, taking the image frame P1 in the image data as an example, after acquiring the image frame P1, feature extraction is performed on the image frame to obtain the first image feature of the image frame P1. The first image feature can be implemented in a variety of ways such as a matrix, a vector or a structure. Thereafter, the reference image in the map database is retrieved based on the first image feature, and the target feature points in the image frame P1 are determined based on the data in the map database. Specifically, the coordinates and identification of each feature point can be obtained, such as the feature point T1 and the feature point T2 shown in the figure. Since the image frame P1 only includes 2 feature points, the image frame P1 is determined as the target image frame.

[0059] Step S103: Determine a first target posture of the terminal device when capturing the target image frame based on a first feature point in the target image frame and a second feature point in at least one other image frame in the image data.

[0060] Exemplarily, after determining the target image frame in the image data, the small number of first feature points in the target image frame may lead to deviation in the pose detection result. Therefore, based on the target image frame, other image frames in the image data are further combined, and the feature points in other image frames, i.e., the second feature points, are fused with the first feature points in the target image frame. This is equivalent to fusing the information in the target image frame with the information in other image frames, thereby obtaining a fused image frame with richer information. Afterwards, pose detection is performed based on the fused image frame, which can obtain better detection results than directly using the target image frame for pose detection. Among them, in a possible implementation method, in the process of continuously performing posture detection on the terminal device, after the target image frame in the image data is detected, the target image frame is feature fused with at least one other image frame before the target image frame in the image data. Due to the specific time and / or space acquisition restrictions between the target image frame and other image frames in the image data, it can be ensured that the target image frame and other image frames have large differences in image content. The feature fusion of the target image frame and other image frames is equivalent to merging the image field of view corresponding to the target image frame and the image field of view of other image frames, and obtaining a fused image frame with a large field of view. The fused image frame must contain more feature points. Therefore, the posture detection based on the fused image frame, the first target posture when the target image frame is acquired, has better accuracy and information. Among them, the specific implementation method of posture judgment based on the image (in this embodiment, the fused image synthesized according to the target image frame and other image frames) is a prior art and will not be repeated here.

[0061] In this embodiment, by acquiring image data, the image data includes at least two frames of image frames collected by the terminal device; detecting the image data to determine the target image frame in the image data, the number of first feature points corresponding to the target image frame is not greater than a preset number, and the first feature point is at least one set of pixel points in the image frame matched to the image feature of the reference image in the map database; according to the first feature point in the target image frame and the second feature point in at least one other image frame in the image data, the first target posture of the terminal device when collecting the target image frame is determined. By determining the target image frame with too few feature points in the image data collected by the terminal device, and combining the feature points in the target image frame and other image frames to jointly determine the posture of the terminal device when collecting the target image frame, the problem of low posture detection success rate and poor accuracy caused by the failure of individual image frames collected by the terminal device to provide sufficient image information is solved.

[0062] refer to Figure 7 , Figure 7 Schematic diagram of the process of the posture detection method provided in the embodiment of the present disclosure Figure 2In this embodiment Figure 2 On the basis of the embodiment shown, step S103 is further refined, and a step of processing non-target image frames is added. The posture detection method includes:

[0063] Step S201: Acquire image data, where the image data includes at least two image frames captured by a terminal device.

[0064] Step S202: Detect the image frame in the image data. If the image frame is a non-target image frame, execute step S203; if the image frame is a target image frame, execute step S204.

[0065] Step S203: For the non-target image frame, a perspective n-point algorithm and a random consistency sampling algorithm are used to solve the non-target image frame to obtain a second target posture corresponding to the non-target image frame.

[0066] For example, reference Figure 2 As described in the related description of the illustrated embodiment, after obtaining the image data, the number of feature points corresponding to each image frame can be determined by performing feature point detection on the image frames in the image data. When the number of feature points corresponding to the image frame is not greater than a preset number, the image frame is a target image frame, and when the number of feature points corresponding to the image frame is greater than a preset number, the image frame is a non-target image frame. Accordingly, different methods are used to process the target image frames and non-target image frames to further improve the efficiency of the pose detection method. Specifically, for a target image frame having too few feature points (first feature points), in order to realize the pose detection corresponding to the target image frame, it is necessary to use multiple frames of images for pose calculation. The specific implementation principle and beneficial effects are described in detail in the following. Figure 2 This has been introduced in the illustrated embodiment and will not be repeated here. However, since in this embodiment, when using multiple frames (target image frames and other image frames) to solve the posture of the terminal device, it is necessary to transform the coordinates of the feature points corresponding to the target image frame and other image frames to form a bearing vector (bearing vector) in the corresponding camera coordinate system, and complete the subsequent posture detection steps based on the bearing vector (the specific implementation method is described in detail in the subsequent embodiment steps), the process requires matrix inversion, which requires greater computational overhead than the traditional solution of solving the posture directly based on a single frame image. Therefore, although in a possible implementation method, all image frames in all images can be determined as target image frames by setting a larger preset number, and a multi-frame posture solution solution is implemented for each target image frame, this implementation method may lead to a significant increase in the overall computational overhead of the posture detection process, thereby affecting the overall real-time detection.

[0067] In this embodiment, the image frames in the image data are detected to determine the target image frames and non-target image frames. For the non-target image frames with more feature points, the traditional single-frame pose solving method is used to calculate to obtain the second target pose, thereby reducing the calculation overhead. For the target image frames with fewer feature points, the multi-frame pose solving method is used to calculate to improve the pose detection accuracy, thereby achieving a balance between the detection accuracy and resource overhead of the pose detection method and improving the overall performance of the pose detection method.

[0068] Specifically, for non-target image frames (solving poses in single frames), the perspective n-point algorithm (Perspective-n-Point, PnP) and the random consistency sampling algorithm are used to realize pose estimation. The perspective n-point algorithm is an algorithm that solves the pose of the camera (terminal device) given the coordinates of a 3D point, the corresponding 2D point coordinates, and the internal parameter matrix. The perspective n-point algorithm has multiple implementation methods, which can be configured according to needs, and will not be repeated here.

[0069] Step S204: Acquire inertial sensor data corresponding to the image data, where the inertial sensor data represents the motion state of the terminal device when each image frame in the image data is captured.

[0070] Step S205: According to the inertial sensor data, the first feature points in the target image frame and the second feature points in other image frames are merged to obtain fused feature points.

[0071] For example, the inertial sensor data is data collected based on the inertial sensor. The inertial sensor can be set in the terminal device and rigidly connected to the terminal device to collect the motion state of the terminal device, such as movement acceleration, speed, rotation, etc. Figure 2 The inertial sensor data has been described in detail in the embodiment shown, and will not be repeated here. Figure 2 Description of corresponding parts in the illustrated embodiment.

[0072] Furthermore, since the position of the terminal device changes when the target image frame and the other image frames are captured, the target image frame and the other image frames are not in the same coordinate system, resulting in the inability to directly splice the target image frame and the other image frames to generate a fused image frame containing fused feature points. To solve the above problem, in this embodiment, after obtaining the inertial sensor data, based on the terminal device operating state represented by the inertial sensor data, the first feature point in the merged target image frame and the second feature point in the other image frames are merged into the same coordinate system, thereby realizing information fusion of the target image frame and the other image frames.

[0073] For example, Figure 8As shown, the specific implementation of step S205 includes:

[0074] Step S2051: According to the inertial sensor data, the relative posture corresponding to each image frame in the image data is obtained, and the relative posture represents the posture change of the terminal device corresponding to the current image frame relative to the terminal device posture corresponding to the adjacent image frame collected previously.

[0075] Step S2052: According to the first relative posture corresponding to the target image frame and the second relative posture corresponding to the other image frames, the first feature point and the second feature point are transformed to obtain the first orientation vector of the first feature point in the camera coordinate system of the terminal device and the second orientation vector of the second feature point in the camera coordinate system of the terminal device.

[0076] Step S2053: Based on the first orientation vector and the second orientation vector, merge the first feature point and the second feature point in the camera coordinate system of the terminal device to obtain a fused feature point.

[0077] Exemplarily, first, the relative posture corresponding to each image frame in the image data is calculated based on the inertial sensor data, that is, the first relative posture corresponding to the target image frame and the second relative posture corresponding to the other image frames. Wherein, the inertial sensor data is, for example, a vector representing acceleration, and the relative posture is the change between the terminal device postures corresponding to different image frames, which can be obtained by time integration of the acceleration vector, wherein the relative posture includes a multidimensional array, such as a 9-dimensional rotation matrix and a 3-dimensional translation matrix, and the rotation change of the terminal device posture is represented by the rotation matrix, and the translation change of the terminal device posture is represented by the translation matrix. Afterwards, the first feature point in the target image frame and the second feature point in the other image frames are respectively transformed in coordinates through the internal parameters of the image acquisition unit in the terminal device (i.e., the camera internal parameters) in combination with the first relative posture and the second relative posture, so as to obtain the first orientation vector of the first feature point in the camera coordinate system of the terminal device and the second orientation vector of the second feature point in the camera coordinate system of the terminal device. This step is equivalent to converting the first feature point and the second feature point to the same coordinate system, and describing the posture of the first feature point and the second feature point respectively based on the first orientation vector and the second orientation vector. Afterwards, based on the first orientation vector and the second orientation vector, the first feature point and the second feature point are merged to obtain the set of the first feature point and the second feature point in the camera coordinate system, i.e., the fused feature point. The fused feature point is equivalent to the fusion result of the information in the target image frame and the other image frames.

[0078] Fig. 9 A schematic diagram of obtaining fused feature points provided by an embodiment of the present disclosure is shown in FIG. Fig. 9As shown, the target image frame (shown as image frame P1 in the figure) and other image frames (shown as image frame P2 in the figure) are processed, and based on the inertial sensor data, the first relative posture corresponding to the target image frame and the second relative posture corresponding to the other image frames are obtained. Afterwards, coordinate transformation is performed based on the first relative posture and the second relative posture to obtain the first orientation vector and the second orientation vector in the camera coordinate system, and then the first orientation vector and the second orientation vector are merged to obtain the fusion feature point in the camera coordinate system.

[0079] In this embodiment, by utilizing inertial sensor data, the first feature points in the target image frame and the second feature points in the other image frames are converted and fused to obtain fused feature points, so that the information in the target image frame and the information in the other image frames are converted to the same coordinate system, thereby avoiding the problem of incorrect and inaccurate fusion of feature points due to the difference between the first posture when the target image frame is shot and the second posture when the other image frames are shot, thereby ensuring the accuracy and effectiveness of posture detection.

[0080] Step S206: Determine the first target posture according to the fused feature points.

[0081] Exemplarily, after obtaining the fused feature points, a map database search is performed based on the fused feature points in the camera coordinate system, and the corresponding three-dimensional data points are associated through the traditional two-step method. Matrix decomposition is then performed to obtain the first target pose corresponding to the target image frame.

[0082] Exemplarily, the specific implementation of step S206 includes:

[0083] Step S2061: Acquire three-dimensional reference feature points corresponding to the reference image in the map database according to the fused feature points.

[0084] Step S2062: Obtain a corresponding optimization matrix according to the optimization function corresponding to the reference feature point.

[0085] Step S2063: Perform matrix decomposition on the optimization matrix to obtain the first target pose.

[0086] For example, after obtaining the fusion feature points represented by the orientation vector, the three-dimensional reference feature points corresponding to the reference image in the corresponding map database are associated through the traditional two-step method, and then an optimization function is established based on the three-dimensional reference feature points, and the function is optimized with the purpose of minimizing the optimization function value. After that, the optimization function is written as a matrix and the matrix is ​​decomposed to obtain the first target posture. Among them, the specific implementation method of SVD matrix decomposition is a prior art and will not be repeated here.

[0087] In this embodiment, step S201 is consistent with step S101 in the above embodiment. For a detailed discussion, please refer to the discussion of step S101, which will not be repeated here.

[0088] Corresponding to the posture detection method of the above embodiment, Fig.10 This is a structural block diagram of a posture detection device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Fig.10 , the posture detection device 3 includes:

[0089] An acquisition module 31 is used to acquire image data, wherein the image data includes at least two image frames acquired by a terminal device;

[0090] A processing module 32 is used to detect the image data and determine a target image frame in the image data, wherein the number of first feature points corresponding to the target image frame is not greater than a preset number, and the first feature point is at least one set of pixel points in the image frame matched to the image feature of the reference image in the map database;

[0091] The prediction module 33 is used to determine the first target posture of the terminal device when capturing the target image frame based on the first feature point in the target image frame and the second feature point in at least one other image frame in the image data.

[0092] In one embodiment of the present disclosure, the prediction module 33 is specifically used to: obtain inertial sensor data corresponding to the image data, the inertial sensor data representing the motion state of the terminal device when each image frame in the image data is captured; merge the first feature point in the target image frame and the second feature point in other image frames according to the inertial sensor data to obtain a fused feature point; and determine the first target posture according to the fused feature point.

[0093] In one embodiment of the present disclosure, when the prediction module 33 merges the first feature point in the target image frame and the second feature point in the other image frames according to the inertial sensor data to obtain the fused feature point, it is specifically used to: obtain the relative posture corresponding to each image frame in the image data according to the inertial sensor data, the relative posture characterizing the posture change of the terminal device posture corresponding to the current image frame relative to the terminal device posture corresponding to the previously collected adjacent image frame; perform coordinate transformation on the first feature point and the second feature point according to the first relative posture corresponding to the target image frame and the second relative posture corresponding to the other image frames to obtain the first orientation vector of the first feature point in the camera coordinate system of the terminal device and the second orientation vector of the second feature point in the camera coordinate system of the terminal device; merge the first feature point and the second feature point in the camera coordinate system of the terminal device based on the first orientation vector and the second orientation vector to obtain the fused feature point.

[0094] In one embodiment of the present disclosure, when the prediction module 33 determines the first target posture based on the fused feature points, it is specifically used to: obtain the three-dimensional reference feature points corresponding to the reference image in the map database based on the fused feature points; obtain the corresponding optimization matrix based on the optimization function corresponding to the reference feature points; and perform matrix decomposition on the optimization matrix to obtain the first target posture.

[0095] In one embodiment of the present disclosure, the processing module 32 is specifically used to: extract features from an image frame in the image data to obtain a first image feature; perform feature comparison between the first image feature and a reference image in a map database to determine a target feature point in the image frame, wherein a feature similarity between the target feature point and a corresponding reference feature point in the reference image is greater than a similarity threshold; and determine a target image frame based on the number of target feature points.

[0096] In one embodiment of the present disclosure, at least one of the following items is also included: the acquisition interval between each image frame is not less than the first time length; the relative displacement between the acquisition positions corresponding to each image frame is not less than the first length.

[0097] In one embodiment of the present disclosure, the acquisition module 31 is specifically used to: collect at least two frames of environmental images through a terminal device, and obtain motion information corresponding to each environmental image, the motion information representing the motion state of the terminal device when shooting the environmental image; determine the relative displacement corresponding to the environmental image according to the motion information; based on the relative displacement corresponding to each environmental image, extract at least two frames of environmental images to generate image data.

[0098] In one embodiment of the present disclosure, the prediction module 33 is also used to: obtain a non-target image frame in the image data; and solve the non-target image frame using a perspective n-point algorithm and a random consistency sampling algorithm to obtain a second target pose corresponding to the non-target image frame.

[0099] The acquisition module 31, the processing module 32 and the prediction module 33 are connected in sequence. The posture detection device 3 provided in this embodiment can implement the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.

[0100] Fig.11 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown in FIG. Fig.11 As shown, the electronic device 4 includes:

[0101] A processor 41, and a memory 42 communicatively connected to the processor 41;

[0102] The memory 42 stores computer executable instructions;

[0103] The processor 41 executes the computer execution instructions stored in the memory 42 to implement the following Figure 2-Figure 9 The posture detection method in the illustrated embodiment.

[0104] Optionally, the processor 41 and the memory 42 are connected via a bus 43 .

[0105] For related instructions, please refer to Figure 2-Figure 9 The relevant descriptions and effects corresponding to the steps in the corresponding embodiments can be understood, and no further elaboration is made here.

[0106] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, which are used to implement the present invention when executed by a processor. Figure 2-Figure 9 The posture detection method provided in any one of the corresponding embodiments.

[0107] The present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements Figure 2-Figure 9 The posture detection method in the embodiment shown,

[0108] refer to Fig.12 , which shows a schematic diagram of the structure of an electronic device 900 suitable for implementing the embodiment of the present disclosure, and the electronic device 900 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.12 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0109] like Fig.12As shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0110] Typically, the following devices may be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 908 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.12 The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0111] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0112] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0113] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0114] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0115] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0116] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0117] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0118] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0119] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] In a first aspect, according to one or more embodiments of the present disclosure, a posture detection method is provided, including:

[0121] Acquire image data, wherein the image data includes at least two image frames captured by the terminal device; detect the image data to determine a target image frame in the image data, wherein the number of first feature points corresponding to the target image frame is not greater than a preset number, and the first feature point is at least one set of pixel points in the image frame that is matched to image features of a reference image in a map database; determine a first target posture of the terminal device when capturing the target image frame based on the first feature point in the target image frame and a second feature point in at least one other image frame in the image data.

[0122] According to one or more embodiments of the present disclosure, determining a first target posture of the terminal device when capturing the target image frame based on a first feature point in the target image frame and a second feature point in at least one other image frame in the image data includes: acquiring inertial sensor data corresponding to the image data, the inertial sensor data characterizing a motion state of the terminal device when each image frame in the image data is captured; merging the first feature point in the target image frame and the second feature point in the other image frames based on the inertial sensor data to obtain a fused feature point; and determining the first target posture based on the fused feature point.

[0123] According to one or more embodiments of the present disclosure, the merging of the first feature point in the target image frame and the second feature point in the other image frames to obtain fused feature points based on the inertial sensor data includes: obtaining a relative posture corresponding to each image frame in the image data based on the inertial sensor data, the relative posture characterizing a posture change amount of a terminal device posture corresponding to a current image frame relative to a terminal device posture corresponding to a previously acquired adjacent image frame; performing coordinate transformation on the first feature point and the second feature point based on the first relative posture corresponding to the target image frame and the second relative posture corresponding to the other image frames to obtain a first orientation vector of the first feature point in a camera coordinate system of the terminal device and a second orientation vector of the second feature point in the camera coordinate system of the terminal device; and merging the first feature point and the second feature point in the camera coordinate system of the terminal device based on the first orientation vector and the second orientation vector to obtain the fused feature point.

[0124] According to one or more embodiments of the present disclosure, determining the first target posture based on the fused feature points includes: obtaining three-dimensional reference feature points corresponding to the reference image in the map database based on the fused feature points; obtaining a corresponding optimization matrix based on the optimization function corresponding to the reference feature points; and performing matrix decomposition on the optimization matrix to obtain the first target posture.

[0125] According to one or more embodiments of the present disclosure, the detecting the image data and determining the target image frame in the image data includes: extracting features from the image frame in the image data to obtain a first image feature; performing feature comparison between the first image feature and a reference image in the map database to determine target feature points in the image frame, wherein a feature similarity between the target feature point and a corresponding reference feature point in the reference image is greater than a similarity threshold; and determining the target image frame based on the number of the target feature points.

[0126] According to one or more embodiments of the present disclosure, at least one of the following items is also included: the acquisition interval between each of the image frames is not less than the first time length; the relative displacement between the acquisition positions corresponding to each of the image frames is not less than the first length.

[0127] According to one or more embodiments of the present disclosure, the acquiring of image data includes: collecting at least two frames of environmental images through the terminal device, and acquiring motion information corresponding to each of the environmental images, the motion information representing the motion state of the terminal device when shooting the environmental image; determining the relative displacement corresponding to the environmental image according to the motion information; and extracting the at least two frames of environmental images based on the relative displacement corresponding to each of the environmental images to generate the image data.

[0128] According to one or more embodiments of the present disclosure, the method further includes: acquiring a non-target image frame in the image data; solving the non-target image frame using a perspective n-point algorithm and a random consistency sampling algorithm to obtain a second target pose corresponding to the non-target image frame.

[0129] In a second aspect, according to one or more embodiments of the present disclosure, a posture detection device is provided, comprising:

[0130] An acquisition module, used to acquire image data, wherein the image data includes at least two image frames acquired by a terminal device;

[0131] a processing module, configured to detect the image data, determine a target image frame in the image data, wherein the number of first feature points corresponding to the target image frame is not greater than a preset number, and the first feature points are at least one set of pixel points in the image frame matched to image features of a reference image in a map database;

[0132] A prediction module is used to determine a first target posture of the terminal device when capturing the target image frame based on a first feature point in the target image frame and a second feature point in at least one other image frame in the image data.

[0133] According to one or more embodiments of the present disclosure, the prediction module is specifically used to: obtain inertial sensor data corresponding to the image data, the inertial sensor data representing the motion state of the terminal device when each image frame in the image data is captured; based on the inertial sensor data, merge the first feature point in the target image frame and the second feature point in the other image frames to obtain a fused feature point; and determine the first target posture based on the fused feature point.

[0134] According to one or more embodiments of the present disclosure, when the prediction module merges the first feature point in the target image frame and the second feature point in the other image frames according to the inertial sensor data to obtain the fused feature point, it is specifically used to: obtain the relative posture corresponding to each image frame in the image data according to the inertial sensor data, and the relative posture characterizes the posture change of the terminal device posture corresponding to the current image frame relative to the terminal device posture corresponding to the previously collected adjacent image frame; perform coordinate transformation on the first feature point and the second feature point according to the first relative posture corresponding to the target image frame and the second relative posture corresponding to the other image frames to obtain the first orientation vector of the first feature point in the camera coordinate system of the terminal device and the second orientation vector of the second feature point in the camera coordinate system of the terminal device; merge the first feature point and the second feature point in the camera coordinate system of the terminal device based on the first orientation vector and the second orientation vector to obtain the fused feature point.

[0135] According to one or more embodiments of the present disclosure, when the prediction module determines the first target posture based on the fused feature points, it is specifically used to: obtain the three-dimensional reference feature points corresponding to the reference image in the map database based on the fused feature points; obtain the corresponding optimization matrix based on the optimization function corresponding to the reference feature points; and perform matrix decomposition on the optimization matrix to obtain the first target posture.

[0136] According to one or more embodiments of the present disclosure, the processing module is specifically used to: perform feature extraction on the image frame in the image data to obtain a first image feature; perform feature comparison between the first image feature and a reference image in the map database to determine a target feature point in the image frame, wherein a feature similarity between the target feature point and a corresponding reference feature point in the reference image is greater than a similarity threshold; and determine the target image frame based on the number of the target feature points.

[0137] According to one or more embodiments of the present disclosure, at least one of the following items is also included: the acquisition interval between each of the image frames is not less than the first time length; the relative displacement between the acquisition positions corresponding to each of the image frames is not less than the first length.

[0138] According to one or more embodiments of the present disclosure, the acquisition module is specifically used to: collect at least two frames of environmental images through the terminal device, and obtain motion information corresponding to each of the environmental images, the motion information representing the motion state of the terminal device when shooting the environmental image; determine the relative displacement corresponding to the environmental image according to the motion information; based on the relative displacement corresponding to each of the environmental images, extract the at least two frames of environmental images to generate the image data.

[0139] According to one or more embodiments of the present disclosure, the square prediction module is also used to: obtain a non-target image frame in the image data; and solve the non-target image frame using a perspective n-point algorithm and a random consistency sampling algorithm to obtain a second target pose corresponding to the non-target image frame.

[0140] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0141] The memory stores computer-executable instructions;

[0142] The processor executes the computer-executable instructions stored in the memory to implement the posture detection method described in the first aspect and various possible designs of the first aspect.

[0143] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the posture detection method described in the first aspect and various possible designs of the first aspect is implemented.

[0144] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the posture detection method described in the first aspect and various possible designs of the first aspect.

[0145] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0146] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0147] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A posture detection method, characterized in that: Applied to terminal equipment, including: Acquire image data, wherein the image data includes at least two image frames captured by the terminal device; Detecting the image data to determine a target image frame in the image data, wherein the number of first feature points corresponding to the target image frame is not greater than a preset number, and the first feature points are at least one set of pixel points in the image frame matched to image features of a reference image in a map database; A first target posture of the terminal device when acquiring the target image frame is determined based on a first feature point in the target image frame and a second feature point in at least one other image frame in the image data.

2. The method according to claim 1, characterized in that: The determining, based on the first feature point in the target image frame and the second feature point in at least one other image frame in the image data, a first target posture of the terminal device when acquiring the target image frame comprises: Acquire inertial sensor data corresponding to the image data, wherein the inertial sensor data represents a motion state of the terminal device when each image frame in the image data is collected; According to the inertial sensor data, merging the first feature point in the target image frame and the second feature point in the other image frames to obtain a fused feature point; The first target posture is determined according to the fused feature points.

3. The method according to claim 2, characterized in that The step of combining the first feature point in the target image frame and the second feature point in the other image frames according to the inertial sensor data to obtain a fused feature point includes: According to the inertial sensor data, a relative posture corresponding to each image frame in the image data is obtained, wherein the relative posture represents a posture change amount of the terminal device posture corresponding to the current image frame relative to the terminal device posture corresponding to the adjacent image frame collected previously; According to the first relative posture corresponding to the target image frame and the second relative posture corresponding to the other image frames, coordinate transformation is performed on the first feature point and the second feature point to obtain a first orientation vector of the first feature point in the camera coordinate system of the terminal device and a second orientation vector of the second feature point in the camera coordinate system of the terminal device; Based on the first orientation vector and the second orientation vector, the first feature point and the second feature point are merged in the camera coordinate system of the terminal device to obtain the fused feature point.

4. The method according to claim 2, characterized in that: The determining the first target posture according to the fused feature points includes: According to the fused feature points, obtaining three-dimensional reference feature points corresponding to the reference image in the map database; According to the optimization function corresponding to the reference feature point, a corresponding optimization matrix is ​​obtained; Perform matrix decomposition on the optimization matrix to obtain the first target pose.

5. The method according to claim 1, characterized in that The detecting the image data to determine the target image frame in the image data includes: Extracting features from image frames in the image data to obtain first image features; Performing feature comparison of the first image feature with a reference image in the map database to determine a target feature point in the image frame, wherein a feature similarity between the target feature point and a corresponding reference feature point in the reference image is greater than a similarity threshold; The target image frame is determined according to the number of the target feature points.

6. The method according to claim 1, characterized in that Also includes at least one of the following: The acquisition interval between each of the image frames is not less than the first duration; The relative displacement between the acquisition positions corresponding to the image frames is not less than the first length.

7. The method according to claim 1, characterized in that The acquiring of image data comprises: Collecting at least two frames of environmental images through the terminal device, and obtaining motion information corresponding to each of the environmental images, wherein the motion information represents the motion state of the terminal device when shooting the environmental images; Determining a relative displacement corresponding to the environment image according to the motion information; Based on the relative displacement corresponding to each of the environmental images, the at least two frames of environmental images are extracted to generate the image data.

8. The method according to claim 1, characterized in that The method further comprises: Acquire a non-target image frame in the image data; The non-target image frame is solved by using a perspective n-point algorithm and a random consistency sampling algorithm to obtain a second target pose corresponding to the non-target image frame.

9. A posture detection device, characterized in that: include: An acquisition module, used to acquire image data, wherein the image data includes at least two image frames acquired by a terminal device; a processing module, configured to detect the image data, determine a target image frame in the image data, wherein the number of first feature points corresponding to the target image frame is not greater than a preset number, and the first feature points are at least one set of pixel points in the image frame matched to image features of a reference image in a map database; A prediction module is used to determine a first target posture of the terminal device when capturing the target image frame based on a first feature point in the target image frame and a second feature point in at least one other image frame in the image data.

10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the posture detection method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the posture detection method as described in any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the posture detection method described in any one of claims 1 to 8.

Images