Pose detection method and apparatus, electronic device, storage medium, and computer program product

Abstract

A pose detection method, a pose detection apparatus, an electronic device, a storage medium, and a computer program product. The method comprises: acquiring image data, wherein the image data comprises at least two image frames collected by a terminal device; testing the image data to determine a target image frame in the image data, wherein the number of first feature points corresponding to the target image frame is not greater than a preset number, and the first feature point is at least one pixel point set in the image frames obtained by performing image feature matching with reference images in a map database; and on the basis of the first feature point in the target image frame and a second feature point in at least one other image frame in the image data, determining a first target pose of the terminal device when collecting the target image frame. The pose detection method provided by the embodiments of the present disclosure solves the problems of low success rate and poor accuracy of pose detection caused by the fact that the individual image frames collected by the terminal device cannot provide enough image information.

Description

Posture detection method, device, electronic device, storage medium and computer program product

[0001] This application claims priority to Chinese Patent Application No. 202311014646.0 filed on August 11, 2023, and the contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application. Technical Field

[0002] The embodiments of the present disclosure relate to the field of visual positioning technology, and in particular to a posture detection method, a posture detection device, an electronic device, a storage medium, and a computer program product. Background Art

[0003] Visual positioning technology refers to the use of image acquisition devices installed on terminal devices, such as cameras, to collect and process surrounding image information, thereby achieving accurate estimation of the terminal device's position and posture. It is often used in application scenarios such as indoor navigation and high-precision vehicle navigation.

[0004] Conventional solutions for detecting the pose of a terminal device typically compare feature points in a captured image of the environment with those in pre-captured images stored in a map database. However, due to the complex environments in which the terminal device resides, these conventional solutions suffer from low pose detection success rates and poor accuracy.

[0005] Summary of the Invention

[0006] The embodiments of the present disclosure provide a posture detection method, a posture detection device, an electronic device, a storage medium, and a computer program product to overcome the problems of low posture detection success rate and poor accuracy.

[0007] In a first aspect, an embodiment of the present disclosure provides a posture detection method, comprising:

[0008] Acquire image data, wherein the image data includes at least two image frames captured by a terminal device; detect the image data to determine a target image frame in the image data, wherein the number of first feature points corresponding to the target image frame is not greater than a preset number, and the first feature point is at least one set of pixel points in the image frame matched to image features of a reference image in a map database; determine a first target posture of the terminal device when capturing the target image frame based on the first feature point in the target image frame and a second feature point in at least one other image frame in the image data.

[0009] In a second aspect, an embodiment of the present disclosure provides a posture detection device, comprising:

[0010] an acquisition module configured to acquire image data, wherein the image data includes at least two image frames captured by a terminal device;

[0011] a processing module configured to detect the image data and determine a target image frame in the image data, wherein the number of first feature points corresponding to the target image frame is no greater than a preset number, and the first feature points are at least one set of pixel points in the image frame matched based on image features of a reference image in a map database;

[0012] The prediction module is configured to determine a first target pose of the terminal device when capturing the target image frame based on a first feature point in the target image frame and a second feature point in at least one other image frame in the image data.

[0013] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0014] a processor, and a memory communicatively connected to the processor;

[0015] The memory stores computer-executable instructions;

[0016] The processor executes the computer-executable instructions stored in the memory to implement the posture detection method described in the first aspect and various possible designs of the first aspect.

[0017] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the posture detection method described in the first aspect and various possible designs of the first aspect is implemented.

[0018] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the posture detection method described in the first aspect and various possible designs of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] FIG1 is a diagram of an application scenario of the posture detection method provided by an embodiment of the present disclosure;

[0021] FIG2 is a flow chart of a posture detection method according to an embodiment of the present disclosure;

[0022] FIG3 is a flow chart of a specific implementation of step S101 in the embodiment shown in FIG2 ;

[0023] FIG4 is a schematic diagram of a process for generating image data provided by an embodiment of the present disclosure;

[0024] FIG5 is a flowchart of a specific implementation of step S102 in the embodiment shown in FIG2 ;

[0025] FIG6 is a schematic diagram of a target feature point provided by an embodiment of the present disclosure;

[0026] FIG7 is a second flow chart of the posture detection method provided by an embodiment of the present disclosure;

[0027] FIG8 is a flowchart of a specific implementation of step S205 in the embodiment shown in FIG7 ;

[0028] FIG9 is a schematic diagram of obtaining fused feature points according to an embodiment of the present disclosure;

[0029] FIG10 is a structural block diagram of a posture detection device provided by an embodiment of the present disclosure;

[0030] FIG11 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure;

[0031] FIG12 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0033] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0034] The following explains the application scenarios of the embodiments of the present disclosure:

[0035] FIG1 is a diagram of an application scenario of the posture detection method provided by an embodiment of the present disclosure. The posture detection method provided by an embodiment of the present disclosure can be applied to scenarios such as pedestrian navigation and vehicle precise navigation. Specifically, the method provided by an embodiment of the present disclosure can be applied to terminal devices, such as handheld devices such as smartphones, or wearable devices such as smart glasses and helmets. As shown in FIG1 , the terminal device takes smart glasses as an example. The user installs and runs navigation software on the smart glasses 11 to achieve precise navigation indoors or outdoors. Specifically, a rigidly connected image acquisition unit, such as a camera, is provided in the smart glasses 11. After the navigation software is run, the smart glasses 11 collects environmental images in real time through the image acquisition unit, and uses the method provided by this embodiment, combined with a pre-generated map database, to process the environmental images, and uses the content in the environmental images to determine the real-time position and orientation of the smart glasses 11, that is, the posture of the terminal device, and displays the posture on the display screen 12 of the terminal device, thereby achieving the positioning of the target object in the terminal device / environment.

[0036] In the prior art, for the above-mentioned application scenarios such as pedestrian navigation and vehicle precision navigation, the scheme for realizing the posture detection of the terminal device is usually based on comparing the feature points in the collected environmental image with the feature points in the pre-collected image in the map database, and then mapping them into three-dimensional world coordinates to realize the posture detection of the terminal device. In this process, it is usually required that the collected image frame include multiple feature points (usually more than 3) to improve the accuracy of the detection results. However, in actual application, due to the complex and changeable environment in which the terminal device is located, when the terminal device moves or shakes significantly, or faces a wall, resulting in a low amount of information in the collected environmental image, the number of feature points in the environmental image will be less than 3, or even no feature points can be matched, resulting in the posture detection for the frame image being invalid or inaccurate. This will lead to problems such as stuck and jumping navigation coordinates in the user interface of the terminal device, affecting the user experience and causing safety hazards.

[0037] The present disclosure provides a posture detection method to solve the above-mentioned problem. By detecting image data collected by a terminal, a target image frame with too few feature points is determined. The target image frame is then combined with feature points in other image frames in the image data to jointly determine the posture of the terminal device when the target image frame was collected. This solves the problem of low posture detection success rate and poor accuracy caused by individual image frames not providing sufficient image information.

[0038] Refer to Figure 2, which is a flow chart of the posture detection method provided by the embodiment of the present disclosure. The method of this embodiment can be applied to a terminal device or a server. When the method provided by this embodiment is executed by the server, it is necessary to obtain corresponding data by communicating with the terminal device, and after obtaining the detection result, the detection result can be sent to the terminal device for display. In the following embodiments, the terminal device is used as the execution subject of the method provided by this embodiment. For example, the posture detection method includes:

[0039] Step S101: Acquire image data, where the image data includes at least two image frames captured by a terminal device.

[0040] For example, referring to the application scenario diagram shown in FIG1 , the terminal device captures images of the external environment through the image acquisition unit provided, and can generate corresponding image frames, wherein the collection of at least two image frames collected is image data. In one possible implementation, the terminal device continuously captures images of the external environment at equal sampling intervals over a period of time, and uses the collection of all environmental images (image frames) collected during the time period as image data for subsequent processing. Furthermore, in one possible implementation, the above-mentioned acquisition process and subsequent processing process can be performed synchronously, that is, after the above-mentioned acquisition process is completed and the determined image data is obtained, the subsequent image processing steps are performed; for example, the terminal device captures 10 frames of environmental images at equal time intervals over a period of time from the 0th second to the 5th second: environmental images P0 to P9. Starting from the 5th second, environmental images P0 to P9 are used as image data for subsequent processing.

[0041] In another possible implementation, the acquisition process and subsequent processing can be performed asynchronously. That is, after the terminal device obtains at least two frames of image data, it can proceed to the subsequent processing steps for the image data. At the same time, the terminal device continues to acquire images and continues to obtain environmental images. The subsequent processing steps are performed using the collection of real-time environmental images (image frames) as image data. For example, the terminal device begins acquiring environmental images at the 0th second and continuously acquires environmental images at equal time intervals. For example, the terminal device acquires environmental image P0 at the 0.5th second, environmental image P1 at the 1st second, and so on. After the terminal device captures the environmental image P1 at the first second, it uses the environmental images P0 and P1 as image data for subsequent processing, while continuing to capture new environmental images. After the terminal device captures the environmental image P_N at the Nth second, it uses the environmental image P_N and the m environmental images preceding it as image data for subsequent processing, ensuring that the image data always represents the most recently captured m+1 frame of environmental image. This allows for continuous posture detection of the terminal based on the most recently captured m+1 frame of environmental image. This implementation requires the cooperation of at least two processing threads. The specific implementation method for multi-threaded data asynchronous processing is well known to those skilled in the art and will not be elaborated upon here.

[0042] In another possible implementation, after capturing environmental images, the terminal device screens the environmental images, selects those that meet the requirements as image frames, and uses a set of at least two image frames as image data to obtain image data. Exemplarily, the environmental images that meet the requirements include at least one of the following: the capture interval between each environmental image is not less than a first duration; the relative displacement between the capture positions corresponding to each environmental image is not less than a first length. Specifically, for example, every 0.5 seconds, one frame of the captured environmental image is extracted as the image frame constituting the image data; for another example, based on the capture position when the environmental image is captured, when the relative displacement between the capture positions corresponding to two adjacent frames is greater than 1 meter, the corresponding environmental image is used as the image frame constituting the image data. The relative displacement between the capture positions when the environmental image is captured can be determined by other reference information captured by the terminal device, such as inertial sensor data captured by an inertial measurement unit (IMU).

[0043] As shown in FIG3 , illustratively, the specific implementation of step S101 includes:

[0044] Step S1011: At least two frames of environmental images are captured by the terminal device, and motion information corresponding to each environmental image is obtained. The motion information represents the motion state of the terminal device when capturing the environmental image.

[0045] Step S1012: Determine the relative displacement corresponding to the environment image according to the motion information.

[0046] Step S1013: extracting at least two frames of environmental images based on the relative displacement corresponding to each environmental image to generate image data.

[0047] Based on the above introduction, illustratively, while continuously performing posture detection, the terminal device uses its own image acquisition unit, such as a camera, to capture environmental images based on a specific image sampling frequency, for example, capturing an environmental image every 0.5 seconds, and setting a capture timestamp for each environmental image. Simultaneously, the terminal device captures motion information of the terminal device at a motion information sampling frequency no less than the aforementioned image sampling frequency. The motion information represents the motion state of the terminal device when capturing the environmental image, such as acceleration and velocity. In one possible implementation, the acquisition phases of the environmental image acquisition process and the motion information acquisition process are synchronized, i.e., the terminal device simultaneously captures the environmental image and motion information at the same (logical) moment, thereby giving the environmental image and motion information the same acquisition timestamp, and thereby establishing a timestamp-based mapping between the two. In another possible implementation, the environmental image acquisition process and the motion information acquisition process are asynchronously phased. That is, the terminal device acquires the environmental image and motion information at different (logical) moments, resulting in different acquisition timestamps for the environmental image and motion information. In this case, the acquisition timestamp of the motion information closest to the environmental image can be determined based on the acquisition timestamp of the environmental image, thereby establishing a timestamp-based mapping between the two. The specific implementation method can be configured as needed and is not further described here. Motion information can be obtained through inertial sensor data collected by the inertial sensor and is not further described here.

[0048] Furthermore, motion information corresponding to each environmental image is obtained, and based on the motion information, the relative displacement corresponding to the environmental image is determined. Specifically, for example, the motion information is such as the acceleration or velocity vector when the terminal device shoots the environmental image. By integrating the motion information, the relative displacement when the terminal device shoots each environmental image can be obtained. Thereafter, the relative displacement is accumulated. Whenever the accumulated relative displacement is greater than or equal to a preset first length, the corresponding environmental image is used as an image frame constituting the image data.

[0049] Figure 4 is a schematic diagram of a process for generating image data provided by an embodiment of the present disclosure. The above process is described in detail below with reference to Figure 4. As shown in Figure 4, referring to the acquisition time axis, the terminal device acquires environmental images at time intervals of 0.1 seconds (shown as 0.1 in the figure, the same below), and acquires motion information at time intervals of 0.5 seconds. Starting from the 0th second, environmental image acquisition and motion information acquisition are performed simultaneously. After that, the terminal device extracts and acquires a frame of environmental image every 0.5 seconds, including at least the environmental image P1, environmental image P2, environmental image P3, environmental image P4, and environmental image P5 shown in the figure (shown as P1, P2, P3, P4, and P5, respectively, in the figure). And obtain the motion information of the above-mentioned extracted environmental images (shown as S1, S2, S3, S4, and S5 in the figure respectively). Then, based on the motion information of the above-mentioned extracted environmental images, determine the relative displacements between the acquisition positions of environmental images P1, environmental image P2, environmental image P3, environmental image P4, and environmental image P5 when they are acquired. As shown in the reference acquisition position axis, environmental image P1 is the initial point and the relative displacement is 0; the relative displacement of environmental image P2 is the displacement between the position when environmental image P2 is acquired and the position when environmental image P1 is acquired, for example, Dis_1; similarly, the relative displacement of environmental image P3 is Dis_2, the relative displacement of environmental image P4 is Dis_3, and the relative displacement of environmental image P5 is Dis_4. The relative displacements are accumulated and compared with a preset first length Dis_0. Since Dis_1 is greater than Dis_0, the environmental image P2 corresponding to Dis_1 is set as the image frame constituting the image data. Then, starting from the location where environmental image P2 was captured, the sum of Dis_2 and Dis_3 is greater than Dis_0, so the environmental image P4 corresponding to Dis_3 is set as another image frame constituting the image data. Starting from the location where environmental image P5 was captured, since Dis_4 is greater than Dis_0, the environmental image P5 corresponding to Dis_4 is set as another image frame constituting the image data. This process continues in this manner until all captured environmental images have been detected. At this point, the environmental image that meets the requirements after the above steps is detected is selected as the image frame in the image data. Alternatively, the environmental image that meets the requirements after the above steps is continuously selected as the image frame in the image data.

[0050] In the steps of this embodiment, the initial environmental image collected is extracted by collecting intervals and / or relative displacements between collection positions to obtain image data that meets the requirements in time and space. On the one hand, the amount of image data to be processed subsequently is reduced, reducing the operating load and resource consumption. On the other hand, since similar image frames that are close in time and / or space are removed, the effective information density and information validity of the image data are improved, thereby improving the computational efficiency of posture detection.

[0051] Step S102: Detect the image data and determine the target image frame in the image data. The number of first feature points corresponding to the target image frame is not greater than a preset number. The first feature point is at least one set of pixel points in the image frame that is matched with the image features of the reference image in the map database.

[0052] Exemplarily, after obtaining image data, the image data is inspected to determine a target image frame. The target image frame has a number of first feature points no greater than a preset number. A first feature point is a set of at least one pixel in the image frame that matches the image features of a reference image in a map database. Simply put, a target image frame is an image frame that does not meet the feature point quantity requirement. The first feature points corresponding to the target image frame are referred to as feature points in the target image frame. Feature points are pixels in the image frame that have similar image features after comparing the image frame with a pre-captured reference image in a map database. In visual localization technology based on a priori map databases, the map database contains a 3D image model and a 2D image model. By querying the map database, the 2D image in the 2D image model is first matched, and then the 2D image in the map database is mapped to the 3D image model, thereby determining the pose of the image acquisition point. In the above process, the step of querying the map database and matching the 2D image in the 2D image model, i.e., determining the feature points, is followed by counting the feature points to determine the number of feature points in the image frame. The specific method for determining the feature points can be implemented by a random consensus sampling (RANSAC) algorithm, and its implementation process is an existing technology known in the art and will not be described here in detail.

[0053] Furthermore, by detecting the number of feature points corresponding to each image frame in the image data, the image frame with a number of feature points less than a preset number is determined as a target image frame, and the feature points in the target image frame are first feature points, that is, the number of first feature points is less than the preset number; correspondingly, the image frame with a number of feature points greater than or equal to the preset number is determined as a non-target image frame, and the feature points in the non-target image frame are second feature points, that is, the number of second feature points is greater than or equal to the preset number.

[0054] For example, as shown in FIG5 , the specific implementation of step S102 includes:

[0055] Step S1021: extracting features from image frames in the image data to obtain first image features.

[0056] Step S1022: performing feature comparison on the first image feature and the reference image in the map database to determine the target feature point in the image frame, wherein the feature similarity between the target feature point and the corresponding reference feature point in the reference image is greater than a similarity threshold.

[0057] Step S1023: Determine the target image frame according to the number of target feature points.

[0058] For example, first, after obtaining the image data, based on the acquisition timestamp of each image frame in the image data, feature extraction is performed in sequence to obtain the corresponding pixel feature matrix, i.e., the first image feature. Then, based on the first image feature, feature comparison is performed on the reference image in the map database to retrieve a reference image with similar image features to the image frame. Since the reference image is an image acquired in a real environment, when the image frame in the image data has similar features (pixel point sets) as those in the reference image, it means that the object in the image frame can be mapped to the object in the real environment to a certain extent, that is, the pose of the acquired image frame is similar to the pose when the reference image was acquired (in order to capture similar content). The higher the feature similarity between the feature point in the image frame and the reference feature point in the reference frame, the more likely the feature point is the target feature point. At the same time, the more target feature points in the image frame, the higher the possibility that the image frame and the reference image were acquired in the same or similar pose. The method for determining the target feature points in the image frame can be implemented by a random consistency sampling algorithm. The implementation process is a prior art known in the art and will not be repeated here.

[0059] Figure 6 is a schematic diagram of a target feature point provided by an embodiment of the present disclosure. As shown in Figure 6, taking image frame P1 in the image data as an example, after acquiring image frame P1, feature extraction is performed on the image frame to obtain a first image feature of image frame P1. The first image feature can be implemented in various ways such as a matrix, a vector or a structure. Thereafter, the reference image in the map database is retrieved based on the first image feature, and the target feature points in image frame P1 are determined based on the data in the map database. Specifically, the coordinates and identification of each feature point can be obtained, such as the feature point T1 and the feature point T2 shown in the figure. Since image frame P1 only includes two feature points, image frame P1 is determined as the target image frame.

[0060] Step S103: Determine a first target posture of the terminal device when capturing the target image frame based on the first feature point in the target image frame and the second feature point in at least one other image frame in the image data.

[0061] For example, after determining the target image frame in the image data, the small number of first feature points in the target image frame may cause deviation in the pose detection result. Therefore, based on the target image frame, other image frames in the image data are further combined, and the feature points in other image frames, i.e., the second feature points, are fused with the first feature points in the target image frame. This is equivalent to fusing the information in the target image frame with the information in other image frames, thereby obtaining a fused image frame with richer information. Afterwards, pose detection is performed based on the fused image frame, which can obtain better detection results than directly using the target image frame for pose detection. In one possible implementation, during the process of continuously performing posture detection on the terminal device, after the target image frame in the image data is detected, the target image frame is feature-fused with at least one other image frame before the target image frame in the image data. Due to the specific time and / or spatial acquisition limitations between the target image frame and other image frames in the image data, it is possible to ensure that the target image frame and other image frames have large differences in image content. Performing feature fusion on the target image frame and other image frames is equivalent to merging the image field of view corresponding to the target image frame and the image field of view of the other image frames, thereby obtaining a fused image frame with a large field of view. The fused image frame must contain more feature points. Therefore, the first target posture obtained when the target image frame is acquired by posture detection based on the fused image frame has better accuracy and information. The specific implementation method for posture judgment based on the image (in this embodiment, the fused image synthesized based on the target image frame and other image frames) is a prior art and will not be described in detail here.

[0062] In this embodiment, image data is acquired, the image data including at least two image frames captured by a terminal device; the image data is detected to determine a target image frame in the image data, the number of first feature points corresponding to the target image frame is no more than a preset number, the first feature point being at least one set of pixel points in the image frame matched to the image feature of a reference image in a map database; and the first target posture of the terminal device when capturing the target image frame is determined based on the first feature point in the target image frame and the second feature point in at least one other image frame in the image data. By determining a target image frame with too few feature points in the image data captured by the terminal device, and combining the feature points in the target image frame and other image frames to jointly determine the posture of the terminal device when capturing the target image frame, the problem of low posture detection success rate and poor accuracy caused by the inability of individual image frames captured by the terminal device to provide sufficient image information is solved.

[0063] Referring to FIG7 , FIG7 is a second flow chart of the posture detection method provided by an embodiment of the present disclosure. Based on the embodiment shown in FIG2 , this embodiment further refines step S103 and adds a step of processing non-target image frames. The posture detection method includes:

[0064] Step S201: Acquire image data, where the image data includes at least two image frames captured by a terminal device.

[0065] Step S202: Detect the image frame in the image data. If the image frame is a non-target image frame, execute step S203; if the image frame is a target image frame, execute step S204.

[0066] Step S203: For the non-target image frame, a perspective n-point algorithm and a random consistency sampling algorithm are used to solve the non-target image frame to obtain a second target pose corresponding to the non-target image frame.

[0067] Exemplarily, referring to the relevant introduction in the embodiment shown in FIG2 , after obtaining the image data, the number of feature points corresponding to each image frame can be determined by performing feature point detection on the image frames in the image data. When the number of feature points corresponding to the image frame is not greater than a preset number, the image frame is a target image frame, and when the number of feature points corresponding to the image frame is greater than a preset number, the image frame is a non-target image frame. Accordingly, different methods are used to process the target image frame and the non-target image frame to further improve the efficiency of the posture detection method. Specifically, for a target image frame with too few feature points (first feature points), in order to realize the posture detection corresponding to the target image frame, it is necessary to use multiple frames of images for posture calculation. The specific implementation principle and beneficial effects have been introduced in the embodiment shown in FIG2 and will not be repeated here. However, since in this embodiment, when multiple frames (target image frame and other image frames) are used to solve the posture of the terminal device, it is necessary to transform the coordinates of the feature points corresponding to the target image frame and the other image frames to form a bearing vector (bearing vector) in the corresponding camera coordinate system, and complete the subsequent posture detection steps based on the bearing vector (the specific implementation method is described in detail in the subsequent embodiment steps). Matrix inversion is required in this process, which requires greater computational overhead than the traditional solution of directly solving the posture based on a single frame image. Therefore, although in a possible implementation method, a larger preset number can be set to determine all image frames in all images as target image frames, and a multi-frame posture solution solution is implemented for each target image frame, this implementation method may lead to a significant increase in the overall computational overhead of the posture detection process, thereby affecting the overall real-time detection.

[0068] In this embodiment, the image frames in the image data are detected to determine the target image frames and non-target image frames. For the non-target image frames with more feature points, the traditional single-frame pose solving method is used for calculation to obtain the second target pose, thereby reducing the computational overhead. For the target image frames with fewer feature points, the multi-frame pose solving method is used for calculation to improve the pose detection accuracy, thereby achieving a balance between the detection accuracy and resource overhead of the pose detection method and improving the overall performance of the pose detection method.

[0069] Specifically, for non-target image frames (single-frame pose estimation), we use the Perspective-n-Point (PnP) algorithm and the random consistency sampling algorithm to achieve pose estimation. The Perspective-n-Point algorithm is an algorithm that solves the pose of the camera (end device) given the coordinates of a 3D point, the corresponding 2D point coordinates, and an intrinsic parameter matrix. The Perspective-n-Point algorithm has multiple implementations, which can be configured as needed and will not be detailed here.

[0070] Step S204: Acquire inertial sensor data corresponding to the image data, where the inertial sensor data represents the motion state of the terminal device when each image frame in the image data is captured.

[0071] Step S205: Based on the inertial sensor data, the first feature points in the target image frame and the second feature points in the other image frames are merged to obtain fused feature points.

[0072] Exemplarily, the inertial sensor data is data collected based on the inertial sensor. The inertial sensor can be set in the terminal device and rigidly connected to the terminal device to collect the motion state of the terminal device, such as mobile acceleration, speed, rotation, etc. The inertial sensor data has been introduced in detail in the embodiment shown in Figure 2 and will not be repeated here. Please refer to the description of the corresponding part of the embodiment shown in Figure 2.

[0073] Furthermore, because the position of the terminal device changes during the acquisition of the target image frame and the other image frames, the target image frame and the other image frames are not in the same coordinate system. This makes it impossible to directly splice the target image frame and the other image frames to generate a fused image frame containing fused feature points. To address this issue, in this embodiment, after obtaining inertial sensor data, based on the terminal device operating state represented by the inertial sensor data, the first feature point in the merged target image frame and the second feature point in the other image frames are merged into the same coordinate system, thereby achieving information fusion between the target image frame and the other image frames.

[0074] For example, as shown in FIG8 , the specific implementation of step S205 includes:

[0075] Step S2051: Based on the inertial sensor data, the relative posture corresponding to each image frame in the image data is obtained. The relative posture represents the posture change of the terminal device corresponding to the current image frame relative to the terminal device posture corresponding to the adjacent image frame previously collected.

[0076] Step S2052: According to the first relative posture corresponding to the target image frame and the second relative posture corresponding to the other image frames, the first feature point and the second feature point are coordinate transformed to obtain the first orientation vector of the first feature point in the camera coordinate system of the terminal device and the second orientation vector of the second feature point in the camera coordinate system of the terminal device.

[0077] Step S2053: Based on the first orientation vector and the second orientation vector, the first feature point and the second feature point are merged in the camera coordinate system of the terminal device to obtain a fused feature point.

[0078] Exemplarily, first, the relative pose corresponding to each image frame in the image data is calculated based on the inertial sensor data, namely, a first relative pose corresponding to the target image frame and a second relative pose corresponding to the other image frames. The inertial sensor data may be, for example, a vector representing acceleration. The relative pose is the change between the terminal device poses corresponding to different image frames, which can be obtained by time-integrating the acceleration vector. The relative pose includes a multidimensional array, such as a 9-dimensional rotation matrix and a 3-dimensional translation matrix. The rotation matrix represents the rotational change of the terminal device pose, and the translation matrix represents the translational change of the terminal device pose. Afterwards, the internal parameters of the image acquisition unit in the terminal device (i.e., the camera internal parameters) are combined with the first relative posture and the second relative posture to respectively transform the coordinates of the first feature point in the target image frame and the second feature point in the other image frames, and obtain the first orientation vector of the first feature point in the camera coordinate system of the terminal device and the second orientation vector of the second feature point in the camera coordinate system of the terminal device. This step is equivalent to transforming the first feature point and the second feature point to the same coordinate system, and describing the postures of the first feature point and the second feature point respectively based on the first orientation vector and the second orientation vector. Afterwards, based on the first orientation vector and the second orientation vector, a set of the first feature point and the second feature point located in the camera coordinate system can be obtained, i.e., the fused feature point. The fused feature point is equivalent to the fusion result of the information in the target image frame and the other image frames.

[0079] FIG9 is a schematic diagram of obtaining fused feature points according to an embodiment of the present disclosure. As shown in FIG9 , a target image frame (shown as image frame P1) and other image frames (shown as image frame P2) are processed, and based on inertial sensor data, a first relative pose corresponding to the target image frame and a second relative pose corresponding to the other image frames are obtained. Subsequently, coordinate transformation is performed based on the first relative pose and the second relative pose to obtain a first orientation vector and a second orientation vector in the camera coordinate system. The first orientation vector and the second orientation vector are then merged to obtain a fused feature point in the camera coordinate system.

[0080] In this embodiment, by utilizing inertial sensor data, the first feature point in the target image frame and the second feature point in the other image frames are converted and fused to obtain fused feature points, so that the information in the target image frame and the information in the other image frames are converted to the same coordinate system, avoiding the problem of incorrect and inaccurate feature point fusion caused by the difference between the first posture when the target image frame is shot and the second posture when the other image frames are shot, thereby ensuring the accuracy and effectiveness of posture detection.

[0081] Step S206: Determine the first target pose based on the fused feature points.

[0082] For example, after obtaining the fused feature points, a map database search is performed based on the fused feature points in the camera coordinate system, and the corresponding three-dimensional data points are associated through the traditional two-step method. After matrix decomposition is performed, the first target pose corresponding to the target image frame can be obtained.

[0083] Exemplarily, the specific implementation of step S206 includes:

[0084] Step S2061: Acquire three-dimensional reference feature points corresponding to the reference image in the map database based on the fused feature points.

[0085] Step S2062: Obtain the corresponding optimization matrix according to the optimization function corresponding to the reference feature point.

[0086] Step S2063: Perform matrix decomposition on the optimization matrix to obtain the first target pose.

[0087] For example, after obtaining fused feature points represented by orientation vectors, they are linked to the corresponding 3D reference feature points in the reference image in the corresponding map database using a traditional two-step method. An optimization function is then established based on the 3D reference feature points and optimized to minimize the function's value. The optimization function is then expressed as a matrix and the matrix is ​​decomposed to obtain the first target pose. The specific implementation of SVD matrix decomposition is conventional and will not be detailed here.

[0088] In this embodiment, step S201 is consistent with step S101 in the above embodiment. For detailed discussion, please refer to the discussion of step S101, which will not be repeated here.

[0089] Corresponding to the posture detection method of the above embodiment, FIG10 is a structural block diagram of a posture detection device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Referring to FIG10 , the posture detection device 3 includes:

[0090] An acquisition module 31 is configured to acquire image data, where the image data includes at least two image frames captured by a terminal device;

[0091] a processing module 32 configured to detect the image data and determine a target image frame in the image data, wherein the number of first feature points corresponding to the target image frame is no greater than a preset number, the first feature point being at least one set of pixels in the image frame matched to image features of a reference image in a map database;

[0092] The prediction module 33 is configured to determine a first target posture of the terminal device when capturing the target image frame based on a first feature point in the target image frame and a second feature point in at least one other image frame in the image data.

[0093] In one embodiment of the present disclosure, the prediction module 33 is specifically configured to: obtain inertial sensor data corresponding to the image data, the inertial sensor data representing the motion state of the terminal device when each image frame in the image data is captured; based on the inertial sensor data, merge the first feature point in the target image frame and the second feature point in the other image frames to obtain a fused feature point; and determine the first target posture based on the fused feature point.

[0094] In one embodiment of the present disclosure, when the prediction module 33 merges the first feature point in the target image frame and the second feature point in the other image frames according to the inertial sensor data to obtain the fused feature point, it is specifically configured as follows: according to the inertial sensor data, obtain the relative posture corresponding to each image frame in the image data, the relative posture characterizing the posture change of the terminal device posture corresponding to the current image frame relative to the terminal device posture corresponding to the previously collected adjacent image frame; according to the first relative posture corresponding to the target image frame and the second relative posture corresponding to the other image frames, perform coordinate transformation on the first feature point and the second feature point to obtain the first orientation vector of the first feature point in the camera coordinate system of the terminal device and the second orientation vector of the second feature point in the camera coordinate system of the terminal device; based on the first orientation vector and the second orientation vector, merge the first feature point and the second feature point in the camera coordinate system of the terminal device to obtain the fused feature point.

[0095] In one embodiment of the present disclosure, when the prediction module 33 determines the first target posture based on the fused feature points, it is specifically configured as follows: based on the fused feature points, obtain the three-dimensional reference feature points corresponding to the reference image in the map database; based on the optimization function corresponding to the reference feature points, obtain the corresponding optimization matrix; perform matrix decomposition on the optimization matrix to obtain the first target posture.

[0096] In one embodiment of the present disclosure, the processing module 32 is specifically configured to: extract features from the image frame in the image data to obtain a first image feature; perform feature comparison between the first image feature and a reference image in a map database to determine a target feature point in the image frame, wherein the feature similarity between the target feature point and the corresponding reference feature point in the reference image is greater than a similarity threshold; and determine a target image frame based on the number of target feature points.

[0097] In one embodiment of the present disclosure, at least one of the following items is also included: the acquisition interval between each image frame is not less than the first time length; the relative displacement between the acquisition positions corresponding to each image frame is not less than the first length.

[0098] In one embodiment of the present disclosure, the acquisition module 31 is specifically configured to: collect at least two frames of environmental images through a terminal device, and obtain motion information corresponding to each environmental image, where the motion information represents the motion state of the terminal device when shooting the environmental image; determine the relative displacement corresponding to the environmental image based on the motion information; and extract at least two frames of environmental images based on the relative displacement corresponding to each environmental image to generate image data.

[0099] In one embodiment of the present disclosure, the prediction module 33 is further configured to: obtain a non-target image frame in the image data; and solve the non-target image frame using the perspective n-point algorithm and the random consistency sampling algorithm to obtain a second target pose corresponding to the non-target image frame.

[0100] The acquisition module 31, the processing module 32 and the prediction module 33 are connected in sequence. The posture detection device 3 provided in this embodiment can implement the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.

[0101] FIG11 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. As shown in FIG11 , the electronic device 4 includes:

[0102] A processor 41, and a memory 42 communicatively connected to the processor 41;

[0103] Memory 42 stores computer-executable instructions;

[0104] The processor 41 executes the computer-executable instructions stored in the memory 42 to implement the posture detection method in the embodiments shown in Figures 2 to 9.

[0105] Optionally, the processor 41 and the memory 42 are connected via a bus 43 .

[0106] The relevant explanations can be understood by referring to the relevant descriptions and effects corresponding to the steps in the embodiments corresponding to Figures 2 to 9, and no further details will be given here.

[0107] An embodiment of the present disclosure provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, they are used to implement the posture detection method provided in any of the embodiments corresponding to Figures 2 to 9 of the present disclosure.

[0108] The present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the posture detection method in the embodiments shown in FIG. 2 to FIG. 9 .

[0109] Referring to FIG12 , there is shown a schematic diagram of the structure of an electronic device 900 suitable for implementing an embodiment of the present disclosure. The electronic device 900 may be a terminal device or a server. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG12 is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0110] As shown in FIG12 , the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. Various programs and data required for the operation of the electronic device 900 are also stored in the RAM 903. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0111] Typically, the following devices may be connected to the I / O interface 905: an input device 906 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 908 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although FIG12 shows an electronic device 900 with various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0112] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 909, or installed from the storage device 908, or installed from the ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0113] It should be noted that the computer-readable medium described above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0114] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0115] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0116] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0117] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0118] The units described in the embodiments of the present disclosure may be implemented in software or hardware. The name of a unit does not necessarily limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0119] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0120] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0121] In a first aspect, according to one or more embodiments of the present disclosure, a posture detection method is provided, comprising:

[0122] Acquire image data, wherein the image data includes at least two image frames captured by the terminal device; detect the image data to determine a target image frame in the image data, wherein the number of first feature points corresponding to the target image frame is not greater than a preset number, and the first feature point is at least one set of pixel points in the image frame matched to image features of a reference image in a map database; determine a first target posture of the terminal device when capturing the target image frame based on the first feature point in the target image frame and a second feature point in at least one other image frame in the image data.

[0123] According to one or more embodiments of the present disclosure, determining the first target posture of the terminal device when capturing the target image frame based on the first feature point in the target image frame and the second feature point in at least one other image frame in the image data includes: acquiring inertial sensor data corresponding to the image data, the inertial sensor data representing the motion state of the terminal device when each image frame in the image data is captured; merging the first feature point in the target image frame and the second feature point in the other image frame according to the inertial sensor data to obtain a fused feature point; and determining the first target posture based on the fused feature point.

[0124] According to one or more embodiments of the present disclosure, the merging of the first feature point in the target image frame and the second feature point in the other image frames to obtain a fused feature point based on the inertial sensor data includes: obtaining a relative posture corresponding to each image frame in the image data based on the inertial sensor data, the relative posture representing a posture change of the terminal device posture corresponding to the current image frame relative to the terminal device posture corresponding to the previously acquired adjacent image frame; performing coordinate transformation on the first feature point and the second feature point based on the first relative posture corresponding to the target image frame and the second relative posture corresponding to the other image frames to obtain a first orientation vector of the first feature point in the camera coordinate system of the terminal device and a second orientation vector of the second feature point in the camera coordinate system of the terminal device; and merging the first feature point and the second feature point in the camera coordinate system of the terminal device based on the first orientation vector and the second orientation vector to obtain the fused feature point.

[0125] According to one or more embodiments of the present disclosure, determining the first target pose based on the fused feature points includes: obtaining three-dimensional reference feature points corresponding to the reference image in the map database based on the fused feature points; obtaining a corresponding optimization matrix based on the optimization function corresponding to the reference feature points; and performing matrix decomposition on the optimization matrix to obtain the first target pose.

[0126] According to one or more embodiments of the present disclosure, the detecting the image data and determining the target image frame in the image data includes: performing feature extraction on the image frame in the image data to obtain a first image feature; performing feature comparison between the first image feature and a reference image in the map database to determine a target feature point in the image frame, wherein the feature similarity between the target feature point and the corresponding reference feature point in the reference image is greater than a similarity threshold; and determining the target image frame based on the number of the target feature points.

[0127] According to one or more embodiments of the present disclosure, at least one of the following items is also included: the acquisition interval between each of the image frames is not less than the first time length; the relative displacement between the acquisition positions corresponding to each of the image frames is not less than the first length.

[0128] According to one or more embodiments of the present disclosure, the acquiring of image data includes: collecting at least two frames of environmental images through the terminal device, and acquiring motion information corresponding to each of the environmental images, the motion information representing the motion state of the terminal device when shooting the environmental image; determining the relative displacement corresponding to the environmental image based on the motion information; and extracting the at least two frames of environmental images based on the relative displacement corresponding to each of the environmental images to generate the image data.

[0129] According to one or more embodiments of the present disclosure, the method further includes: obtaining a non-target image frame in the image data; solving the non-target image frame using a perspective n-point algorithm and a random consistency sampling algorithm to obtain a second target pose corresponding to the non-target image frame.

[0130] In a second aspect, according to one or more embodiments of the present disclosure, a posture detection device is provided, comprising:

[0131] an acquisition module configured to acquire image data, wherein the image data includes at least two image frames captured by a terminal device;

[0132] a processing module configured to detect the image data and determine a target image frame in the image data, wherein the number of first feature points corresponding to the target image frame is no greater than a preset number, and the first feature points are at least one set of pixel points in the image frame matched based on image features of a reference image in a map database;

[0133] The prediction module is configured to determine a first target pose of the terminal device when capturing the target image frame based on a first feature point in the target image frame and a second feature point in at least one other image frame in the image data.

[0134] According to one or more embodiments of the present disclosure, the prediction module is specifically configured to: obtain inertial sensor data corresponding to the image data, the inertial sensor data representing the motion state of the terminal device when each image frame in the image data is captured; based on the inertial sensor data, merge the first feature point in the target image frame and the second feature point in the other image frames to obtain a fused feature point; and determine the first target posture based on the fused feature point.

[0135] According to one or more embodiments of the present disclosure, when the prediction module merges the first feature point in the target image frame and the second feature point in the other image frames according to the inertial sensor data to obtain the fused feature point, the prediction module is specifically configured to: obtain the relative posture corresponding to each image frame in the image data according to the inertial sensor data, and the relative posture represents the posture change of the terminal device posture corresponding to the current image frame relative to the terminal device posture corresponding to the previously collected adjacent image frame; perform coordinate transformation on the first feature point and the second feature point according to the first relative posture corresponding to the target image frame and the second relative posture corresponding to the other image frames to obtain the first orientation vector of the first feature point in the camera coordinate system of the terminal device and the second orientation vector of the second feature point in the camera coordinate system of the terminal device; merge the first feature point and the second feature point in the camera coordinate system of the terminal device based on the first orientation vector and the second orientation vector to obtain the fused feature point.

[0136] According to one or more embodiments of the present disclosure, when the prediction module determines the first target posture based on the fused feature points, it is specifically configured to: obtain the three-dimensional reference feature points corresponding to the reference image in the map database based on the fused feature points; obtain the corresponding optimization matrix based on the optimization function corresponding to the reference feature points; and perform matrix decomposition on the optimization matrix to obtain the first target posture.

[0137] According to one or more embodiments of the present disclosure, the processing module is specifically configured to: perform feature extraction on the image frame in the image data to obtain a first image feature; perform feature comparison on the first image feature with a reference image in the map database to determine a target feature point in the image frame, wherein the feature similarity between the target feature point and the corresponding reference feature point in the reference image is greater than a similarity threshold; and determine the target image frame based on the number of the target feature points.

[0138] According to one or more embodiments of the present disclosure, at least one of the following items is also included: the acquisition interval between each of the image frames is not less than the first time length; the relative displacement between the acquisition positions corresponding to each of the image frames is not less than the first length.

[0139] According to one or more embodiments of the present disclosure, the acquisition module is specifically configured to: collect at least two frames of environmental images through the terminal device, and obtain motion information corresponding to each of the environmental images, wherein the motion information represents the motion state of the terminal device when shooting the environmental image; determine the relative displacement corresponding to the environmental image based on the motion information; and extract the at least two frames of environmental images based on the relative displacement corresponding to each of the environmental images to generate the image data.

[0140] According to one or more embodiments of the present disclosure, the square prediction module is further configured to: obtain a non-target image frame in the image data; and solve the non-target image frame using a perspective n-point algorithm and a random consistency sampling algorithm to obtain a second target pose corresponding to the non-target image frame.

[0141] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0142] The memory stores computer-executable instructions;

[0143] The processor executes the computer-executable instructions stored in the memory to implement the posture detection method described in the first aspect and various possible designs of the first aspect.

[0144] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the posture detection method described in the first aspect and various possible designs of the first aspect is implemented.

[0145] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the posture detection method described in the first aspect and various possible designs of the first aspect.

[0146] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0147] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0148] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A posture detection method, applied to a terminal device, comprising: Acquiring image data, wherein the image data includes at least two image frames captured by the terminal device; Detecting the image data to determine a target image frame in the image data, wherein the number of first feature points corresponding to the target image frame is not greater than a preset number, and the first feature points are at least one set of pixel points in the image frame matched to image features of a reference image in a map database; and A first target posture of the terminal device when acquiring the target image frame is determined based on a first feature point in the target image frame and a second feature point in at least one other image frame in the image data.

2. The method according to claim 1, wherein: The determining, based on the first feature point in the target image frame and the second feature point in at least one other image frame in the image data, a first target posture of the terminal device when acquiring the target image frame comprises: Acquire inertial sensor data corresponding to the image data, wherein the inertial sensor data represents a motion state of the terminal device when each image frame in the image data is collected; According to the inertial sensor data, merging the first feature point in the target image frame and the second feature point in the other image frames to obtain a fused feature point; and The first target posture is determined according to the fused feature points.

3. The method according to claim 2, wherein: The step of combining the first feature point in the target image frame and the second feature point in the other image frames according to the inertial sensor data to obtain a fused feature point includes: According to the inertial sensor data, a relative posture corresponding to each image frame in the image data is obtained, wherein the relative posture represents a posture change amount of the terminal device posture corresponding to the current image frame relative to the terminal device posture corresponding to the adjacent image frame collected previously; According to the first relative posture corresponding to the target image frame and the second relative posture corresponding to the other image frames, coordinate transformation is performed on the first feature point and the second feature point to obtain a first orientation vector of the first feature point in the camera coordinate system of the terminal device and a second orientation vector of the second feature point in the camera coordinate system of the terminal device. A second orientation vector in the camera coordinate system of the terminal device; and Based on the first orientation vector and the second orientation vector, the first feature point and the second feature point are merged in the camera coordinate system of the terminal device to obtain the fused feature point.

4. The method according to claim 2 or 3, wherein: The determining the first target posture according to the fused feature points includes: According to the fused feature points, obtaining three-dimensional reference feature points corresponding to the reference image in the map database; According to the optimization function corresponding to the reference feature point, a corresponding optimization matrix is ​​obtained; and Perform matrix decomposition on the optimization matrix to obtain the first target pose.

5. The method according to any one of claims 1 to 4, wherein: The detecting the image data to determine the target image frame in the image data includes: Extracting features from image frames in the image data to obtain first image features; Performing feature comparison of the first image feature with a reference image in the map database to determine a target feature point in the image frame, wherein a feature similarity between the target feature point and a corresponding reference feature point in the reference image is greater than a similarity threshold; and The target image frame is determined according to the number of the target feature points.

6. The method according to any one of claims 1 to 5, wherein: Also includes at least one of the following: The acquisition interval between each of the image frames is not less than the first duration; The relative displacement between the acquisition positions corresponding to the image frames is not less than the first length.

7. The method according to any one of claims 1 to 6, wherein: The acquiring of image data comprises: Collecting at least two frames of environmental images through the terminal device, and obtaining motion information corresponding to each of the environmental images, wherein the motion information represents the motion state of the terminal device when shooting the environmental images; Determining a relative displacement corresponding to the environment image according to the motion information; and Based on the relative displacement corresponding to each of the environmental images, the at least two frames of environmental images are extracted to generate the image data.

8. The method according to any one of claims 1 to 7, wherein: The method further comprises: Acquiring a non-target image frame in the image data; and For the non-target image frame, the perspective n-point algorithm and the random consistency sampling algorithm are used to perform Solve to obtain the second target pose corresponding to the non-target image frame.

9. A posture detection device, wherein: include: An acquisition module is configured to acquire image data, wherein the image data includes at least two image frames acquired by a terminal device; a processing module configured to detect the image data, determine a target image frame in the image data, the number of first feature points corresponding to the target image frame is not greater than a preset number, wherein the first feature point is at least one set of pixel points in the image frame matched to an image feature of a reference image in a map database; and The prediction module is configured to determine a first target posture of the terminal device when capturing the target image frame based on a first feature point in the target image frame and a second feature point in at least one other image frame in the image data.

10. An electronic device, wherein: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the posture detection method according to any one of claims 1 to 8.

11. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the posture detection method as described in any one of claims 1 to 8 is implemented.

12. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the posture detection method according to any one of claims 1 to 8 is implemented.

Images