Method and device for determining posture, electronic device and storage medium

Abstract

The present disclosure relates to a method and device for determining a posture, an electronic device, and a storage medium. The method comprises: obtaining collected data collected by a first terminal in a target scene; obtaining a global map including the target scene, wherein the global map is generated based on map data obtained by a second terminal collecting data from a global scene including the target scene, and the global map satisfies an accuracy condition; and determining at least one first posture of the first terminal during the collection process according to the feature correspondence between the collected data and the global map. Through the above process, a large amount of first posture data can be collected on a large scale, reducing the additional equipment settings for the target scene or the additional calibration synchronization between multiple devices, and the data of the first posture also has a high accuracy.

Description

[0001] This application is a divisional application of the Chinese patent application submitted to the China Patent Office on August 17, 2020, with application number 202010826704.X and invention name "Position determination method and device, electronic device and storage medium". Technical Field

[0002] The present disclosure relates to the field of computer vision technology, and in particular to a method and device for determining a posture, an electronic device, and a storage medium. Background Art

[0003] With the rapid development of mobile sensors, network infrastructure, and cloud computing, the scale of augmented reality application scenarios is expanding from small and medium-sized to large environments. Positioning in large-scale environments is a key requirement for augmented reality applications. Most commonly used positioning technologies require a large amount of motion truth data (such as the position data of the device during movement) and background information for positioning in order to perform algorithm benchmarking or model training. Therefore, how to obtain a large amount of motion truth data at a low cost has become a problem that needs to be solved urgently. Summary of the invention

[0004] The present disclosure proposes a technical solution for posture determination.

[0005] According to one aspect of the present disclosure, a method for determining a position and posture is provided, comprising:

[0006] Acquire collected data collected by a first terminal in a target scene; acquire a global map including the target scene, wherein the global map is generated based on map data obtained by a second terminal performing data collection on a global scene including the target scene, and the global map meets an accuracy condition; determine at least one first position of the first terminal during the collection process according to a feature correspondence relationship between the collected data and the global map.

[0007] In a possible implementation, the global map includes at least one frame of visual point cloud, and the visual point cloud includes at least one three-dimensional feature point in the global scene; the collected data includes a first collected image; and determining at least one first position of the first terminal in the collection process based on the feature correspondence between the collected data and the global map includes: performing feature matching on the first collected image and the at least one frame of visual point cloud to obtain a global feature matching result; and determining at least one first position of the first terminal in the collection process based on the global feature matching result.

[0008] In a possible implementation, the global map includes at least one frame of visual point cloud in the target scene; the collected data includes at least two frames of first collected images; and determining at least one first posture of the first terminal in the collection process based on the feature correspondence between the collected data and the global map includes: performing feature matching on the first collected image and the at least one frame of visual point cloud to obtain a global feature matching result; performing feature matching based on the at least two frames of first collected images to obtain a local feature matching result; and determining at least one first posture of the first terminal in the collection process based on the global feature matching result and the local feature matching result.

[0009] In a possible implementation, the collected data also includes first inertial measurement IMU data; determining at least one first posture of the first terminal in the collection process based on the global feature matching results and the local feature matching results includes: obtaining first constraint information based on the global feature matching results and / or the local feature matching results; obtaining second constraint information based on the first IMU data; processing the global feature matching results and the local feature matching results based on at least one of the first constraint information and the second constraint information to obtain at least one first posture of the first terminal in the collection process.

[0010] In a possible implementation manner, the processing the global feature matching result and the local feature matching result includes: processing the global feature matching result and the local feature matching result by bundle adjustment.

[0011] In a possible implementation, the feature matching of the first acquired image with the at least one frame of visual point cloud to obtain a global feature matching result includes: matching the two-dimensional feature points in the first acquired image with the three-dimensional feature points included in the at least one frame of visual point cloud to obtain a global feature matching result.

[0012] In a possible implementation, the method further includes: determining true motion data according to at least one first position of the first terminal during the acquisition process.

[0013] In a possible implementation, determining the motion true value data based on at least one first position of the first terminal during the acquisition process includes: taking at least one first position of the first terminal during the acquisition process as the motion true value data; and / or taking at least one of the acquired data and at least one first position of the first terminal during the acquisition process as the motion true value data, wherein the acquired data includes: one or more of wireless network WiFi data, Bluetooth data, geomagnetic data, ultra-wideband UWB data, a first acquired image and a first IMU data.

[0014] In a possible implementation, the motion truth data is used for at least one of the following operations: determining the accuracy of a positioning result, training a neural network, and fusing information with the global map.

[0015] In a possible implementation, the map data includes: a laser point cloud, a second acquired image, and a second IMU data in the global scene; the method further includes: acquiring map data of the global scene acquired by a second terminal; and based on the map data, performing offline reconstruction of the global scene to generate a global map of the global scene.

[0016] In a possible implementation, the global scene is reconstructed offline based on the map data to generate a global map of the global scene, including: determining at least one second posture of the second terminal during the data acquisition process based on the second IMU data and the laser point cloud; based on the at least one second posture, in combination with the second acquired image, performing a visual map reconstruction of the global scene to obtain at least one frame of visual point cloud, wherein the visual point cloud includes at least one three-dimensional feature point in the global scene; and obtaining a global map of the global scene based on the at least one frame of visual point cloud.

[0017] In a possible implementation, the visual map of the global scene is reconstructed according to the at least one second posture in combination with the second acquired image to obtain at least one frame of visual point cloud, including: according to the at least one second posture in combination with the second acquired image, the visual map of the global scene is reconstructed to obtain at least one frame of initial visual point cloud; according to the laser point cloud and / or the second acquired image, third constraint information in the visual map reconstruction process is obtained; according to the third constraint information, the at least one frame of initial visual point cloud is optimized to obtain at least one frame of visual point cloud.

[0018] In a possible implementation, the second terminal includes: a radar for acquiring a laser point cloud in the global scene; a visual sensor for acquiring a second collected image in the global scene; and an IMU sensor for acquiring second IMU data in the global scene.

[0019] In a possible implementation, before the global scene is offline reconstructed based on the map data to generate a global map of the global scene, the method further includes: calibrating the coordinate transformation relationship between the visual sensor and the IMU sensor to obtain a first calibration result; calibrating the coordinate transformation relationship between the radar and the visual sensor to obtain a second calibration result; and jointly calibrating the coordinate transformation relationship between the visual sensor, the IMU sensor and the radar based on the first calibration result and the second calibration result.

[0020] In a possible implementation, the method further includes: in the process of collecting map data at the second terminal, reconstructing the global scene in real time according to the map data to generate a real-time map of the global scene; sending the map data and / or the real-time map to a target device, wherein the target device is used to display the geographical scope of the data collection completed for the global scene.

[0021] According to one aspect of the present disclosure, there is provided a posture determination device, comprising:

[0022] A collection data acquisition module is used to acquire collection data collected by a first terminal in a target scene; a global map acquisition module is used to acquire a global map including the target scene, wherein the global map is generated based on map data obtained by a second terminal performing data collection on a global scene including the target scene, and the global map meets an accuracy condition; a posture determination module is used to determine at least one first posture of the first terminal during the collection process according to a feature correspondence between the collection data and the global map.

[0023] In a possible implementation, the global map includes at least one frame of visual point cloud, and the visual point cloud includes at least one three-dimensional feature point in the global scene; the collected data includes a first collected image; the posture determination module is used to: perform feature matching on the first collected image and the at least one frame of visual point cloud to obtain a global feature matching result; and determine at least one first posture of the first terminal during the collection process based on the global feature matching result.

[0024] In a possible implementation, the global map includes at least one frame of visual point cloud in the target scene; the collected data includes at least two frames of first collected images; the posture determination module is used to: perform feature matching on the first collected image and the at least one frame of visual point cloud to obtain a global feature matching result; perform feature matching based on the at least two frames of first collected images to obtain a local feature matching result; and determine at least one first posture of the first terminal during the collection process based on the global feature matching result and the local feature matching result.

[0025] In a possible implementation, the collected data also includes first inertial measurement IMU data; the posture determination module is further used to: obtain first constraint information based on the global feature matching result and / or the local feature matching result; obtain second constraint information based on the first IMU data; and process the global feature matching result and the local feature matching result based on at least one of the first constraint information and the second constraint information to obtain at least one first posture of the first terminal during the collection process.

[0026] In a possible implementation, the posture determination module is further used to: process the global feature matching results and the local feature matching results through bundle adjustment.

[0027] In a possible implementation, the posture determination module is further used to: match the two-dimensional feature points in the first acquired image with the three-dimensional feature points included in the at least one frame of visual point cloud to obtain a global feature matching result.

[0028] In a possible implementation, the device further includes: a motion true value data acquisition module, configured to determine motion true value data according to at least one first position of the first terminal during an acquisition process.

[0029] In one possible implementation, the motion true value data acquisition module is used to: use at least one first position of the first terminal during the acquisition process as the motion true value data; and / or use at least one of the acquired data and at least one first position of the first terminal during the acquisition process as the motion true value data, wherein the acquired data includes: one or more of wireless network WiFi data, Bluetooth data, geomagnetic data, ultra-wideband UWB data, a first acquired image and a first IMU data.

[0030] In a possible implementation, the motion truth data is used for at least one of the following operations: determining the accuracy of a positioning result, training a neural network, and fusing information with the global map.

[0031] In a possible implementation, the map data includes: a laser point cloud, a second acquired image, and a second IMU data in the global scene; the device also includes: a map data acquisition module, used to acquire the map data of the global scene acquired by the second terminal; a global map generation module, used to perform offline reconstruction of the global scene according to the map data to generate a global map of the global scene.

[0032] In one possible implementation, the global map generation module is used to: determine at least one second posture of the second terminal during the data acquisition process based on the second IMU data and the laser point cloud; reconstruct a visual map of the global scene based on the at least one second posture and in combination with the second acquired image to obtain at least one frame of visual point cloud, wherein the visual point cloud includes at least one three-dimensional feature point in the global scene; and obtain a global map of the global scene based on the at least one frame of visual point cloud.

[0033] In a possible implementation, the global map generation module is further used to: reconstruct a visual map of the global scene according to the at least one second posture in combination with the second acquired image to obtain at least one frame of initial visual point cloud; obtain third constraint information in the visual map reconstruction process according to the laser point cloud and / or the second acquired image; and optimize the at least one frame of initial visual point cloud according to the third constraint information to obtain at least one frame of visual point cloud.

[0034] In a possible implementation, the second terminal includes: a radar for acquiring a laser point cloud in the global scene; a visual sensor for acquiring a second collected image in the global scene; and an IMU sensor for acquiring second IMU data in the global scene.

[0035] In a possible implementation, the device is also used to: calibrate the coordinate transformation relationship between the visual sensor and the IMU sensor to obtain a first calibration result; calibrate the coordinate transformation relationship between the radar and the visual sensor to obtain a second calibration result; and jointly calibrate the coordinate transformation relationship between the visual sensor, the IMU sensor and the radar based on the first calibration result and the second calibration result.

[0036] In a possible implementation, the device is also used to: during the process of collecting map data at the second terminal, reconstruct the global scene in real time based on the map data to generate a real-time map of the global scene; send the map data and / or the real-time map to a target device, wherein the target device is used to display the geographical scope of the data collection completed for the global scene.

[0037] According to one aspect of the present disclosure, there is provided an electronic device, including:

[0038] A processor; a memory for storing processor executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above-mentioned posture determination method.

[0039] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above-mentioned posture determination method is implemented.

[0040] In the disclosed embodiment, at least one first pose of the first terminal in the collection process is determined by obtaining the collected data collected by the first terminal in the target scene, and obtaining the global map containing the target scene, and according to the feature correspondence between the collected data and the global map. Through the above process, the global map of the global scene can be reused, and a large amount of first pose data can be collected on a large scale through the first terminal after the global map is generated; and the method of obtaining the collected data for generating the first pose is also relatively simple, and the collection can be achieved only through the first terminal, reducing the additional equipment settings for the target scene or the additional calibration synchronization between multiple devices; in addition, because the global map meets the accuracy conditions, the first pose data obtained based on the feature correspondence between the collected data and the global map also has high accuracy.

[0041] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

[0042] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0044] Figure 1 A flowchart of a method for determining a posture according to an embodiment of the present disclosure is shown.

[0045] Figure 2 A schematic diagram showing the comparison of visual point cloud before and after optimization according to an embodiment of the present disclosure is shown.

[0046] Figure 3 A schematic structural diagram of a second terminal according to an embodiment of the present disclosure is shown.

[0047] Figure 4 A schematic diagram showing an application example according to the present disclosure.

[0048] Figure 5 A block diagram of a posture determination device according to an embodiment of the present disclosure is shown.

[0049] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0050] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0051] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0052] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0053] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0054] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0055] Figure 1 A flow chart of a method for determining a position and posture according to an embodiment of the present disclosure is shown, and the method can be applied to a position and posture determining device, and the position determining device can be a terminal device, a server or other processing device, etc. Among them, the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc.

[0056] In some possible implementations, the posture determination method may be implemented by a processor calling computer-readable instructions stored in a memory.

[0057] like Figure 1 As shown, the posture determination method may include:

[0058] Step S11, acquiring collected data collected by a first terminal in a target scene.

[0059] Step S12, obtaining a global map including the target scene, wherein the global map is generated based on map data obtained by collecting data of the global scene including the target scene by the second terminal, and the global map meets the accuracy condition.

[0060] Step S13: determining at least one first position of the first terminal during the collection process according to the feature correspondence between the collected data and the global map.

[0061] Among them, the target scene can be any scene in which the first terminal obtains the collected data, and its implementation form can be flexibly determined according to actual needs, and is not limited in the embodiments of the present disclosure. In one possible implementation, the target scene may include an outdoor scene, such as a square, a street, or an open space; in one possible implementation, the target scene may include an indoor scene, such as a classroom, an office building, or a residential building; in one possible implementation, the target scene may include both an outdoor scene and an indoor scene.

[0062] The first terminal may be a mobile terminal with a data collection function. The implementation of the first terminal is not limited in the embodiments of the present disclosure. Any device with a mobile and data collection function may be used as an implementation of the first terminal. In one possible implementation, the first terminal may be an augmented reality (AR) device, such as a mobile phone or AR glasses.

[0063] The collected data may be data collected by the first terminal in the target scene. The implementation form of the collected data and the data content contained therein may be flexibly determined according to the data collection method of the first terminal and the actual implementation form of the first terminal, and is not limited to the following disclosed embodiments. In one possible implementation, when the first terminal is an AR device, the collected data may include a first collected image obtained by the first terminal performing image collection on the target scene, etc.; in one possible implementation, when the first terminal is an AR device, the collected data may also include a first IMU data obtained by an inertial measurement unit (IMU) in the first terminal collecting data on the target scene, etc.

[0064] The method for the first terminal to obtain the collected data is also not limited in the embodiment of the present disclosure, and can be flexibly selected according to the actual situation of the first terminal and the target scene. In a possible implementation, the first terminal can collect the collected data by moving in the target scene. The specific movement process and method can be flexibly selected according to the actual situation.

[0065] In step S11, the method of obtaining the collected data is not limited in the embodiment of the present disclosure. In a possible implementation method, the collected data can be obtained by reading the collected data from the first terminal, or receiving the collected data transmitted by the first terminal. In a possible implementation method, the posture determination method proposed in the embodiment of the present disclosure can also be applied to the first terminal. In this case, the collected data collected by the first terminal when moving in the target scene can be directly obtained.

[0066] It can be seen from the above-mentioned disclosed embodiments that in a possible implementation method, the disclosed embodiments can also obtain a global map including the target scene through step S12, wherein the global map can be generated based on map data and meet the accuracy conditions, and the map data can be obtained by collecting data of the global scene of the target scene through the second terminal.

[0067] Among them, the global scene can be any scene that includes the target scene, and the range of scenes included can be flexibly determined according to actual conditions, and is not limited in the embodiments disclosed herein. As described in the above disclosed embodiments, the target scene can include outdoor scenes and / or indoor scenes. Therefore, the global scene can also include outdoor scenes and / or indoor scenes according to the actual conditions of the target scene. For example, in one possible implementation, when the target scene is an outdoor scene including an open space or a square, the global scene can be a scene in the district or urban area where the open space or the square is located, and the global scene can include both outdoor scenes in the district or urban area and indoor scenes in the district or urban area.

[0068] The map data may be the corresponding data obtained by the second terminal through data collection of the global scene, and the map data may be used to generate a global map corresponding to the global scene. The data content contained in the map data may be flexibly determined according to actual needs. In one possible implementation, the map data may include a second captured image obtained by performing image collection on the global scene; in one possible implementation, the map data may include second IMU data obtained by performing IMU data collection on the global scene; in one possible implementation, the map data may also include laser point cloud data obtained by performing radar scanning on the global scene, etc. The specific implementation form of the map data may refer to the subsequent disclosed embodiments, which will not be expanded here.

[0069] The second terminal may be a mobile terminal with a map data acquisition function, and the implementation method of the second terminal may be flexibly selected according to the data content that the map data needs to contain. For example, when the map data contains a second acquired image, the second terminal may include a visual sensor for image acquisition, when the map data contains a second IMU data, the second terminal may include an IMU sensor for collecting IMU data, and when the map data contains laser point cloud data, the second terminal may include a radar for collecting laser point cloud, etc. The hardware structure and connection method included in the second terminal can also be found in the subsequent disclosed embodiments, which will not be expanded here.

[0070] The global map can be a map generated based on map data, and its implementation form can be determined based on the actual situation of the global scene and the data content of the map data. In a possible implementation, the global map can include relevant information of each three-dimensional feature point in the global scene. What information is included and how this information is reflected can be flexibly determined according to the actual situation. In a possible implementation, the global map can include relevant information of each three-dimensional feature point in the global scene. In a possible implementation, the three-dimensional feature points in the global scene can be observed in the form of an image, and the information content included in the relevant information of the three-dimensional feature points can be flexibly determined according to the actual situation, such as the coordinates of the three-dimensional feature points and the feature information of the three-dimensional feature points, wherein the feature information of the three-dimensional feature points can include feature descriptors corresponding to the three-dimensional feature points, communication signal fingerprints corresponding to the three-dimensional feature points, or one or more of the semantic information and other feature-related information. What specific relevant information is included, how these relevant information are included in the global map, and what the specific meanings of the above-mentioned relevant information are, can be found in the subsequent public embodiments, which will not be expanded here.

[0071] The accuracy of the global map can be the position accuracy of each 3D feature point in the global map, for example, it can be the position difference between the coordinates of the 3D feature point contained in the global map and the actual position of the 3D feature point in the global scene. Therefore, the accuracy condition of the global map can reflect whether the position of each 3D feature point in the global map meets the accuracy requirements, and the specific value of the accuracy condition can be flexibly set according to the actual situation.

[0072] In a possible implementation, it may be difficult to directly determine the position difference between the coordinates of the three-dimensional feature points in the global map and their actual positions. Therefore, it is possible to indirectly determine whether the global map meets the accuracy conditions by determining whether the amount of map data collected reaches a certain data value or whether the accuracy of the method for generating the global map meets the requirements. For example, in one example, it is possible to indirectly infer whether the global map meets the accuracy conditions by determining whether the ratio between the geographical range corresponding to the collected map data and the geographical range covered by the global scene reaches a preset threshold.

[0073] The method of obtaining the global map can be flexibly selected according to the actual situation and is not limited to the following disclosed embodiments. In one possible implementation, the global map can be generated in the position determination device according to the map data by obtaining the map data collected by the second terminal; in one possible implementation, the global map can also be generated in other devices or equipment. In this case, the method of obtaining the global map can be to read the global map directly from the device that stores or generates the global map.

[0074] The process of the second terminal collecting data from the global scene to obtain map data is not limited in the present disclosed embodiment and can be flexibly determined according to actual conditions. In a possible implementation, the second terminal can move in the global scene to collect corresponding map data. For details on how to move and how to collect map data, please refer to the subsequent disclosed embodiments, which will not be expanded here.

[0075] Furthermore, how to generate a global map of a global scene based on the map data, this generation process can be flexibly determined according to the data contained in the map data. Please refer to the subsequent disclosed embodiments for details, which will not be expanded here.

[0076] In the above disclosed embodiment, the implementation order of step S11 and step S12 is not limited in the disclosed embodiment. In one possible implementation, step S11 and step S12 can be executed sequentially in a certain order. In one possible implementation, step S11 and step S12 can also be executed simultaneously.

[0077] After acquiring the collected data in the target scene and the global map including the target scene, at least one first position of the first terminal in the collection process can be determined according to the feature correspondence between the collected data and the global map.

[0078] As described in the above-mentioned disclosed embodiments, the collected data may be data obtained by collecting the target scene, which may reflect the characteristics of the target scene, and the global scene corresponding to the global map may also include the characteristics of the target scene because it includes the target scene. Therefore, in a possible implementation, the feature correspondence between the collected data and the global map may include the feature correspondence between the collected data and the global map. In addition, since the first terminal can collect a large amount of collected data when moving in the target scene, the collected data may also reflect the characteristics of the target scene. Therefore, in a possible implementation, the feature correspondence between the collected data and the global map may also include the feature correspondence between the data contained in the collected data. Which feature relationships are used to determine at least one first posture can be flexibly selected according to actual conditions. Please refer to the subsequent disclosed embodiments for details, which will not be expanded here.

[0079] The first pose may be one or more poses generated when the first terminal collects data during the movement of the target scene, and the number of first poses determined may be flexibly determined according to the actual situation. In one possible implementation, the first pose may correspond to the collected data, that is, each determined first pose may be a pose generated by the first terminal when each collected data is collected. How to determine the first pose based on the feature correspondence can also be flexibly selected according to the actual situation. Please refer to the subsequent public embodiments for details, which will not be expanded here.

[0080] In the disclosed embodiment, at least one first pose of the first terminal in the collection process is determined by obtaining the collected data collected by the first terminal in the target scene, and obtaining the global map containing the target scene, and according to the feature correspondence between the collected data and the global map. Through the above process, the global map of the global scene can be reused, and a large amount of first pose data can be collected on a large scale through the first terminal after the global map is generated; and the method of obtaining the collected data for generating the first pose is also relatively simple, and the collection can be achieved only through the first terminal, reducing the additional equipment settings for the target scene or the additional calibration synchronization between multiple devices; in addition, because the global map meets the accuracy conditions, the first pose data obtained based on the feature correspondence between the collected data and the global map also has high accuracy.

[0081] As described in the above disclosed embodiments, the implementation form of the map data can be flexibly determined according to the actual situation, and the method of generating the global map based on the map data can be flexibly determined according to the actual situation of the map data. Therefore, in a possible implementation, the map data may include: a laser point cloud in the global scene, a second collected image, and a second IMU data;

[0082] The posture determination method proposed in the embodiment of the present disclosure also includes:

[0083] Acquire map data of the global scene collected by the second terminal;

[0084] According to the map data, the global scene is reconstructed offline to generate a global map of the global scene.

[0085] Among them, the laser point cloud can be a point cloud composed of multiple laser points obtained by the second terminal performing a radar scan on the global scene. The number of laser points included in the laser point cloud can be flexibly determined based on the radar scanning situation of the second terminal and the movement trajectory of the second terminal in the global scene, and is not limited in the present disclosed embodiment.

[0086] The second captured images may be a plurality of images captured by the second terminal during movement in the global scene. The number of second captured images may be determined based on the movement of the second terminal in the global scene and the number of hardware devices for capturing images included in the second terminal, and is not limited in the present disclosed embodiment.

[0087] The second IMU data can be the relevant inertial measurement data collected by the second terminal during the movement of the second terminal in the global scene. The amount of the second IMU data can also be determined based on the movement of the second terminal in the global scene and the number of hardware devices for collecting IMU data contained in the second terminal, which is not limited in the present disclosed embodiment.

[0088] It can be seen from the above-mentioned disclosed embodiments that in a possible implementation method, the posture determination method proposed in the disclosed embodiments may also include offline reconstruction of the global scene based on the map data of the global scene acquired by the second terminal, so as to generate a global map of the global scene.

[0089] The offline reconstruction may be a process of reconstructing the global map of the global scene based on the collected map data after the second data completes the collection of the map data in the global scene. How to perform offline reconstruction of the global scene based on the map data including the laser point cloud, the second collected image, and the second IMU data can be flexibly determined according to the actual situation. For details, please refer to the subsequent disclosed embodiments, which will not be expanded here.

[0090] In the embodiment of the present disclosure, map data including laser point cloud, second collected image and second IMU data are obtained, and the global scene is reconstructed offline based on the obtained map data to generate a global map of the global scene. Through the above process, after completing a relatively comprehensive map data collection of the global scene, a large amount of map data collected can be integrated to comprehensively reconstruct the global scene offline, so that the generated global map has a higher accuracy, and then the result of at least one first pose determined based on the global map and the collected data is more accurate; at the same time, because the map data includes laser point cloud, second collected image and second IMU data, the acquisition method of these data is relatively easy and the acquisition process is less restricted by space. Therefore, the posture determination method proposed in the embodiment of the present disclosure has less difficulty in obtaining map data and global maps, thereby greatly improving the application scope of the posture determination method proposed in the embodiment of the present disclosure.

[0091] As described in the above disclosed embodiments, the process of offline reconstruction can be flexibly determined according to actual conditions. In a possible implementation, the global scene is offline reconstructed based on the map data to generate a global map of the global scene, including:

[0092] Determine at least one second posture of the second terminal during the data collection process according to the second IMU data and the laser point cloud;

[0093] Reconstructing a visual map of the global scene according to the at least one second posture in combination with the second acquired image to obtain at least one frame of visual point cloud, wherein the visual point cloud includes at least one three-dimensional feature point in the global scene;

[0094] A global map of the global scene is obtained according to at least one frame of visual point cloud.

[0095] Among them, according to the second IMU data and the laser point cloud, the method of determining at least one second posture of the second terminal during the data collection process can be flexibly determined according to the actual situation. Any method that can restore the posture of the second terminal based on the laser point cloud and IMU data can be used as an implementation method in the embodiments of the present disclosure, and is not limited to the following disclosed embodiments. In one possible implementation method, according to the second IMU data, at different moments in the data collection process of the second terminal, the acquired laser points can be projected onto the laser radar frame at that moment, so that the second posture of the second terminal at different moments in the data collection process can be estimated based on the projection results of the laser points.

[0096] After determining at least one second posture of the second terminal during the data collection process, the visual map of the global scene can be reconstructed according to the at least one second posture and in combination with the second collected image to obtain at least one frame of visual point cloud. The visual point cloud may include at least one three-dimensional feature point in the global scene. The number of visual point clouds and the number of three-dimensional feature points included are not limited in the embodiments of the present disclosure. In one possible implementation, the global map may include one or more frames of visual point clouds. As described in the above-mentioned disclosed embodiments, the global map may include relevant information of each three-dimensional feature point in the global scene, and in one possible implementation, the three-dimensional feature points in the global scene can be observed in the form of an image. Therefore, in one possible implementation, the visual point cloud can be observed through a visual image. In this case, the global map may also include at least one or more frames of visual images for observing the visual point cloud.

[0097] Furthermore, the feature information of the three-dimensional feature points included in the visual point cloud can also be stored in the global map, so the visual point cloud can also correspond to the feature information of the three-dimensional feature points. For example, in a possible implementation, the feature descriptor of the three-dimensional feature point can be determined based on the features extracted from the second acquired image, so the visual point cloud can correspond to the feature descriptor of the three-dimensional feature point; in a possible implementation, the acquired map data can also include communication-related signal data, such as WiFi signals, Bluetooth signals, or ultra-wideband (UWB, Ultra WideBand) signals, etc. These signals can be used as signal fingerprints to correspond to the three-dimensional feature points, thereby serving as feature information of the three-dimensional feature points, so the visual point cloud can correspond to the communication signal fingerprints of the three-dimensional feature points; in a possible implementation, the second acquired image can also include some semantic information, and the semantic information can also establish a correspondence with the three-dimensional feature points, thereby serving as feature information of the three-dimensional feature points. In this case, the visual point cloud can establish a correspondence with the semantic information, etc.

[0098] The process of reconstructing a visual map of a global scene based on at least one second posture and in combination with a second acquired image can be implemented through relevant technologies. The specific method to be adopted can be flexibly selected according to the actual situation and is not limited to the following disclosed embodiments. In one possible implementation, the second acquired image can be feature extracted and matched by a scale-invariant feature transform (SIFT) to generate at least one frame of visual point cloud. Furthermore, based on at least one second posture determined by the laser point cloud and the second IMU data, the coordinates of each three-dimensional feature point observed in at least one frame of visual point cloud can be further obtained.

[0099] After obtaining at least one frame of visual point cloud through visual map reconstruction, a global map of the global scene can be obtained based on the at least one frame of visual point cloud. In a possible implementation, all the obtained visual point clouds and the feature information of the three-dimensional feature points corresponding to these visual point clouds can be used as a global map; in a possible implementation, one or more frames can be selected from the obtained visual point cloud, and the feature information of the three-dimensional feature points corresponding to the one or more frames of visual point cloud can be used as a global map.

[0100] In the embodiment of the present disclosure, at least one second posture of the second terminal during the data collection process is determined based on the second IMU data and the laser point cloud, and a visual map of the global scene is reconstructed based on the at least one second posture in combination with the second collected image to obtain at least one frame of visual point cloud, thereby obtaining a global map of the global scene based on the at least one frame of visual point cloud; through the above process, the laser point cloud, the second IMU data and the second collected image can be comprehensively utilized, and the position and characteristics of each three-dimensional feature point in the global scene can be represented by the visual point cloud. The reconstruction of the global map can be achieved using data that is easier to obtain, and the reconstruction result is more accurate, which improves the convenience and determination accuracy of the entire posture determination process.

[0101] In a possible implementation, reconstructing a visual map of the global scene according to the at least one second posture in combination with the second acquired image to obtain at least one frame of visual point cloud includes:

[0102] Reconstructing a visual map of the global scene according to the at least one second posture and in combination with the second acquired image to obtain at least one frame of initial visual point cloud;

[0103] Acquire third constraint information in a visual map reconstruction process according to the laser point cloud and / or the second acquired image;

[0104] At least one frame of initial visual point cloud is optimized according to the third constraint information to obtain at least one frame of visual point cloud.

[0105] In one possible implementation, the accuracy of the second pose determined according to the laser point cloud may be low. In this case, the visual point cloud obtained by directly using the determined second pose in combination with the second acquired image to reconstruct the visual map may contain a large amount of noise. Therefore, in one possible implementation, after reconstructing the visual map of the global scene according to the second pose and the second acquired image, the image obtained by visual map reconstruction can be used as the initial visual point cloud, and the initial visual point cloud can be further optimized according to the third constraint information generated by the laser point cloud and / or the second acquired image, thereby reducing the noise in the initial visual point cloud to obtain a visual point cloud with higher accuracy.

[0106] Among them, the process of reconstructing the visual map according to the second posture and the second acquired image to obtain at least one frame of initial visual point cloud can be referred to the above-mentioned disclosed embodiment and will not be repeated here.

[0107] The third constraint information may be constraint information determined based on the laser point cloud and / or the second acquired image. The specific method of obtaining the third constraint information based on the laser point cloud and / or the second acquired image may be flexibly selected according to actual conditions and is not limited to the following disclosed embodiments.

[0108] In a possible implementation, obtaining the third constraint information in the process of visual map reconstruction according to the laser point cloud may include:

[0109] Through the real-time laser odometer and LOAM (Lidar Odometry and Mapping in real-time) method, the laser point cloud is feature extracted to determine the plane features and edge features of the laser point cloud;

[0110] According to the plane features of the laser point cloud, the plane constraints of the laser point cloud in the process of visual map reconstruction are determined;

[0111] According to the edge features of the laser point cloud, the edge constraints of the laser point cloud in the process of visual map reconstruction are determined;

[0112] According to the plane constraint of the laser point cloud and / or the edge constraint of the laser point cloud, the third constraint information in the visual map reconstruction process is obtained.

[0113] The plane features of the laser point cloud can be flexibly determined according to the actual situation of the laser point cloud. The specific form of the plane constraint determined based on the plane features of the laser point cloud can be flexibly selected according to the actual situation. In an example, the plane constraint can be expressed by the following formula (1):

[0114]

[0115] Among them, n and m are two different laser point cloud coordinate systems. m n is the characteristic point in the coordinate system m m The plane eigennormal vector at q, m n T for m The transpose of n, is the transformation relationship between coordinate systems n and m, n p is the characteristic point in the coordinate system n, m q is the characteristic point in the coordinate system m, Based on This coordinate transformation relationship is n The coordinate transformation performed by p, ∑p is the covariance matrix of the plane features of the laser point cloud, where ∑ p The value of can be flexibly set according to the actual situation. In an example, ∑ p Can be set to 0.2㎡.

[0116] Similarly, the edge features of the laser point cloud can also be flexibly determined according to the actual situation of the laser point cloud. The specific form of the edge constraint determined based on the edge features of the laser point cloud can be flexibly selected according to the actual situation. In an example, the edge constraint can be expressed by the following formula (2):

[0117]

[0118] in, m I is the characteristic point in the coordinate system m m The edge feature direction vector at q, ∑ e is the covariance matrix of the edge feature of the laser point cloud, and the meanings of the other parameters can refer to the above embodiment, where ∑ e The value of can be flexibly set according to the actual situation. In an example, ∑ e Can be set to 0.5㎡.

[0119] After respectively determining the plane constraint of the laser point cloud and the edge constraint of the laser point cloud, both the plane constraint and the edge constraint can be used as the third constraint information, or one of the plane constraint or the edge constraint can be used as the third constraint information. The specific selection can be flexibly determined according to actual conditions.

[0120] In a possible implementation, obtaining the third constraint information in the visual map reconstruction process according to the second acquired image may include:

[0121] Projecting the three-dimensional feature points corresponding to the initial visual point cloud onto the initial visual point cloud to obtain a projection result;

[0122] Obtain visual constraints in the visual map reconstruction process according to the error between the projection result and the two-dimensional feature points in the initial visual point cloud, where the two-dimensional feature points are two-dimensional feature points in the initial visual point cloud corresponding to the three-dimensional feature points;

[0123] According to the visual constraint, third constraint information in the visual map reconstruction process is obtained.

[0124] According to the error between the projection result and the two-dimensional feature point corresponding to the three-dimensional feature point in the initial visual point cloud, the specific process of obtaining the visual constraint in the visual map reconstruction process can be flexibly selected according to the actual situation. In an example, the visual constraint can be expressed by the following formula (3):

[0125]

[0126] Among them, X j is the jth 3D feature point corresponding to the visual point cloud, x ij is the 3D feature point X in the initial visual point cloud of the i-th frame j The corresponding two-dimensional feature point, f( W T i ,X j ) is the three-dimensional feature point X j Projection result of the initial visual point cloud projected to the i-th frame, ∑ v is the covariance matrix of the image feature constraints, where ∑ v The value of can be flexibly set according to the actual situation. In one example, ∑ v Can be set to 2 pixels square.

[0127] The specific information included in the third constraint information can be flexibly selected according to the actual situation. It can be seen from the above-mentioned disclosed embodiments that the third constraint information can include one or more of the plane constraint of the laser point cloud, the edge constraint of the laser point cloud, and the visual constraint. In one example, the third constraint information can simultaneously include the plane constraint of the laser point cloud, the edge constraint of the laser point cloud, and the visual constraint. In this case, according to the third constraint information, at least one frame of initial visual point cloud is optimized, and the process of obtaining at least one frame of visual point cloud can be expressed by the following formula (4):

[0128]

[0129] Among them, L p is the point cloud composed of plane points in the laser point cloud, L'p is L p The collection of L e is the point cloud composed of the points belonging to the edge in the laser point cloud, L'e is L e The meanings of the remaining parameters can refer to the above disclosed embodiments.

[0130] It can be seen from formula (4) that in a possible implementation, at least one frame of initial visual point cloud is optimized according to the third constraint information, which may include optimizing the three-dimensional feature points included in the initial visual point cloud, and may also include optimizing the posture of the device for collecting the second acquired image in the second terminal. When optimizing the posture of the device for collecting the second acquired image in the second terminal, the second posture corresponding to the second terminal may also be optimized accordingly, thereby reducing the situation where the visual point cloud contains large noise due to the low accuracy of the second posture. Furthermore, after optimizing the visual point cloud, the third constraint information of the visual map reconstruction process may be obtained again based on the optimization result of the visual point cloud, so as to further iterate and optimize the visual point cloud. The number of iterations may be flexibly selected according to the actual situation, and is not limited in the embodiment of the present disclosure. Figure 2 A schematic diagram showing the comparison of the visual point cloud before and after optimization according to an embodiment of the present disclosure is shown, wherein for the same scene, the upper box shows the visual image corresponding to the visual point cloud before optimization, and the lower box shows the visual image corresponding to the visual point cloud after optimization. It can be seen from the figure that the noise points in the visual point cloud are reduced after optimization, so the optimized visual point cloud has higher accuracy, and accordingly, the accuracy of the three-dimensional feature points corresponding to the optimized visual point cloud is also improved.

[0131] As described in the above disclosed embodiments, the map data collected by the second terminal may include the second collected image, the second IMU data and the laser point cloud. Accordingly, the second terminal may also include a hardware structure having the above data collection function. Therefore, in a possible implementation, the second terminal may include:

[0132] Radar, used to obtain laser point clouds in the global scene;

[0133] A visual sensor, used for acquiring a second collected image in a global scene;

[0134] The IMU sensor is used to obtain the second IMU data in the global scene.

[0135] Among them, the radar can be any radar with a laser point cloud acquisition function, and its implementation form is not limited in any way in the embodiments of the present disclosure. In one possible implementation, the radar can be a 3D radar. The visual sensor can be any sensor with an image acquisition function, such as a camera, and its specific implementation form can also be flexibly determined. In one possible implementation, the second terminal can simultaneously include a 4-array camera with a 360° image acquisition function. The implementation form of the IMU sensor can also be flexibly determined according to actual conditions and is not limited in the embodiments of the present disclosure.

[0136] The setting positions and connection relationships between the radar, visual sensor and IMU sensor in the second terminal can be flexibly selected according to actual conditions, and are not limited to the following disclosed embodiments. In one possible implementation, the radar, visual sensor and IMU sensor can be rigidly connected, and the specific connection sequence can be flexibly selected according to actual conditions. In one possible implementation, the visual sensor and the IMU sensor can be fixedly connected and packaged as a fixed structural unit, and the radar can be set above the fixed structural unit. In one possible implementation, the visual sensor, IMU sensor and radar can also be fixedly set in a backpack, Figure 3 A structural schematic diagram of the second terminal according to an embodiment of the present disclosure is shown. It can be seen from the figure that in one example, the visual sensor and the IMU sensor can be fixedly connected and packaged into a fixed structural unit, the lower end of the fixed structural unit can be set in a backpack for easy carrying, and the radar can be set above the fixed structural unit.

[0137] Through the second terminal including radar, visual sensor and IMU sensor, the second terminal can be used to comprehensively collect map data in the global scene, so as to facilitate the subsequent generation of the global map. Through the above structure, relatively simple and low-cost hardware equipment can be used to realize the generation of the global map, so as to further obtain multiple first poses of the first terminal based on the global map, which greatly reduces the hardware implementation cost and difficulty of obtaining the first pose data.

[0138] Since the second terminal may include hardware devices such as radar, visual sensors, and IMU sensors, these hardware devices may need to be calibrated or the measurement data time may be calibrated before use. Furthermore, while calibrating each hardware, the coordinate transformation relationship between different hardware may also be calibrated to improve the accuracy of the generated global map. Therefore, in a possible implementation, before the global scene is reconstructed offline according to the map data and the global map of the global scene is generated, it may also include:

[0139] Calibrate the coordinate transformation relationship between the visual sensor and the IMU sensor to obtain a first calibration result;

[0140] Calibrate the coordinate transformation relationship between the radar and the visual sensor to obtain a second calibration result;

[0141] According to the first calibration result and the second calibration result, the coordinate transformation relationship among the visual sensor, the IMU sensor and the radar is jointly calibrated.

[0142] Among them, the method of calibrating the coordinate transformation relationship between the visual sensor and the IMU sensor can be flexibly selected according to the actual situation. In one example, the calibration of the visual sensor and the IMU sensor can be realized through the Kalibr tool; the method of calibrating the coordinate transformation relationship between the radar and the visual sensor can also be flexibly selected according to the actual situation. In one example, the calibration of the radar and the visual sensor can be realized through the AutoWare framework. Furthermore, since there may be errors in the calibration process, in a possible implementation method, the coordinate transformation relationship between the visual sensor, the IMU sensor and the radar can be jointly calibrated and optimized based on the first calibration result and the second calibration result to make the coordinate transformation relationship between different hardware devices more accurate.

[0143] The implementation form of the joint calibration can be flexibly determined according to actual conditions and is not limited to the following disclosed embodiments. In a possible implementation, the joint calibration can be expressed by the following formula (5):

[0144]

[0145] Among them, C i is the i-th visual sensor in the second terminal, I is the IMU sensor, L is the radar, is the coordinate transformation relationship between the i-th visual sensor and the IMU sensor, I T L is the coordinate transformation relationship between the radar and IMU sensor, is the coordinate transformation relationship between the radar and the i-th visual sensor, and the covariance ∑ c / ∑ L Respectively represent the errors in the calibration process of the IMU sensor and the radar. The value of the error can be flexibly set according to the actual situation. In an example, ∑ c and∑ L All rotation components in the diagonal matrix of can be set to 0.01 radians 2 ,∑ c All conversion components can be set to 0.03㎡, ∑ L All conversion categories can be set to (0.03, 0.03.0.15)㎡.

[0146] It can be seen from formula (5) and the above calibration process that the coordinate transformation relationship between the visual sensor and the IMU sensor, as well as the coordinate transformation relationship between the radar and the IMU sensor obtained based on the joint calibration, can make the overall calibration error smaller. Generating a global map after calibration can greatly improve the accuracy of the global map, thereby improving the accuracy of the entire posture determination process.

[0147] In a possible implementation, the posture determination method proposed in the embodiment of the present disclosure may further include:

[0148] In the process of collecting map data at the second terminal, the global scene is reconstructed in real time according to the map data to generate a real-time map of the global scene.

[0149] Map data and / or a real-time map are sent to a target device, wherein the target device is used to display the geographic scope of the data collection completed for the global scene.

[0150] It is mentioned in the above-mentioned disclosed embodiments that, in a possible implementation, the global scene can be reconstructed offline based on the map data to generate a global map of the global scene.

[0151] In one possible implementation, if the collected map data is not comprehensive enough, such as missing the collection of map data in some scenes of the global scene, it will easily lead to a decrease in the accuracy of the global map established offline. If the map data of the global scene is collected again, it will require additional human labor and increase the amount of calculation. In some possible implementations, repeated collection may also occur during the map data collection process.

[0152] Therefore, in a possible implementation, in order to facilitate the acquisition of map data, the global scene can be reconstructed in real time according to the map data during the process of the second terminal acquiring map data, and a real-time map of the global scene can be generated. The implementation form of the real-time map can refer to the global map, which will not be described in detail here. In one example, the real-time map can cover the scenes corresponding to the map data that the second terminal has acquired in the global scene.

[0153] The process of real-time reconstruction can refer to the process of offline reconstruction in the above-mentioned public embodiments, and will not be described in detail here. In a possible implementation, since the real-time reconstruction can be reconstructed based on the currently collected map data, the amount of reconstructed data is smaller than that of offline reconstruction based on a large amount of map data obtained after the collection is completed, so it can have a higher reconstruction speed. In a possible implementation, the real-time reconstruction can omit some optimization processes in the offline reconstruction to improve the reconstruction speed. For example, in one example, the real-time reconstruction can omit the process of obtaining the third constraint information and optimizing the visual point cloud according to the third constraint information. In a possible implementation, the real-time reconstruction can be implemented by some specific 3D radar simultaneous localization and mapping (SLAM), also known as CML (Concurrent Mapping and Localization) system. In one example, the open source Cartographer library can be used to reconstruct the global scene in real time to generate a real-time map of the global scene.

[0154] In a possible implementation, it can be seen from the above disclosed embodiments that the posture determination method proposed in the disclosed embodiments may also include sending map data and / or a real-time map to the target device.

[0155] The target device can be used to display the geographical scope of the global scene data collection, that is, in a possible implementation, the target device can display the geographical scope covered by the map data collected by the second terminal, thereby indicating the subsequent movement direction of the second terminal in the global scene and the map data collection requirements. The implementation form of the target device is not limited in the embodiments of the present disclosure. In a possible implementation, the target device can be as follows: Figure 3 The handheld device shown is in the hands of the operator who performs map data collection, such as a tablet computer or a mobile phone; in a possible implementation, if the second terminal is placed on a mobile device (such as an automatic robot, etc.) to collect map data, in this case, the target device can be a controller or display screen of the mobile device.

[0156] The specific data sent to the target device can also be flexibly selected according to the situation. As described in the above-mentioned disclosed embodiment, in one possible implementation method, the collected map data can be sent to the target device. In one possible implementation method, a real-time map can also be sent to the target device. In one possible implementation method, map data and a real-time map can also be sent to the target device at the same time.

[0157] In the process of collecting map data on the second terminal, the global scene is reconstructed in real time according to the map data to generate a real-time map, and the map data and / or the real-time map are sent to the target device. Through the above process, the area for which map data has been collected in the global scene can be previewed in real time based on the real-time map, and the reconstruction quality of the map can be controlled at any time, thereby improving the efficiency and success rate of map data collection.

[0158] A global map can be generated through various combinations of the above disclosed embodiments, so that it is possible to obtain the global map through step S12. After obtaining the collected data and the global map, as described in the above disclosed embodiments, at least one first position of the first terminal during the collection process can be determined through step S13.

[0159] The implementation of step S13 can be flexibly determined. In one possible implementation, the global map may include at least one frame of visual point cloud, and the visual point cloud includes at least one three-dimensional feature point in the global scene; the collected data includes the first collected image; in this case, step S13 may include:

[0160] Performing feature matching on the first acquired image and at least one frame of visual point cloud to obtain a global feature matching result;

[0161] At least one first position of the first terminal during the acquisition process is determined according to the global feature matching result.

[0162] Among them, the first captured image can be an image captured by the first terminal during data collection in the target scene. The number of first captured images can be flexibly selected according to actual conditions. It can be one frame or multiple frames. It can be determined according to actual conditions and is not limited in the present disclosed embodiment.

[0163] The global feature matching result can be a three-dimensional feature point in at least one frame of visual point cloud that matches the two-dimensional feature point in the first acquired image. How to obtain the global feature matching result can be flexibly selected according to actual conditions. Please refer to the subsequent disclosed embodiments for details.

[0164] The implementation form of the visual point cloud can refer to the above-mentioned disclosed embodiments, which will not be repeated here.

[0165] It can be seen from the above disclosed embodiments that in a possible implementation, the first acquired image can be feature matched with at least one frame of visual point cloud to obtain a global feature matching result. Among them, the feature matching relationship between the first acquired image and the visual point cloud can be flexibly selected according to the actual situation. Any method that can achieve feature matching between images can be used as a feature matching method between the first acquired image and the visual point cloud, and is not limited to the following disclosed embodiments. In a possible implementation, SIFT and / or sparse optical flow tracking method (KLT, Kanade-Lucas-Tomasi Tracking Method) can be used to achieve feature matching, etc.

[0166] In a possible implementation, performing feature matching on the first acquired image and the at least one frame of visual point cloud to obtain a global feature matching result may include:

[0167] The two-dimensional feature points in the first acquired image are matched with the three-dimensional feature points included in at least one frame of visual point cloud to obtain a global feature matching result.

[0168] As described in the above-mentioned disclosed embodiments, the visual point cloud may include at least one three-dimensional feature point in the global scene, and various types of feature information corresponding to these three-dimensional feature points may also be in the global map. Therefore, in one possible implementation, the two-dimensional feature points in the first acquired image may be feature matched with the three-dimensional feature points included in at least one frame of the visual point cloud to obtain a global matching result. Among them, the feature information used for feature matching may be one or more of the various types of feature information such as feature descriptors, communication signal fingerprints, or semantic information mentioned in the above-mentioned disclosed embodiments. The specific feature information included and how to match can be flexibly selected according to the actual situation and are not limited to the following disclosed embodiments.

[0169] In a possible implementation, the two-dimensional feature points in the first acquired image are matched with the three-dimensional feature points included in at least one frame of the visual point cloud to obtain the global feature matching result, which can be implemented by an approximate nearest neighbor search ANN (Approximate Nearest Neighbor). In one example, for the features contained in the first acquired image, the K features closest to the feature can be found in the global map (the number of K can be flexibly set according to the actual situation). Then these K features can vote for each frame of the visual point cloud in the global map to indicate whether the visual point cloud corresponds to the first acquired image. If the number of votes for a certain frame or several frames of the visual point cloud exceeds the set threshold, it can be considered that the visual image corresponding to the certain frame or several frames of the visual point cloud is a co-viewing image of the first acquired image. In the co-viewing image, each three-dimensional feature point that matches the two-dimensional feature point in the first acquired image can be used as a global feature matching result.

[0170] By using ANN to match the two-dimensional feature points in the first acquired image with the three-dimensional feature points corresponding to at least one frame of visual point cloud, a global feature matching result can be obtained. This can reduce the occurrence of mismatches in the feature matching process, improve the accuracy of the global feature matching result, and thus improve the accuracy of posture determination.

[0171] After obtaining the global feature matching results, at least one first pose of the first terminal during the acquisition process can be determined based on the global feature matching results. The implementation method of this process can also be flexibly selected according to the actual situation and is not limited to the following disclosed embodiments. In one possible implementation method, the global feature matching results can be used to estimate the pose through methods such as RANSAC (Random Sample Consensus) and perspective N-point positioning (PnP), and the estimated pose is optimized through the optimization method of reprojection error, thereby obtaining at least one first pose of the first terminal during the acquisition process.

[0172] Through the above process, the features corresponding to the visual point cloud in the global map can be used to match the features between the first acquired image, so as to estimate the posture of the first terminal using the matched features in the first acquired image to obtain at least one posture of the first terminal. Since the accuracy of the global map meets the accuracy conditions, the first posture determined based on the result of matching with the global map features also has a higher accuracy, thereby improving the accuracy of the posture determination process.

[0173] In a possible implementation, the collected data may include at least two frames of first collected images, and step S13 may include:

[0174] Step S131, performing feature matching on the first collected image and at least one frame of visual point cloud to obtain a global feature matching result;

[0175] Step S132, performing feature matching based on at least two frames of first collected images to obtain a local feature matching result;

[0176] Step S133: determining at least one first pose of the first terminal during the acquisition process according to the global feature matching result and the local feature matching result.

[0177] Among them, the manner of performing feature matching between the first collected image and at least one frame of visual point cloud to obtain a global feature matching result can refer to the above-mentioned disclosed embodiments and will not be repeated here.

[0178] In a possible implementation, since the generated global map may not be able to achieve complete coverage of the global scene, the method of determining the first pose based only on the global feature matching result obtained by feature matching between the first acquired image and the visual point cloud may result in inaccurate results in determining the first pose or the inability to determine the first pose due to reasons such as the visual point cloud including incomplete or small number of three-dimensional feature points. Therefore, in a possible implementation, when the acquired data includes at least two frames of first acquired images, local feature matching results can be further obtained based on the feature matching relationship between different first acquired images, so as to jointly determine at least one first pose of the first terminal during the acquisition process based on the global feature matching results and the local feature matching results.

[0179] The local feature matching result can be a two-dimensional feature point that matches each other between different first acquisition image frames. The process of performing feature matching based on at least two frames of first acquisition images can be flexibly selected according to actual conditions. Any method that can achieve feature matching between different images can be used as a corresponding implementation method, and is not limited to the following disclosed embodiments. In a possible implementation method, the KLT method mentioned in the above disclosed embodiment can be used to perform feature matching using the optical flow features between different first acquisition images to obtain a local feature matching result.

[0180] After obtaining the global feature matching results and the local feature matching results, at least one first pose of the first terminal in the acquisition process can be jointly determined according to the global feature matching results and the local feature matching results through step S133, and the implementation method of step S133 can be flexibly determined according to the actual situation. In a possible implementation method, the implementation method of step S133 can refer to the method of determining the first pose based on the global feature matching results in the above-mentioned public embodiment, for example, the global feature matching results and the local feature matching results can be used to estimate the pose and further optimize it through the RANSAC and PnP methods.

[0181] A global feature matching result is obtained by performing feature matching between the first acquired image and at least one frame of visual point cloud, and a local feature matching result is obtained based on at least two frames of the first acquired image, thereby jointly determining at least one first pose of the first terminal during the acquisition process based on the global feature matching result and the local feature matching result. Through the above process, the global feature matching result can be assisted by the local feature matching result, thereby reducing the impact of the incomplete coverage of the global scene by the global map on the pose determination result, and improving the accuracy of the first pose.

[0182] In a possible implementation, the collected data may further include first IMU data. In this case, step S133 may include:

[0183] Acquire first constraint information according to the global feature matching result and / or the local feature matching result;

[0184] Acquire second constraint information according to the first IMU data;

[0185] According to at least one of the first constraint information and the second constraint information, the global feature matching result and the local feature matching result are processed to obtain at least one first position of the first terminal during the acquisition process.

[0186] The first IMU data may be inertial measurement data collected by the first terminal during data collection in the target scene.

[0187] It can be seen from the above-mentioned disclosed embodiments that in a possible implementation method, in the process of determining the first pose through the global feature matching results and the local feature matching results, the first constraint information and the second constraint information can also be obtained to add constraints to the process of obtaining the first pose.

[0188] The first constraint information may be constraint information obtained according to the global feature matching result and / or the local feature matching result. The specific implementation of how to obtain the first constraint information can be flexibly selected according to actual conditions and is not limited to the following disclosed embodiments.

[0189] In a possible implementation, the first constraint information can be obtained by using the information of the matched three-dimensional feature points and two-dimensional feature points in the global feature matching result. In one example, according to the global feature matching result, the process of obtaining the first constraint information can be expressed by the following formula (6):

[0190]

[0191] in, W T iis the position and posture of the device for acquiring the first acquired image in the first terminal when acquiring the i-th frame of the first acquired image, is the jth 3D feature point matched in the global feature matching result, is the global feature matching result Matched 2D feature points, To transform the three-dimensional feature points The projection result projected onto the first acquired image of the i-th frame.

[0192] In a possible implementation, the first constraint information can be obtained by using the information of the matched three-dimensional feature points and two-dimensional feature points in the local feature matching result. In one example, according to the local feature matching result, the process of obtaining the first constraint information can be expressed by the following formula (7):

[0193]

[0194] Among them, x ij is the jth two-dimensional feature point matched in the local feature matching result, X j x is the local feature matching result ij The three-dimensional feature points mapped in the target scene, f( W T i ,X j ) is the three-dimensional feature point X j The projection result projected onto the first acquired image of the i-th frame, and the meanings of other parameters can refer to the above disclosed embodiments.

[0195] The calculation results of formula (6) or formula (7) can be used as the first constraint information. In a possible implementation, the first constraint information can also be obtained based on the global feature matching result and the local feature matching result. In this case, the first constraint information can be obtained by combining the method of obtaining the first constraint information in formula (6) with that in formula (7).

[0196] Similarly, the second constraint information may be constraint information obtained based on the first IMU data. The specific implementation of how to obtain the second constraint information can be flexibly selected according to actual conditions and is not limited to the following disclosed embodiments.

[0197] In a possible implementation, the second constraint information may be obtained by using the relevant parameters of the device for collecting the first captured image and the first IMU data in the first terminal. In one example, the process of obtaining the second constraint information according to the first IMU data may be expressed by the following formula (8):

[0198]

[0199] Among them, Ci =( W T i ,Wv i ,b a ,b g ) is the parameter of the first terminal when collecting the first collected image of the i-th frame, Wv i is the speed of the first terminal, b a is the acceleration bias of the device measuring the first IMU data in the first terminal, b g is the gyroscope measurement bias of the device measuring the first IMU data in the first terminal, h(·) is the IMU cost function, and the meanings of the remaining parameters can refer to the above-mentioned disclosed embodiments.

[0200] The calculation results of formula (8) can be used as the second constraint information. It can be seen from the above disclosed embodiments that in a possible implementation, the second constraint information can be determined according to the change of the first IMU data during the process of the first terminal acquiring the first acquired image.

[0201] After obtaining the first constraint information and the second constraint information, the global feature matching results and the local feature matching results can be processed according to at least one of the first constraint information and the second constraint information to obtain at least one first position of the first terminal during the acquisition process.

[0202] Among them, the way of processing the global feature matching results and the local feature matching results according to at least one of the first constraint information and the second constraint information is not limited in the embodiment of the present disclosure. Any calculation method that can process the feature matching results based on the constraint information to obtain the posture can be used as an implementation method in the embodiment of the present disclosure.

[0203] In a possible implementation, processing the global feature matching results and the local feature matching results may include: processing the global feature matching results and the local feature matching results by bundle adjustment.

[0204] Among them, bundle adjustment (BA) is a way to implement posture solution. The specific process of solving the posture through BA can be flexibly determined according to the actual situation. In an example, at least one of formulas (6) to (8) can be used as constraint information in the process of obtaining the first posture, and the constraint information can be solved through BA to calculate the first posture with the minimum error. Among them, which data is used as constraint information is not limited in the embodiment of the present disclosure. In a possible implementation method, the first constraint information and the second constraint information can be used together as constraint information. In this case, the process of solving the constraint information through BA can be expressed by the following formula (9):

[0205]

[0206] The meaning of each parameter can be found in the above disclosed embodiments and will not be elaborated here.

[0207] The specific process of solving formula (9) by BA to determine at least one first pose is not limited in the embodiments of the present disclosure, and any method using BA for solving can be used as a solving method applied in the embodiments of the present disclosure. In a possible implementation, in the process of solving formula (9), the key frame solution and the incremental BA (ICE-BA) solution method can be used to calculate formula (9), thereby determining at least one first pose of the first terminal during the acquisition process.

[0208] At least one first pose of the first terminal during the acquisition process is obtained by processing the global feature matching results and the local feature matching results through bundle adjustment based on at least one of the first constraint information and the second constraint information. Through the above process, the first pose obtained can be optimized using at least one of the first constraint information and the second constraint information, so that the first pose finally determined is smoother as a whole and has less jitter.

[0209] Furthermore, solving the first pose by using key frames and ICE-BA can effectively reduce the amount of calculation in the process of determining the first pose, thereby improving the efficiency of the pose determination process.

[0210] As described in the above-mentioned disclosed embodiments, the accuracy of the first position determined in the disclosed embodiments is relatively high. Therefore, the method proposed in the disclosed embodiments can be applied to various scenarios in the field of mobile positioning, and the specific application scenario can be selected according to actual conditions.

[0211] In a possible implementation, the posture determination method proposed in the embodiment of the present disclosure can be used to determine the posture of the device offline. In a possible implementation, the posture determination method proposed in the embodiment of the present disclosure can be used as ground truth data to evaluate the accuracy of the results of some neural network algorithms related to mobile positioning. Therefore, in a possible implementation, the posture determination method proposed in the embodiment of the present disclosure also includes:

[0212] Determine motion truth data according to at least one first position of the first terminal during the acquisition process, wherein the motion truth data is used for at least one of the following operations:

[0213] Determine the accuracy of positioning results, train neural networks, and fuse information with global maps.

[0214] The motion truth data may be data whose results are determined to be true values ​​in neural network training, i.e., groundtruth data in the neural network algorithm. Since the first position determined in the disclosed embodiment is the position data of the first terminal during the data collection process, and the accuracy is relatively high, in one possible implementation, the first position determined may be considered to be true, and thus the first position may be used as the motion truth data.

[0215] The implementation method of the process of determining the true value data of the motion according to at least one first position of the first terminal during the acquisition process in the embodiments of the present disclosure can be flexibly determined according to actual conditions and is not limited to the following disclosed embodiments.

[0216] In a possible implementation, determining the true motion data according to at least one first position of the first terminal during the acquisition process may include:

[0217] Using at least one first position of the first terminal during the acquisition process as the true motion data; and / or,

[0218] At least one of the collected data and at least one first position of the first terminal during the collection process are used as motion true value data, wherein the collected data includes:

[0219] One or more of wireless network WiFi data, Bluetooth data, geomagnetic data, ultra-wideband UWB data, a first collected image, and a first IMU data.

[0220] It can be seen from the above-mentioned disclosed embodiments that in a possible implementation, at least one first position determined can be directly used as motion truth data. Since the number of first positions determined is not limited in the disclosed embodiments, the number of motion truth data obtained is also not limited in the disclosed embodiments. In a possible implementation, each first position determined can be used as motion truth data; in a possible implementation, one or more first positions can be selected from multiple first positions determined as motion truth data randomly or in a certain manner.

[0221] In a possible implementation, in addition to using the first pose as the motion true value data in any of the above-mentioned ways, at least one of the collected data can also be used as the motion true value data. The implementation method of the collected data is not limited in the embodiments of the present disclosure. As described in the above-mentioned disclosed embodiments, in a possible implementation, the collected data can include a first collected image and / or a first IMU data; in a possible implementation, since the implementation method of the first terminal is not limited, the type of data collected may also change and expand flexibly. Therefore, in a possible implementation, the collected data may also include one or more of wireless network WiFi data, Bluetooth data, geomagnetic data, and UWB data.

[0222] Since different types of collected data can be collected by the first terminal, these collected data can have a corresponding relationship with the determined first pose, and can also provide corresponding constraints in the pose determination process to assist in pose determination. Therefore, in a possible implementation, multiple types of collected data can also be used as motion true value data.

[0223] By using at least one first position and at least one of the collected data as motion true value data, the amount of motion true value data can be further increased, so that the application of motion true value data in different scenarios has better effects.

[0224] It can also be seen from the above disclosed embodiments that the motion truth data can be used in different scenarios. In one possible implementation, the motion truth data can be used to determine the accuracy of the positioning result. The specific method of determination is not limited in the disclosed embodiments. In one possible implementation, the motion truth data can be used as data in a benchmark data set used to determine the accuracy of the algorithm in a neural network evaluation algorithm, thereby determining the accuracy of the positioning result.

[0225] In a possible implementation, the motion truth data can also be used to train the neural network, and how to apply it in the training process is not limited in the embodiments of the present disclosure. In a possible implementation, the motion truth data can be used as training data and / or test data in the neural network, and thus applied to the training process of the neural network.

[0226] In a possible implementation, the motion truth data may also be fused with the global map. As described in the above-mentioned disclosed embodiments, in a possible implementation, the motion truth data may also include collected data such as WiFi data, Bluetooth data, geomagnetic data, or UWB data, and there is a correspondence between these collected data and the first pose. Therefore, in a possible implementation, these collected data may be used as additional auxiliary data. Through the correspondence between the first pose and the global map, these collected data may also be fused into the global map, thereby further improving the data accuracy and comprehensiveness of the global map, and may also further improve the accuracy of determining the remaining poses using the fused global map.

[0227] Application scenario examples

[0228] With the rapid development of deep learning technology, the demand for the quality, quantity, and scene diversity of motion truth data has become increasingly urgent. However, related technologies often require expensive equipment or additional site arrangements to collect motion truth data. Therefore, how to obtain a large amount of motion truth data at a lower cost has become a problem that needs to be solved urgently.

[0229] Figure 4 A schematic diagram according to an application example of the present disclosure is shown. As shown in the figure, the present disclosure embodiment proposes a method for obtaining motion true value data. The specific process may include:

[0230] As shown in the figure, in a possible implementation, the process of acquiring the motion truth data proposed in the application example of the present disclosure can be divided into two parts: global map reconstruction and motion truth data positioning.

[0231] Among them, global map reconstruction may include the following processes:

[0232] The global scene is scanned by the second terminal to obtain map data in the global scene. The structure of the second terminal is shown in the figure. The second terminal can be composed of a radar, a visual sensor and an IMU sensor, and can be carried by a backpack. As shown in the figure, in one example, the operator can carry the second terminal on his back and move in the global scene, so as to collect the laser point cloud in the global scene by radar, collect the second collected image in the global scene by visual sensor, and collect the second IMU data in the global scene by IMU sensor.

[0233] During the process of the second terminal scanning the global scene, the global map can be reconstructed in real time using the acquired laser point cloud, the second acquired image, and the second IMU data to obtain a real-time map. The process of establishing a real-time map can refer to the above-mentioned disclosed embodiments and will not be repeated here. In one possible implementation, the real-time map can reflect the scope of the operator's map data collection in the global scene, so the real-time map can be sent to the target device. In one example, the target device can be a handheld device in the operator's hand as shown in the figure.

[0234] After the second terminal completes scanning the global scene, the global map can be reconstructed offline using the laser point cloud, the second acquisition image, and the second IMU data in the acquired global scene to obtain a global map. The process of offline reconstruction is shown in the figure. In one example, the laser point cloud and the second IMU data can be calculated by the radar SLAM system to determine at least one position of the radar during the map data acquisition process, and the radar position can be converted into the position of the visual sensor through the coordinate transformation relationship between the radar and the visual sensor, thereby obtaining at least one second position of the second terminal; at the same time, the second acquisition image can be reconstructed by visual map by feature matching to obtain at least one frame of initial visual point cloud. Furthermore, the determined at least one second position can be used as the initial position, and the features in the second acquisition image can provide third constraint information for the visual map reconstruction process, so as to perform visual-radar joint optimization on the obtained initial visual point cloud. The optimization process can refer to the formula (4) in the above-mentioned disclosed embodiment. Through the above process, the optimized visual point cloud and the position and feature information of the three-dimensional feature points included in the visual point cloud can be obtained. Furthermore, the visual point cloud and the three-dimensional feature points can be used as a global map to achieve global map reconstruction.

[0235] After completing the global map reconstruction, the process of moving truth data positioning can be entered. In a possible implementation, the process of moving truth data positioning is as follows:

[0236] As shown in the figure, in a possible implementation, a first terminal such as a mobile phone or AR glasses can be used to move in a target scene in a global scene to obtain collected data. The collected data may include a first collected image and a first IMU data.

[0237] On the one hand, the first collected image can be feature matched with the global map to achieve visual positioning and obtain a global feature matching result (i.e., global feature tracking in the figure). The feature matching process between the first collected image and the global map can refer to the above-mentioned disclosed embodiments, which will not be repeated here.

[0238] On the other hand, feature matching can also be performed between different frame images in the first acquired image, thereby obtaining a local feature matching result (i.e., local feature tracking in the figure). The feature matching process between different frame images in the first acquired image can refer to the above-mentioned disclosed embodiments, which will not be repeated here.

[0239] After obtaining the global feature matching results and the local feature matching results, visual-inertial joint optimization can be performed based on the global feature matching results, the local feature matching results and the collected first IMU data through the formula (9) mentioned in the above-mentioned public embodiment, so as to determine at least one first pose of the first terminal in the movement process of the target scene. The joint optimization process can be found in the above-mentioned public embodiments in detail and will not be repeated here.

[0240] Furthermore, after obtaining at least one first pose, the obtained first pose can be used as motion true value data and stored in a benchmark database for evaluating the performance of a neural network algorithm.

[0241] The method for acquiring motion truth data proposed in the disclosed application example mainly requires high-precision map acquisition equipment that integrates lidar, camera and IMU, and the overall cost is relatively low. The global scene and target scene do not need to be pre-arranged, and the scale scalability is significantly better than the related solutions that require pre-arranged scenes. The upper limit of the scale mainly depends on the offline computing power, and the existing algorithms and computing power can already meet the needs of scenes of hundreds of thousands of square meters, so it can be used for large-scale scenes. At the same time, the global map in the same global scene can be reused, and after collecting and reconstructing the global map, massive data from mobile terminals can be collected on a large scale. The collection of mobile data only relies on the built-in sensors of the mobile device, so before each collection, there is no need to perform additional operations such as calibration and synchronization with other external devices to limit large-scale collection. In addition, it is not limited by the application scenario and can be applied to both indoor and outdoor scenes.

[0242] It should be noted that the motion truth value obtained by the embodiment of the present disclosure is not limited to being used in the evaluation or training of neural networks, but can also be extended to other scenarios, and the present disclosure does not limit this.

[0243] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0244] In addition, the present disclosure also provides an image processing device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any image processing method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0245] Figure 5 A block diagram of a posture determination device according to an embodiment of the present disclosure is shown. The posture determination device may be a terminal device, a server, or other processing devices, etc. The terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc.

[0246] In some possible implementations, the image processing apparatus may be implemented by a processor calling computer-readable instructions stored in a memory.

[0247] like Figure 5 As shown, the posture determination device 20 may include:

[0248] The collected data acquisition module 21 is used to obtain the collected data collected by the first terminal in the target scene.

[0249] The global map acquisition module 22 is used to acquire a global map including the target scene, wherein the global map is generated based on map data obtained by the second terminal collecting data on the global scene including the target scene, and the global map meets the accuracy condition.

[0250] The posture determination module 23 is used to determine at least one first posture of the first terminal during the collection process according to the feature correspondence between the collected data and the global map.

[0251] In one possible implementation, the global map includes at least one frame of visual point cloud, and the visual point cloud includes at least one three-dimensional feature point in the global scene; the collected data includes a first collected image; the posture determination module is used to: perform feature matching on the first collected image and at least one frame of visual point cloud to obtain a global feature matching result; and determine at least one first posture of the first terminal during the collection process based on the global feature matching result.

[0252] In one possible implementation, the global map includes at least one frame of visual point cloud in the target scene; the collected data includes at least two frames of first collected images; the posture determination module is used to: perform feature matching on the first collected image and the at least one frame of visual point cloud to obtain a global feature matching result; perform feature matching based on the at least two frames of first collected images to obtain a local feature matching result; and determine at least one first posture of the first terminal during the collection process based on the global feature matching result and the local feature matching result.

[0253] In a possible implementation, the collected data also includes first inertial measurement IMU data; the posture determination module is further used to: obtain first constraint information based on global feature matching results and / or local feature matching results; obtain second constraint information based on the first IMU data; and process the global feature matching results and the local feature matching results based on at least one of the first constraint information and the second constraint information to obtain at least one first posture of the first terminal during the collection process.

[0254] In a possible implementation, the posture determination module is further used to: process the global feature matching results and the local feature matching results through bundle adjustment.

[0255] In a possible implementation, the posture determination module is further used to: match the two-dimensional feature points in the first acquired image with the three-dimensional feature points included in at least one frame of visual point cloud to obtain a global feature matching result.

[0256] In a possible implementation, the device further includes: a motion true value data acquisition module, configured to determine the motion true value data according to at least one first position of the first terminal during the acquisition process.

[0257] In one possible implementation, the motion truth data acquisition module is used to: use at least one first position of the first terminal during the acquisition process as motion truth data; and / or use at least one of the acquired data and at least one first position of the first terminal during the acquisition process as motion truth data, wherein the acquired data includes: one or more of wireless network WiFi data, Bluetooth data, geomagnetic data, ultra-wideband UWB data, a first acquired image, and a first IMU data.

[0258] In one possible implementation, the motion truth data is used for at least one of the following operations: determining the accuracy of the positioning result, training the neural network, and fusing information with the global map.

[0259] In a possible implementation, the map data includes: a laser point cloud, a second acquired image, and a second IMU data in a global scene; the device also includes: a map data acquisition module, used to acquire map data of the global scene acquired by the second terminal; a global map generation module, used to perform offline reconstruction of the global scene based on the map data to generate a global map of the global scene.

[0260] In one possible implementation, the global map generation module is used to: determine at least one second posture of the second terminal during the data acquisition process based on the second IMU data and the laser point cloud; reconstruct a visual map of the global scene based on the at least one second posture and in combination with the second acquired image to obtain at least one frame of visual point cloud, wherein the visual point cloud corresponds to multiple three-dimensional feature points in the global scene; and obtain a global map of the global scene based on the at least one frame of visual point cloud.

[0261] In one possible implementation, the global map generation module is further used to: reconstruct a visual map of the global scene based on at least one second posture in combination with the second acquired image to obtain at least one frame of initial visual point cloud; obtain third constraint information in the visual map reconstruction process based on the laser point cloud and / or the second acquired image; and optimize at least one frame of initial visual point cloud based on the third constraint information to obtain at least one frame of visual point cloud.

[0262] In a possible implementation, the second terminal includes: a radar for acquiring a laser point cloud in a global scene; a visual sensor for acquiring a second collected image in the global scene; and an IMU sensor for acquiring second IMU data in the global scene.

[0263] In a possible implementation, the device is also used to: calibrate the coordinate transformation relationship between the visual sensor and the IMU sensor to obtain a first calibration result; calibrate the coordinate transformation relationship between the radar and the visual sensor to obtain a second calibration result; and jointly calibrate the coordinate transformation relationship between the visual sensor, the IMU sensor and the radar based on the first calibration result and the second calibration result.

[0264] In one possible implementation, the device is also used to: during the process of collecting map data at the second terminal, reconstruct the global scene in real time based on the map data to generate a real-time map of the global scene; send map data and / or real-time map to a target device, wherein the target device is used to display the geographical scope of the data collection completed for the global scene.

[0265] The embodiment of the present disclosure also provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0266] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0267] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code. When the computer-readable code is executed on a device, a processor in the device executes instructions for implementing the image processing method provided in any of the above embodiments.

[0268] The embodiment of the present disclosure also provides another computer program product for storing computer-readable instructions, which, when executed, enables a computer to perform the operations of the image processing method provided by any of the above embodiments.

[0269] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0270] Figure 6 A block diagram of an electronic device 800 according to an embodiment of the present disclosure is shown. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0271] Reference Figure 6 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0272] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0273] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0274] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0275] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0276] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0277] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0278] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0279] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0280] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0281] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above method.

[0282] Figure 7 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. Figure 7, the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0283] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , Mac OS X TM , UnixTM, Linux TM , FreeBSD TM or similar.

[0284] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0285] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0286] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0287] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0288] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0289] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0290] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0291] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0292] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0293] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0294] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for determining a posture, characterized in that: The method comprises: Acquire collected data collected by a first terminal in a target scene; Acquire a global map including the target scene, wherein the global map is generated based on map data obtained by collecting data of the global scene including the target scene by the second terminal, and the global map meets the accuracy condition; Determining at least one first position of the first terminal during the collection process according to a feature correspondence between the collected data and the global map; The method further comprises: Determining motion true value data according to at least one first position of the first terminal during the acquisition process; The determining of the motion true value data according to at least one first position of the first terminal during the acquisition process includes: Using at least one first position of the first terminal during the acquisition process as the true motion data; and / or, At least one of the collected data and at least one first position of the first terminal during the collection process are used as the true motion data, wherein the collected data includes: One or more of wireless network WiFi data, Bluetooth data, geomagnetic data, ultra-wideband UWB data, a first collected image, and a first IMU data.

2. The method according to claim 1, characterized in that: The global map includes at least one frame of visual point cloud, and the visual point cloud includes at least one three-dimensional feature point in the global scene; the acquisition data includes a first acquisition image; The determining, according to the feature correspondence between the collected data and the global map, at least one first posture of the first terminal in the collection process includes: Performing feature matching on the first acquired image and the at least one frame of visual point cloud to obtain a global feature matching result; At least one first pose of the first terminal during the acquisition process is determined according to the global feature matching result.

3. The method according to claim 1 or 2, characterized in that: The global map includes at least one frame of visual point cloud in the target scene; the acquisition data includes at least two frames of first acquisition images; The determining, according to the feature correspondence between the collected data and the global map, at least one first posture of the first terminal in the collection process includes: Performing feature matching on the first acquired image and the at least one frame of visual point cloud to obtain a global feature matching result; Performing feature matching based on the at least two frames of first acquired images to obtain a local feature matching result; At least one first posture of the first terminal during the acquisition process is determined according to the global feature matching result and the local feature matching result.

4. The method according to claim 3, characterized in that The collected data also includes first inertial measurement IMU data; The determining, according to the global feature matching result and the local feature matching result, at least one first posture of the first terminal in the acquisition process includes: Acquire first constraint information according to the global feature matching result and / or the local feature matching result; Acquire second constraint information according to the first IMU data; Processing the global feature matching result and the local feature matching result according to at least one of the first constraint information and the second constraint information to obtain at least one first pose of the first terminal during the acquisition process; Among them, the first constraint information is visual constraint, and the second constraint information is IMU constraint.

5. The method according to claim 4, characterized in that The processing of the global feature matching result and the local feature matching result includes: The global feature matching result and the local feature matching result are processed through bundle adjustment.

6. The method according to claim 2, characterized in that The performing feature matching on the first acquired image and the at least one frame of visual point cloud to obtain a global feature matching result includes: The two-dimensional feature points in the first acquired image are matched with the three-dimensional feature points included in the at least one frame of visual point cloud to obtain a global feature matching result.

7. The method according to claim 1, characterized in that The motion truth data is used for at least one of the following operations: Determine the accuracy of the positioning result, train the neural network, and fuse information with the global map.

8. The method according to claim 1 or 2, characterized in that: The map data includes: a laser point cloud in the global scene, a second collected image, and second IMU data; The method further comprises: Acquire map data of the global scene collected by the second terminal; Reconstructing the global scene offline according to the map data to generate a global map of the global scene; The process of reconstructing the global scene offline according to the map data to generate a global map of the global scene includes: Determine at least one second posture of the second terminal during data collection according to the second IMU data and the laser point cloud; Reconstructing a visual map of the global scene according to the at least one second posture and in combination with the second acquired image to obtain at least one frame of visual point cloud, wherein the visual point cloud includes at least one three-dimensional feature point in the global scene; A global map of the global scene is obtained according to the at least one frame of visual point cloud.

9. The method according to claim 8, characterized in that The reconstructing a visual map of the global scene according to the at least one second posture in combination with the second acquired image to obtain at least one frame of visual point cloud includes: Reconstructing a visual map of the global scene according to the at least one second posture and in combination with the second acquired image to obtain at least one frame of initial visual point cloud; Acquire third constraint information in a visual map reconstruction process according to the laser point cloud and / or the second acquired image; Optimizing the at least one frame of initial visual point cloud according to the third constraint information to obtain at least one frame of visual point cloud; The third constraint information includes at least one of the following: a plane constraint of the laser point cloud, an edge constraint of the laser point cloud, and a visual constraint.

10. The method according to claim 8, characterized in that The second terminal includes: A radar, used to obtain a laser point cloud in the global scene; A visual sensor, used to acquire a second collected image in the global scene; The IMU sensor is used to obtain second IMU data in the global scene.

11. The method according to claim 10, characterized in that Before the global scene is offline reconstructed according to the map data to generate a global map of the global scene, the method further includes: Calibrate the coordinate transformation relationship between the visual sensor and the IMU sensor to obtain a first calibration result; Calibrate the coordinate transformation relationship between the radar and the visual sensor to obtain a second calibration result; According to the first calibration result and the second calibration result, the coordinate transformation relationship among the visual sensor, the IMU sensor and the radar is jointly calibrated.

12. The method according to claim 8, characterized in that The method further comprises: In the process of collecting map data by the second terminal, the global scene is reconstructed in real time according to the map data to generate a real-time map of the global scene; The map data and / or the real-time map are sent to a target device, wherein the target device is used to display a geographical range in which data collection is completed for the global scene.

13. A posture determination device, characterized in that: include: A collection data acquisition module, used to acquire the collection data collected by the first terminal in the target scene; A global map acquisition module, used to acquire a global map including the target scene, wherein the global map is generated based on map data obtained by the second terminal collecting data on the global scene including the target scene, and the global map meets the accuracy condition; A posture determination module, used to determine at least one first posture of the first terminal in the collection process according to the feature correspondence between the collected data and the global map; The device also includes: A motion true value data acquisition module, used to determine the motion true value data according to at least one first position of the first terminal during the acquisition process; The motion true value data acquisition module is used to: use at least one first position of the first terminal during the acquisition process as the motion true value data; and / or use at least one of the acquired data and at least one first position of the first terminal during the acquisition process as the motion true value data, wherein the acquired data includes: one or more of wireless network WiFi data, Bluetooth data, geomagnetic data, ultra-wideband UWB data, a first acquired image and a first IMU data.

14. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images