Method, device and system for determining posture change information of movable device

Abstract

The present disclosure relates to a method, device and system for determining the posture change information of a movable device, and belongs to the field of machine vision technology. The method comprises: determining the relative position information of the physical point corresponding to each feature point relative to the movable device according to the three-dimensional position information of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is in the initial position; in the process of the movable device moving, in the image currently captured by each three-dimensional image capturing component, determining the three-dimensional position information of the feature point contained in each image; determining the posture change information of the movable device relative to the initial position according to the relative position information corresponding to the determined physical point, the three-dimensional position information of the currently determined feature point and the pre-stored relative posture information between different three-dimensional image capturing components. By adopting the present disclosure, the posture change information can also be reliably determined under harsh environmental conditions.

Description

Technical Field

[0001] The present disclosure relates to the field of machine vision technology, and in particular to a method, device and system for determining posture change information of a movable device. Background Art

[0002] In the related art, visual navigation technology can be used to enable a mobile device to perform navigation processing in an unfamiliar environment by relying on images captured by a three-dimensional image capturing component installed in the mobile device. The three-dimensional image capturing component may include two cameras. The images captured by the three-dimensional image capturing component may include two two-dimensional images, namely, a two-dimensional image A and a two-dimensional image B. The depth information of a pixel point in the two-dimensional image A may be determined based on the two-dimensional image A, the two-dimensional image B, and the baseline distance between the two cameras. The three-dimensional position information of the pixel point may include the two-dimensional position information and depth information of the pixel point in the two-dimensional image A.

[0003] When the mobile device is turned on, it can be considered that the mobile device is in the initial position. The mobile device can identify whether the image captured by the three-dimensional image shooting component contains an image of an obstacle. If so, it is necessary to continuously detect the relative position information of the obstacle relative to the mobile device after the movement of the mobile device during the movement of the mobile device. In this way, based on the relative position information of the obstacle relative to the mobile device after the movement, it is determined how to control the mobile device to avoid the obstacle and move. The relative position information of the obstacle relative to the mobile device after the movement can be determined based on the relative position information of the obstacle relative to the mobile device at the initial position and the posture change information of the mobile device relative to the initial position. Among them, the posture change information records the degree of change in the position and posture of the mobile device relative to the initial position. The above relative position information is the relative position information of the obstacle relative to the mobile device, taking into account the position and posture of the mobile device. For example, the relative position information can be the coordinates of the obstacle in the coordinate system established based on the mobile device, or the relative position information can be the distance of the obstacle at a certain angle to the left (or right) of the positive direction of the mobile device.

[0004] The following describes a method for determining the posture change information: the posture change information can be determined based on the relative position information of the physical point corresponding to the feature point relative to the initial position of the movable device, and the three-dimensional position information of the feature point in the image continuously captured by the three-dimensional image capturing component of the movable device during the movement. Among them, the feature point can be a pixel point with certain image characteristics in the image, which can be extracted from the image by a preset feature extraction algorithm, such as FAST (Features from Accelerated Segment Test, a corner point detection method) corner point, for example, the four corners of the display will be identified as FAST corner points.

[0005] In the process of implementing the present disclosure, the inventors found that there are at least the following problems:

[0006] Only one 3D image capturing component is installed in the mobile device. If the 3D image capturing component is capturing strong light, the captured image will be too bright, making it difficult to accurately identify feature points in the image, and the determined posture change information will also be inaccurate. Summary of the invention

[0007] In order to overcome the problems existing in the related art, the present disclosure provides the following technical solutions:

[0008] According to a first aspect of an embodiment of the present disclosure, a method for determining posture change information of a movable device is provided, wherein the movable device includes at least two three-dimensional image capturing components, and the method includes:

[0009] Determine relative position information of a physical point corresponding to each feature point relative to the movable device according to three-dimensional position information of at least one feature point in an image captured by each three-dimensional image capturing component when the movable device is at an initial position;

[0010] During the movement of the movable device, in the images currently captured by each 3D image capturing component, determining the 3D position information of the feature points contained in each image;

[0011] According to the relative position information corresponding to the determined physical point, the three-dimensional position information of the currently determined feature point and the pre-stored relative posture information between different three-dimensional image shooting components, the posture change information of the movable device relative to the initial position is determined.

[0012] Optionally, determining the relative position information of the physical point corresponding to each feature point relative to the movable device according to the three-dimensional position information of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is at the initial position includes:

[0013] Determine relative position information of a physical point corresponding to each feature point relative to the corresponding three-dimensional image capturing component according to three-dimensional position information of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is at the initial position;

[0014] Based on the relative position information of the physical point corresponding to each feature point relative to the corresponding three-dimensional image capturing component, and the relative posture information between the pre-stored reference three-dimensional image capturing component and other three-dimensional image capturing components in the at least two pre-stored three-dimensional image capturing components, the relative position information of the physical point corresponding to each feature point relative to the reference three-dimensional image capturing component is determined as the relative position information of the physical point corresponding to each feature point relative to the movable device.

[0015] Optionally, determining the posture change information of the movable device relative to the initial position according to the relative position information corresponding to the determined physical point, the three-dimensional position information of the currently determined feature point, and the pre-stored relative posture information between different three-dimensional image capturing components includes:

[0016] Determine the three-dimensional position information of each feature point relative to the image currently captured by the reference three-dimensional image capturing component according to the three-dimensional position information of the feature point currently determined and the pre-stored relative position information between the reference three-dimensional image capturing component and other three-dimensional image capturing components;

[0017] Based on the relative position information of the physical point corresponding to each feature point relative to the reference three-dimensional image shooting component, and the three-dimensional position information of each feature point relative to the image currently shot by the reference three-dimensional image shooting component, the posture change information of the movable device relative to the initial position is determined.

[0018] Optionally, the method further comprises:

[0019] Determine, among the feature points contained in the image currently captured by each 3D image capturing component, a target feature point that matches the image feature of any feature point among the at least one feature point, and determine the 3D position information of each target feature point;

[0020] The determining of the posture change information of the movable device relative to the initial position according to the relative position information corresponding to the determined physical point, the three-dimensional position information of the currently determined feature point and the pre-stored relative posture information between different three-dimensional image capturing components comprises:

[0021] According to the relative position information corresponding to the determined physical point, the three-dimensional position information of each target feature point and the pre-stored relative posture information between different three-dimensional image shooting components, the posture change information of the movable device relative to the initial position is determined.

[0022] Optionally, determining the posture change information of the movable device relative to the initial position according to the determined relative position information corresponding to the physical point, the three-dimensional position information of each target feature point and the pre-stored relative posture information between different three-dimensional image capturing components includes:

[0023] If the number of the target feature points is greater than or equal to a preset threshold value, the posture change information of the movable device relative to the initial position is determined based on the relative position information corresponding to the determined physical points, the three-dimensional position information of each target feature point and the relative posture information between different pre-stored three-dimensional image shooting components.

[0024] Optionally, the method further comprises:

[0025] If the number of the target feature points is less than a preset threshold value, the current position is set as the initial position, and the relative position information of the physical point corresponding to each feature point relative to the movable device is determined based on the three-dimensional position information of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is at the reset initial position.

[0026] According to a second aspect of an embodiment of the present disclosure, there is provided a device for determining posture change information of a movable device, wherein the movable device includes at least two three-dimensional image capturing components, and the device includes:

[0027] a determination module, configured to determine relative position information of a physical point corresponding to each feature point relative to the movable device according to three-dimensional position information of at least one feature point in an image captured by each three-dimensional image capturing component when the movable device is at an initial position;

[0028] The determination module is further used to determine the three-dimensional position information of the feature points contained in each image currently captured by each three-dimensional image capturing component during the movement of the movable device;

[0029] The determination module is also used to determine the posture change information of the movable device relative to the initial position based on the relative position information corresponding to the determined physical point, the three-dimensional position information of the currently determined feature point and the pre-stored relative posture information between different three-dimensional image shooting components.

[0030] Optionally, the determining module is used to:

[0031] Determine relative position information of a physical point corresponding to each feature point relative to the corresponding three-dimensional image capturing component according to three-dimensional position information of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is at the initial position;

[0032] Based on the relative position information of the physical point corresponding to each feature point relative to the corresponding three-dimensional image capturing component, and the relative posture information between the pre-stored reference three-dimensional image capturing component and other three-dimensional image capturing components in the at least two pre-stored three-dimensional image capturing components, the relative position information of the physical point corresponding to each feature point relative to the reference three-dimensional image capturing component is determined as the relative position information of the physical point corresponding to each feature point relative to the movable device.

[0033] Optionally, the determining module is used to:

[0034] Determine the three-dimensional position information of each feature point relative to the image currently captured by the reference three-dimensional image capturing component according to the three-dimensional position information of the feature point currently determined and the pre-stored relative position information between the reference three-dimensional image capturing component and other three-dimensional image capturing components;

[0035] Based on the relative position information of the physical point corresponding to each feature point relative to the reference three-dimensional image shooting component, and the three-dimensional position information of each feature point relative to the image currently shot by the reference three-dimensional image shooting component, the posture change information of the movable device relative to the initial position is determined.

[0036] Optionally, the determining module is further used to:

[0037] Determine, among the feature points contained in the image currently captured by each 3D image capturing component, a target feature point that matches the image feature of any feature point among the at least one feature point, and determine the 3D position information of each target feature point;

[0038] According to the relative position information corresponding to the determined physical point, the three-dimensional position information of each target feature point and the pre-stored relative posture information between different three-dimensional image shooting components, the posture change information of the movable device relative to the initial position is determined.

[0039] Optionally, the determining module is used to:

[0040] When the number of the target feature points is greater than or equal to a preset threshold value, the posture change information of the movable device relative to the initial position is determined based on the relative position information corresponding to the determined physical points, the three-dimensional position information of each target feature point and the relative posture information between different pre-stored three-dimensional image shooting components.

[0041] Optionally, the determining module is further used to:

[0042] When the number of the target feature points is less than a preset threshold value, the current position is set as the initial position, and the relative position information of the physical point corresponding to each feature point relative to the movable device is determined based on the three-dimensional position information of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is at the reset initial position.

[0043] According to a third aspect of an embodiment of the present disclosure, a system for determining posture change information of a movable device is provided, the system comprising a server and a movable device, the movable device comprising at least two three-dimensional image capturing components, wherein:

[0044] The movable device is used to capture images through each 3D image capturing component whenever a preset period arrives, and send the images captured by each 3D image capturing component to the server;

[0045] The server is used to determine the relative position information of the physical point corresponding to each feature point relative to the movable device based on the three-dimensional position information of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is in the initial position; during the movement of the movable device, in the image currently captured by each three-dimensional image capturing component, determine the three-dimensional position information of the feature points contained in each image; and determine the posture change information of the movable device relative to the initial position based on the relative position information corresponding to the determined physical point, the three-dimensional position information of the currently determined feature point and the pre-stored relative posture information between different three-dimensional image capturing components.

[0046] According to a fourth aspect of an embodiment of the present disclosure, a server is provided, the server comprising a processor, a communication interface, a memory, and a communication bus, wherein:

[0047] The processor, the communication interface and the memory communicate with each other via the communication bus;

[0048] The memory is used to store computer programs;

[0049] The processor is used to execute the program stored in the memory to implement the above method for determining the posture change information of the movable device.

[0050] According to a fifth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method for determining the posture change information of a movable device is implemented.

[0051] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0052] Through the method provided by the embodiment of the present disclosure, the information on the change in the posture of the movable device relative to the initial position can be determined through images captured by at least two three-dimensional image capturing components. Even if one of the three-dimensional image capturing components is affected by strong light or other factors, the information on the change in the posture of the movable device relative to the initial position can be determined with the help of images captured by the remaining three-dimensional image capturing components. Furthermore, the information on the change in posture can be reliably determined under harsh environmental conditions.

[0053] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description, are used to explain the principles of the present disclosure. In the drawings:

[0055] Figure 1 is a flow chart of a method for determining posture change information of a movable device according to an exemplary embodiment;

[0056] Figure 2 is a structural schematic diagram showing a method of determining a base of a movable device according to an exemplary embodiment;

[0057] Figure 3 is a schematic diagram showing a method for determining posture change information of a movable device according to an exemplary embodiment;

[0058] Figure 4 is a schematic diagram of transformation relationships of a method for determining posture change information of a movable device according to an exemplary embodiment;

[0059] Figure 5 is a structural schematic diagram of a device for determining posture change information of a movable device according to an exemplary embodiment;

[0060] Figure 6 The figure is a schematic diagram showing the structure of a server according to an exemplary embodiment.

[0061] Legend:

[0062] 1-6 camera; 7 hardware trigger board;

[0063] 8 metal bottom plate; 9-11 wiring holes;

[0064] 12-17 rectangular groove

[0065] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0066] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0067] The embodiment of the present disclosure provides a method for determining the posture change information of a movable device, which can be implemented by the cooperation of a movable terminal device and a server. The movable terminal device can be a sweeper, a robot, an unmanned vehicle, etc.

[0068] The server may include components such as a processor and a memory. The processor may be a CPU (Central Processing Unit) or the like, and may be used to determine the relative position information of the physical point corresponding to each feature point relative to the movable device based on the three-dimensional position information of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is in the initial position, and perform other processing. The memory may be a RAM (Random Access Memory), Flash (flash memory), etc., and may be used to store received data, data required for the processing process, data generated during the processing process, etc., such as the relative position information corresponding to the physical point, etc.

[0069] The server may further include a transceiver, etc. The transceiver may be used to transmit data with a removable device, and the transceiver may include a Bluetooth component, a WiFi (Wireless-Fidelity, wireless high-fidelity technology) component, an antenna, a matching circuit, a modem, etc.

[0070] An exemplary embodiment of the present disclosure provides a method for determining posture change information of a movable device, wherein the movable device includes at least two three-dimensional image capturing components, such as Figure 1 As shown, the processing flow of the method may include the following steps:

[0071] Step S110, determining the relative position information of the physical point corresponding to each feature point relative to the movable device based on the 3D position information of at least one feature point in the image captured by each 3D image capturing component when the movable device is at the initial position.

[0072] In the embodiment of the present disclosure, at least two 3D image capturing components may be provided in the mobile device, and three 3D image capturing components (comprising six cameras in total) are recommended. Figure 2 As shown, it is a structural diagram of the base of the movable device. In the base of the movable device, the components corresponding to numbers 1-6 are six different cameras. The component corresponding to number 7 is a hardware trigger board, which is used to control the six cameras for synchronous exposure. The component corresponding to number 8 is a metal base plate, and a camera and a hardware trigger board can be arranged above the metal base plate. Three wiring holes corresponding to numbers 9-11 can also be arranged on the metal base plate, and the wiring holes can be used to output the relevant wires of the camera and the hardware trigger board to the bottom of the base of the movable device. Six rectangular grooves corresponding to numbers 12-17 can also be arranged on the metal base plate, and the rectangular grooves are used to fix the camera on the metal base plate.

[0073] The six cameras can be independent industrial cameras, and the types of cameras can include ordinary pinhole cameras, fisheye cameras, etc. Two adjacent cameras placed in parallel are a pair of binocular cameras (three-dimensional image shooting components). The viewing angles of different pairs of binocular cameras may or may not overlap.

[0074] The metal base plate can be set to a triangle with three sides, and the three pairs of binocular cameras are respectively set on one of the three sides, and the two cameras in a pair of binocular cameras have the same shooting direction. The baseline distance of each pair of binocular cameras can be set to 12 cm. The movable device can include three three-dimensional image shooting components, and the angle between any two three-dimensional image shooting components and the line connecting the preset reference center of the movable device is equal to 120 degrees, that is, the angle between two adjacent cameras in different pairs of binocular cameras can be set to 120 degrees.

[0075] After the hardware trigger board controls the six cameras to perform synchronous exposure, the captured images can be sent to the server, which performs subsequent processing and returns the processing results to the mobile device. The method provided in this embodiment is mainly executed by the server and cooperated by the mobile device.

[0076] In the embodiment of the present disclosure, a three-dimensional image shooting component may include two cameras. Since the pixel points in the two-dimensional image A shot by a single camera lack depth information, it is necessary to use the two-dimensional image B shot by another camera to determine the depth information of the pixel points in the two-dimensional image A. Thus, a three-dimensional image can be established based on the two-dimensional image A and the depth information of the pixel points in the two-dimensional image A, and each pixel point in the three-dimensional image has corresponding three-dimensional position information.

[0077] Next, feature points in the images captured by all cameras can be extracted. Feature points in the images captured by all cameras can be extracted by algorithms such as ORB (Oriented FAST and Rotated BRIEF, a fast feature point extraction and description algorithm), DAISY (a feature point extraction algorithm) and SIFT (Scale-invariant feature transform, scale-invariant feature transform) algorithm. In the embodiments disclosed herein, the FAST (Features from Accelerated Segment Test, a corner point detection method) corner point extracted by the ORB algorithm is mainly used as an example for explanation. Other algorithms are similar and will not be described in detail here.

[0078] The ORB algorithm mainly uses any pixel in the captured image as the center of the circle and analyzes the grayscale values ​​of the 16 pixels around it. If there are N (generally, N is 9) consecutive pixels among the 16 pixels whose grayscale values ​​are greater than the grayscale value of the center of the circle plus the preset grayscale value threshold, and are all less than the grayscale value of the center of the circle minus the preset grayscale value threshold, then the center of the circle is a FAST corner point.

[0079] With point P in the captured image as the center, there are three states between the grayscale values ​​of the 16 pixels around it and point P, namely, the grayscale of the pixel in the 16 pixels is brighter than point P, the grayscale of the pixel in the 16 pixels is similar to point P, and the grayscale of the pixel in the 16 pixels is darker than point P. It can be determined by formula 1:

[0080]

[0081] Among them, S p→x The grayscale of the 16 pixels is compared with the three states of point P, namely d (darker), s (similar) and b (brighter). I indicates the grayscale value, and T is the preset grayscale value threshold, which can be adjusted according to the environment.

[0082] The properties of FAST corner points can be described by BRIEF descriptors. A circular area with a diameter of 31 can be selected around the FAST corner point, and N pairs of pixels can be selected to compare the grayscale values ​​according to the predetermined rules. A 256-bit binary descriptor is generated based on the comparison results, which can be used as the BRIEF descriptor of the FAST corner point.

[0083] Before determining the relative position information of the physical point corresponding to each feature point relative to the movable device, the captured image can be corrected. Since the captured image has a certain degree of distortion, the captured image can be corrected to obtain an ideal image. Ideally, Figure 3 As shown in FIG. 1 , the camera coordinate system of a pair of binocular cameras is to translate the origin from point O2 to point O1, and the coordinate system does not rotate.

[0084] After obtaining the ideal image, for each pair of binocular cameras, the FAST corner points that match the FAST corner points in the image captured by the left camera can be determined in the image captured by the right camera, and the depth information of the FAST corner points can be determined based on the baseline distance of the cameras.

[0085] In the disclosed embodiment, it can be considered that the right camera assists the left camera in determining the depth information of the FAST corner points in the image taken by the left camera, and the posture change information is mainly determined and processed based on the image taken by the left camera.

[0086] The camera coordinate system can be established based on the left camera in each pair of binocular cameras. The camera coordinate system is a coordinate system established with the optical center of the camera as the origin, the optical axis as the Z axis, the line where the long side of the captured image is located as the X axis, and the line where the wide side of the captured image is located as the Y axis. Figure 2 The device in can obtain three camera coordinate systems. Among the three camera coordinate systems, the camera coordinate system established by the camera corresponding to label 1 can be used as the main body coordinate system.

[0087] Before executing step S110, the relative pose information between the camera coordinate system other than the body coordinate system and the body coordinate system in the three camera coordinate systems can be pre-calibrated, which can also be called the external parameters of the camera coordinate system other than the body coordinate system in the three camera coordinate systems relative to the body coordinate system, and are denoted as T bl (Camera corresponding to label 4) and T br (The camera corresponding to label 5).

[0088] Next, the relative position information of the physical point corresponding to each feature point relative to the corresponding three-dimensional image capturing component can be determined based on the three-dimensional position information of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is at the initial position. Finally, the relative position information of the physical point corresponding to each feature point relative to the corresponding three-dimensional image capturing component and the relative posture information between the pre-stored reference three-dimensional image capturing component and other three-dimensional image capturing components in at least two pre-stored three-dimensional image capturing components can be determined as the relative position information of the physical point corresponding to each feature point relative to the movable device.

[0089] The relative position information is the relative position information of the obstacle relative to the movable device under the current position and posture of the movable device. If the movable device is currently in the initial position, the relative position information is the relative position information of the obstacle relative to the movable device at the initial position. The relative position information of the physical point relative to the movable device can be the coordinate information of the physical point in a coordinate system established with a preset point in the movable device as the origin, or can be the coordinate information of the physical point in a coordinate system established with the optical center of the reference three-dimensional image capturing component of the movable device as the origin, the line where the optical axis is located as the Z axis, the vertical line as the Y axis, and the line perpendicular to both the Z axis and the Y axis as the X axis.

[0090] In the application, the relative position information of the physical point relative to the movable device at the initial position A can indicate that the physical point is 30 degrees and 20 meters to the left of the movable device. After the movable device moves to position B, the relative position information of the physical point relative to the movable device at position B can indicate that the physical point is 60 degrees and 15 meters to the right of the movable device.

[0091] The at least two 3D image capturing components may include a reference 3D image capturing component (camera corresponding to number 1) and a non-reference 3D image capturing component (cameras corresponding to numbers 4 and 5 respectively) other than the reference 3D image capturing component.

[0092] First, the relative position information of the physical point corresponding to at least one feature point in the image captured by each non-reference three-dimensional image capturing component can be determined relative to the non-reference three-dimensional image capturing component based on the three-dimensional position information of at least one feature point in the image captured by each non-reference three-dimensional image capturing component when the movable device is in the initial position, and the conversion relationship between the three-dimensional position information of the pre-stored feature point corresponding to each non-reference three-dimensional image capturing component and the relative position information of the physical point relative to the non-reference three-dimensional image capturing component (also called the camera internal parameter).

[0093] Next, the relative position information (also called map point) of the physical point corresponding to at least one feature point in the image captured by each non-reference 3D image capturing component relative to the non-reference 3D image capturing component can be determined based on the relative position information of the physical point corresponding to at least one feature point in the image captured by each non-reference 3D image capturing component relative to the non-reference 3D image capturing component and the relative posture information of each non-reference 3D image capturing component relative to the reference 3D image capturing component (also called the external parameter of the camera coordinate system other than the body coordinate system in the three camera coordinate systems relative to the body coordinate system). The initial position of the reference 3D image capturing component is the initial position of the movable device.

[0094] Finally, based on the three-dimensional position information of at least one feature point in the image captured by the benchmark three-dimensional image capturing component when the movable device is in the initial position, and the conversion relationship between the three-dimensional position information of the pre-stored feature point corresponding to the benchmark three-dimensional image capturing component and the relative position information of the physical point relative to the benchmark three-dimensional image capturing component, the relative position information (also called map point) of the physical point corresponding to at least one feature point in the image captured by the benchmark three-dimensional image capturing component relative to the benchmark three-dimensional image capturing component can be determined.

[0095] Based on the following formula, the relative position information of the physical point corresponding to each feature point relative to the movable device (also referred to as a map point) can be determined.

[0096]

[0097] in, is the coordinate of the map point in the world coordinate system. K is the camera internal parameter corresponding to different 3D image shooting components. is the FAST corner point, and Z is the depth information of the FAST corner point.

[0098] Among them, the world coordinate system is the body coordinate system of the movable device when it is turned on. The world coordinate system is fixed, while the body coordinate system will translate and rotate as the movable device moves.

[0099] Through the camera internal parameters, the position information of the pixel points in the captured image can be converted into the position information of the physical point in the camera coordinate system. Among them, the camera internal parameters include the camera's inherent attribute parameters such as the camera's shooting focal length.

[0100] After obtaining multiple map points, the map is established. The map records the location information of different map points in the world coordinate system and their corresponding BRIEF descriptors. It should be noted that the number of map points can be determined based on the distance that the movable device moves each time. During the movement of the movable device, new images are continuously captured, and map points are continuously matched with feature points in the new captured images. When the number of matches between the map points and the feature points in the new captured images is less than the preset number threshold, the map is re-established to generate new map points. The currently matched map points no longer participate in the process of generating new map points.

[0101] Since the generation of map points is related to the frame number of the captured image, the frame number of the image can be recorded corresponding to the map point. In practical applications, for example, if the frame number of the currently captured image is N, when performing matching processing, the inserted map point is generated based on the image with the frame number NT, and the projection processing of the map point in the currently captured image is performed. Among them, T is a fixed value and can be adjusted according to specific circumstances.

[0102] Step S120, during the movement of the movable device, in the images currently captured by each 3D image capturing component, determine the 3D position information of the feature points contained in each image.

[0103] During the movement of the movable device, the three-dimensional position information of the feature points contained in each image can be determined in the image currently captured by each three-dimensional image capturing component according to a preset period.

[0104] The three-dimensional position information of the pixel point in the image can be converted into the position information of the physical point in the camera coordinate system through the camera internal parameters. Figure 2 The position information of the physical point corresponding to the pixel point in the image captured by the camera corresponding to number 4 and the camera corresponding to number 5 can be converted to the camera coordinate system of the camera corresponding to number 4 and the camera coordinate system of the camera corresponding to number 5 through the camera internal parameters of the camera corresponding to number 4 and the camera internal parameters of the camera corresponding to number 5. Figure 4 As shown, the relative position information between different three-dimensional image capturing components can be the external parameters of the camera coordinate system other than the body coordinate system in the three camera coordinate systems relative to the body coordinate system, which are respectively denoted as T bl (Camera corresponding to label 4) and T br (The camera corresponding to label 5). bl and T br, the position information of the physical point in the camera coordinate system of the camera corresponding to number 4 and the camera coordinate system of the camera corresponding to number 5 is converted into the position information of the physical point in the body coordinate system. In this way, the position information of the physical point corresponding to the images captured by the three different three-dimensional image capturing components can be converted into the position information of the physical point in the body coordinate system. Furthermore, the position information of the physical point in the body coordinate system can be converted into the position information of the physical point in the world coordinate system through the posture change information.

[0105] The method provided by the embodiment of the present disclosure also includes: determining target feature points that match the image features of any feature point in at least one feature point among the feature points contained in the image currently captured by each three-dimensional image capturing component, and determining the three-dimensional position information of each target feature point.

[0106] Since the current posture change information has not been determined yet, the map point (the position information of the physical point in the world coordinate system) can be projected into the current image with the help of the posture change information determined last time, and the projection point can be obtained in the current image. From the BRIEF descriptors corresponding to the FAST corner points contained in the field of the projection point, select the target feature point m whose feature vector distance with the BRIEF descriptor of the map point is less than the preset distance threshold, then the target feature point m is the projection point that matches the map point. Determine the three-dimensional position information of each target feature point.

[0107] like Figure 4 As shown, if there are 3 map points, namely P1, P2, P n , the matching projection points are m1, m2, m n , then the following formula can be used according to the map points P1, P2, P n and other related parameters to determine m1, m2, m n 3D position information.

[0108] m1=K1T -1 P1;m2=K2T bl -1 T -1 P2;m3=K3T br -1 T -1 P3 (Formula 3)

[0109] Among them, K is the camera internal parameter corresponding to different cameras, T bl and T br is an external parameter, and T is the last determined pose change information. The last determined pose change information is only an estimated pose change information and cannot be directly used as the current pose change information.

[0110] Step S130, determining the posture change information of the movable device relative to the initial position based on the relative position information corresponding to the determined physical point, the three-dimensional position information of the currently determined feature point and the pre-stored relative posture information between different three-dimensional image shooting components.

[0111] First, the three-dimensional position information of each feature point relative to the image currently captured by the reference three-dimensional image capturing component can be determined based on the three-dimensional position information of the feature point currently determined and the pre-stored relative position information between the reference three-dimensional image capturing component and other three-dimensional image capturing components. Next, the position change information of the movable device relative to the initial position can be determined based on the relative position information of the physical point corresponding to each feature point relative to the reference three-dimensional image capturing component and the three-dimensional position information of each feature point relative to the image currently captured by the reference three-dimensional image capturing component.

[0112] The at least two three-dimensional image capturing components include a reference three-dimensional image capturing component and a non-reference three-dimensional image capturing component other than the reference three-dimensional image capturing component.

[0113] First, during the movement of the movable device, the three-dimensional position information of the feature points contained in the image currently captured by each non-reference three-dimensional image capturing component can be determined, and the three-dimensional position information of the feature points contained in the image currently captured by the reference three-dimensional image capturing component can be determined.

[0114] Next, the three-dimensional position information of the feature points contained in the image currently captured by each non-reference three-dimensional image capturing component relative to the image currently captured by the reference three-dimensional image capturing component can be determined respectively based on the three-dimensional position information of the feature points in the image currently captured by each non-reference three-dimensional image capturing component and the relative posture information of each non-reference three-dimensional image capturing component relative to the reference three-dimensional image capturing component.

[0115] Finally, the posture change information of the movable device relative to the initial position can be determined based on the relative position information corresponding to the physical points, the three-dimensional position information of the feature points contained in the image currently taken by each non-reference three-dimensional image shooting component relative to the image currently taken by the reference three-dimensional image shooting component, and the three-dimensional position information of the feature points contained in the image currently taken by the reference three-dimensional image shooting component.

[0116] Optionally, a target feature point that matches the image feature of any feature point in at least one feature point can be determined from the feature points contained in the image currently captured by each 3D image capturing component, and the 3D position information of each target feature point can be determined. Then, based on the relative position information corresponding to the determined physical point, the 3D position information of each target feature point, and the relative position and posture information between different pre-stored 3D image capturing components, the position and posture change information of the movable device relative to the initial position can be determined.

[0117] Optionally, if the number of target feature points is greater than or equal to a preset number threshold, the position change information of the movable device relative to the initial position is determined based on the relative position information corresponding to the determined physical point, the three-dimensional position information of each target feature point, and the pre-stored relative position information between different three-dimensional image capturing components. If the number of target feature points is less than the preset number threshold, the current position is set as the initial position, and the relative position information of the physical point corresponding to each feature point relative to the movable device is determined based on the three-dimensional position information of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is at the reset initial position.

[0118] During the movement of the mobile device, new images are continuously captured, and map points are continuously matched with feature points in the captured new images. When the number of matching map points and feature points in the captured new images is less than a preset number threshold, the map is re-established and new map points are generated. If there are currently matched map points, they will no longer participate in the process of generating new map points.

[0119] The current posture change information can be determined according to Formula 4.

[0120] min[||(m1-K1T -1 P1)|| 2 +||(m2-K2T bl -1 T -1 P2)|| 2 +||(m3-K3T br -1 T -1 P3)|| 2 ] (Formula 4)

[0121] In the above formula, T is unknown and is the current posture change information, and the other parameters are known. The value of T can be optimized iteratively so that the final result of formula 4 is minimized under a certain value of T, that is, T that can minimize the error between the three-dimensional position information of the target feature point that matches the map point and the three-dimensional position information of the projection point theoretically calculated by formula 3 is determined, which is the current posture change information.

[0122] Through the method provided by the embodiment of the present disclosure, the information on the change in the posture of the movable device relative to the initial position can be determined through images captured by at least two three-dimensional image capturing components. Even if one of the three-dimensional image capturing components is affected by strong light or other factors, the information on the change in the posture of the movable device relative to the initial position can be determined with the help of images captured by the remaining three-dimensional image capturing components. Furthermore, the information on the change in posture can be reliably determined under harsh environmental conditions.

[0123] Another exemplary embodiment of the present disclosure provides a device for determining posture change information of a movable device, wherein the movable device includes at least two three-dimensional image capturing components, such as Figure 5 As shown, the device comprises:

[0124] A determination module 510, configured to determine relative position information of a physical point corresponding to each feature point relative to the movable device based on three-dimensional position information of at least one feature point in an image captured by each three-dimensional image capturing component when the movable device is at an initial position;

[0125] The determination module 510 is further configured to determine the three-dimensional position information of the feature points contained in each image currently captured by each three-dimensional image capturing component during the movement of the movable device;

[0126] The determination module 510 is also used to determine the posture change information of the movable device relative to the initial position based on the relative position information corresponding to the determined physical point, the three-dimensional position information of the currently determined feature point and the pre-stored relative posture information between different three-dimensional image shooting components.

[0127] Optionally, the determining module 510 is configured to:

[0128] Determine relative position information of a physical point corresponding to each feature point relative to the corresponding three-dimensional image capturing component according to three-dimensional position information of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is at the initial position;

[0129] Based on the relative position information of the physical point corresponding to each feature point relative to the corresponding three-dimensional image capturing component, and the relative posture information between the pre-stored reference three-dimensional image capturing component and other three-dimensional image capturing components in the at least two pre-stored three-dimensional image capturing components, the relative position information of the physical point corresponding to each feature point relative to the reference three-dimensional image capturing component is determined as the relative position information of the physical point corresponding to each feature point relative to the movable device.

[0130] Optionally, the determining module 510 is configured to:

[0131] Determine the three-dimensional position information of each feature point relative to the image currently captured by the reference three-dimensional image capturing component according to the three-dimensional position information of the feature point currently determined and the pre-stored relative position information between the reference three-dimensional image capturing component and other three-dimensional image capturing components;

[0132] Based on the relative position information of the physical point corresponding to each feature point relative to the reference three-dimensional image shooting component, and the three-dimensional position information of each feature point relative to the image currently shot by the reference three-dimensional image shooting component, the posture change information of the movable device relative to the initial position is determined.

[0133] Optionally, the determining module 510 is further configured to:

[0134] Determine, among the feature points contained in the image currently captured by each 3D image capturing component, a target feature point that matches the image feature of any feature point among the at least one feature point, and determine the 3D position information of each target feature point;

[0135] According to the relative position information corresponding to the determined physical point, the three-dimensional position information of each target feature point and the pre-stored relative posture information between different three-dimensional image shooting components, the posture change information of the movable device relative to the initial position is determined.

[0136] Optionally, the determining module 510 is configured to:

[0137] When the number of the target feature points is greater than or equal to a preset threshold value, the posture change information of the movable device relative to the initial position is determined based on the relative position information corresponding to the determined physical points, the three-dimensional position information of each target feature point and the relative posture information between different pre-stored three-dimensional image shooting components.

[0138] Optionally, the determining module 510 is further configured to:

[0139] When the number of the target feature points is less than a preset threshold value, the current position is set as the initial position, and the relative position information of the physical point corresponding to each feature point relative to the movable device is determined based on the three-dimensional position information of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is at the reset initial position.

[0140] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0141] Through the device provided by the embodiment of the present disclosure, the information on the change in the posture of the movable device relative to the initial position can be determined through images captured by at least two three-dimensional image capturing components. Even if one of the three-dimensional image capturing components is affected by strong light or other factors, the information on the change in the posture of the movable device relative to the initial position can be determined with the help of images captured by the remaining three-dimensional image capturing components. Furthermore, the information on the change in posture can be reliably determined under harsh environmental conditions.

[0142] It should be noted that: the device for determining the posture change information of a movable device provided in the above embodiment only uses the division of the above functional modules as an example when determining the posture change information of a movable device. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the server is divided into different functional modules to complete all or part of the functions described above. In addition, the device for determining the posture change information of a movable device provided in the above embodiment and the method embodiment for determining the posture change information of a movable device belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0143] Another exemplary embodiment of the present disclosure provides a system for determining posture change information of a movable device, the system comprising a server and a movable device, the movable device comprising at least two three-dimensional image capturing components, wherein:

[0144] The movable device is used to capture images through each 3D image capturing component whenever a preset period arrives, and send the images captured by each 3D image capturing component to the server;

[0145] The server is used to determine the relative position information of the physical point corresponding to each feature point relative to the movable device based on the three-dimensional position information of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is in the initial position; during the movement of the movable device, in the image currently captured by each three-dimensional image capturing component, determine the three-dimensional position information of the feature points contained in each image; and determine the posture change information of the movable device relative to the initial position based on the relative position information corresponding to the determined physical point, the three-dimensional position information of the currently determined feature point and the pre-stored relative posture information between different three-dimensional image capturing components.

[0146] Regarding the system in the above embodiment, the specific manner in which the mobile device and the server perform operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0147] Figure 6The schematic diagram of the structure of a server 1900 provided by an exemplary embodiment of the present disclosure is shown. The server 1900 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 1910 and one or more memories 1920. The memory 1920 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 1910 to implement the method for determining the posture change information of the movable device described in the above embodiment.

[0148] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0149] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for determining posture change information of a movable device, characterized in that: The movable device comprises at least two three-dimensional image capturing components, and the method comprises: Determine relative position information of a physical point corresponding to each feature point relative to the corresponding three-dimensional image capturing component according to three-dimensional position information of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is at the initial position; Determine the relative position information of the physical point corresponding to each feature point relative to the corresponding 3D image capturing component according to the relative position information of the physical point corresponding to each feature point relative to the corresponding 3D image capturing component and the relative posture information between the pre-stored reference 3D image capturing component and other 3D image capturing components in the at least two pre-stored 3D image capturing components, as the relative position information of the physical point corresponding to each feature point relative to the movable device; During the movement of the movable device, in the images currently captured by each 3D image capturing component, determining the 3D position information of the feature points contained in each image; According to the relative position information corresponding to the determined physical point, the three-dimensional position information of the currently determined feature point and the pre-stored relative posture information between different three-dimensional image shooting components, the posture change information of the movable device relative to the initial position is determined.

2. The method according to claim 1, characterized in that The determining of the posture change information of the movable device relative to the initial position according to the relative position information corresponding to the determined physical point, the three-dimensional position information of the currently determined feature point and the pre-stored relative posture information between different three-dimensional image capturing components comprises: Determine the three-dimensional position information of each feature point relative to the image currently captured by the reference three-dimensional image capturing component according to the three-dimensional position information of the currently determined feature point and the pre-stored relative position information between the reference three-dimensional image capturing component and other three-dimensional image capturing components; Based on the relative position information of the physical point corresponding to each feature point relative to the reference three-dimensional image shooting component, and the three-dimensional position information of each feature point relative to the image currently shot by the reference three-dimensional image shooting component, the posture change information of the movable device relative to the initial position is determined.

3. The method according to claim 1, characterized in that The method further comprises: Determine, among the feature points contained in the image currently captured by each 3D image capturing component, a target feature point that matches the image feature of any feature point among the at least one feature point, and determine the 3D position information of each target feature point; The determining of the posture change information of the movable device relative to the initial position according to the relative position information corresponding to the determined physical point, the three-dimensional position information of the currently determined feature point and the pre-stored relative posture information between different three-dimensional image capturing components comprises: According to the relative position information corresponding to the determined physical point, the three-dimensional position information of each target feature point and the pre-stored relative posture information between different three-dimensional image shooting components, the posture change information of the movable device relative to the initial position is determined.

4. The method according to claim 3, characterized in that The step of determining the posture change information of the movable device relative to the initial position according to the relative position information corresponding to the determined physical point, the three-dimensional position information of each target feature point, and the pre-stored relative posture information between different three-dimensional image capturing components includes: If the number of the target feature points is greater than or equal to a preset threshold value, the posture change information of the movable device relative to the initial position is determined based on the relative position information corresponding to the determined physical points, the three-dimensional position information of each target feature point and the relative posture information between different pre-stored three-dimensional image shooting components.

5. The method according to claim 4, characterized in that The method further comprises: If the number of the target feature points is less than a preset threshold value, the current position is set as the initial position, and the relative position information of the physical point corresponding to each feature point relative to the movable device is determined based on the three-dimensional position information of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is at the reset initial position.

6. A device for determining position change information of a movable device, characterized in that: The movable device comprises at least two three-dimensional image capturing components, and the device comprises: a determination module, configured to determine, based on the three-dimensional position information of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is at the initial position, the relative position information of the physical point corresponding to each feature point relative to the corresponding three-dimensional image capturing component; and to determine, based on the relative position information of the physical point corresponding to each feature point relative to the corresponding three-dimensional image capturing component and the relative posture information between the pre-stored reference three-dimensional image capturing component and other three-dimensional image capturing components in the at least two pre-stored three-dimensional image capturing components, the relative position information of the physical point corresponding to each feature point relative to the reference three-dimensional image capturing component as the relative position information of the physical point corresponding to each feature point relative to the movable device; The determination module is further used to determine the three-dimensional position information of the feature points contained in each image currently captured by each three-dimensional image capturing component during the movement of the movable device; The determination module is also used to determine the posture change information of the movable device relative to the initial position based on the relative position information corresponding to the determined physical point, the three-dimensional position information of the currently determined feature point and the pre-stored relative posture information between different three-dimensional image shooting components.

7. The device according to claim 6, characterized in that The determining module is used to: Determine the three-dimensional position information of each feature point relative to the image currently captured by the reference three-dimensional image capturing component according to the three-dimensional position information of the currently determined feature point and the pre-stored relative position information between the reference three-dimensional image capturing component and other three-dimensional image capturing components; Based on the relative position information of the physical point corresponding to each feature point relative to the reference three-dimensional image shooting component, and the three-dimensional position information of each feature point relative to the image currently shot by the reference three-dimensional image shooting component, the posture change information of the movable device relative to the initial position is determined.

8. The device according to claim 6, characterized in that The determining module is further used for: Determine, among the feature points contained in the image currently captured by each 3D image capturing component, a target feature point that matches the image feature of any feature point among the at least one feature point, and determine the 3D position information of each target feature point; According to the relative position information corresponding to the determined physical point, the three-dimensional position information of each target feature point and the pre-stored relative posture information between different three-dimensional image shooting components, the posture change information of the movable device relative to the initial position is determined.

9. The device according to claim 8, characterized in that The determining module is used to: When the number of the target feature points is greater than or equal to a preset threshold value, the posture change information of the movable device relative to the initial position is determined based on the relative position information corresponding to the determined physical points, the three-dimensional position information of each target feature point and the relative posture information between different pre-stored three-dimensional image shooting components.

10. The device according to claim 9, characterized in that The determining module is further used for: When the number of the target feature points is less than a preset threshold value, the current position is set as the initial position, and the relative position information of the physical point corresponding to each feature point relative to the movable device is determined based on the three-dimensional position information of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is at the reset initial position.

11. A system for determining position change information of a movable device, characterized in that: The system comprises a server and a movable device, wherein the movable device comprises at least two three-dimensional image capturing components, wherein: The movable device is used to capture images through each 3D image capturing component whenever a preset period arrives, and send the images captured by each 3D image capturing component to the server; The server is used to determine the relative position information of the physical point corresponding to each feature point relative to the corresponding three-dimensional image capturing component based on the three-dimensional position information of at least one feature point in the image captured by each three-dimensional image capturing component when the movable device is at the initial position; determine the relative position information of the physical point corresponding to each feature point relative to the corresponding three-dimensional image capturing component based on the relative position information of the physical point corresponding to each feature point relative to the corresponding three-dimensional image capturing component and the relative posture information between the pre-stored reference three-dimensional image capturing component and other three-dimensional image capturing components in the at least two pre-stored three-dimensional image capturing components as the relative position information of the physical point corresponding to each feature point relative to the movable device; determine the three-dimensional position information of the feature points contained in each image currently captured by each three-dimensional image capturing component during the movement of the movable device; determine the posture change information of the movable device relative to the initial position based on the relative position information corresponding to the determined physical point, the three-dimensional position information of the currently determined feature point and the pre-stored relative posture information between different three-dimensional image capturing components.

Images