Motion state determination method, device, equipment and storage medium

Abstract

The embodiments of the present disclosure disclose a method, apparatus, device and storage medium for determining a motion state, wherein the method comprises: acquiring a current image of an area of ​​a preset orientation of a device; determining a second key pixel point matching the first key pixel point from the current image based on a first key pixel point of a previous frame image acquired; determining a first current motion feature of the device based on the first key pixel point and the second key pixel point; acquiring a speed of a driving component of the device, and determining a second current motion feature of the device based on the speed; and determining a current motion state of the device based on the first current motion feature and the second current motion feature.

Description

Technical Field

[0001] The present disclosure relates to, but is not limited to, the field of computer vision technology, and in particular to a motion state determination method, apparatus, device, and storage medium. Background Art

[0002] Abnormal motion detection is an important problem in the field of computer vision and robotic motion. Abnormal detection technology has important applications in many fields, such as robot positioning, map construction (hereinafter referred to as mapping), and path planning.

[0003] In the related art, vision-based anomaly detection often needs to be based on sufficient mapping. In the case of no initial scene structure or mapping failure, abnormal motion cannot be distinguished. In this case, it is often necessary to add additional sensors, such as collision detectors, inertial measurement units, time-of-flight cameras, etc.; this will increase the size of the equipment and the complexity of the system. In addition, abnormal motion detection based on scene structure information is also prone to produce erroneous abnormal detection results. In other words, the abnormal motion detection method in the related art is not flexible enough, and the detection results are not accurate enough. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure at least provide a motion state determination method, apparatus, device and storage medium.

[0005] The technical solution of the embodiment of the present disclosure is implemented as follows:

[0006] In one aspect, an embodiment of the present disclosure provides a method for determining a motion state, the method comprising:

[0007] Obtaining a current image of an area at a preset orientation of the device;

[0008] Determine, from the current image, a second key pixel point that matches the first key pixel point, according to the first key pixel point of the acquired previous frame image;

[0009] Determining a first current motion feature of the device based on the first key pixel point and the second key pixel point;

[0010] acquiring a speed of a driving component of the device, and determining a second current motion characteristic of the device based on the speed;

[0011] Based on the first current motion feature and the second current motion feature, a current motion state of the device is determined.

[0012] On the other hand, an embodiment of the present disclosure provides a motion state determination device, the device comprising:

[0013] An acquisition module, used to acquire a current image of an area at a preset orientation of the device;

[0014] A determination module, configured to determine, from the current image, a second key pixel point that matches the first key pixel point acquired in the previous frame of image; and determine a first current motion feature of the device based on the first key pixel point and the second key pixel point;

[0015] The acquisition module is further used to acquire the speed of the driving component of the device;

[0016] The determination module is further used to determine a second current motion feature of the device based on the speed; and determine a current motion state of the device based on the first current motion feature and the second current motion feature.

[0017] On the other hand, an embodiment of the present disclosure provides a computer device, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements some or all of the steps in the above method when executing the program.

[0018] On the other hand, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which implements part or all of the steps in the above method when executed by a processor.

[0019] On the other hand, an embodiment of the present disclosure provides a computer program, including computer-readable codes. When the computer-readable codes are executed in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0020] On the other hand, an embodiment of the present disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, it implements some or all of the steps in the above method.

[0021] In the embodiment of the present disclosure, by acquiring a current image of an area of ​​a preset orientation of the device itself, a second key pixel matching the first key pixel is determined from the current image according to the first key pixel of the previous frame of the acquired current image, and a first current motion feature of the device is obtained according to the obtained first key pixel and second key pixel that match each other; in this way, the first current motion feature of the device can be obtained without constructing map information of the area, and then the first current motion state represented by the first current motion feature can be obtained; by acquiring the speed of the driving component of the device, the second current motion feature of the device is determined according to the speed; in this way, the second current motion feature of the device can be obtained, and then the second current motion state represented by the second current motion feature can be obtained; and by determining the current motion state of the device according to the first current motion feature and the second current motion feature, the current correct motion state of the device can be finally determined according to the first current motion state and the second current motion state of the device, so that the real-time motion state of the device can be accurately determined without constructing map information of the area and without increasing the volume of the device and the complexity of the system; finally, the flexibility and accuracy in judging the motion state of the device are improved.

[0022] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0024] Figure 1 A schematic diagram of an implementation flow of a motion state determination method provided in an embodiment of the present disclosure;

[0025] Figure 2 A schematic diagram of an implementation flow of a motion state determination method provided in an embodiment of the present disclosure;

[0026] Figure 3 A schematic diagram of an implementation flow of a motion state determination method provided in an embodiment of the present disclosure;

[0027] Figure 4 A schematic diagram of an implementation flow of a motion state determination method provided in an embodiment of the present disclosure;

[0028] Figure 5 A schematic diagram of an implementation flow of a motion state determination method provided in an embodiment of the present disclosure;

[0029] Fig. 6AAn exemplary schematic diagram of a principle for determining a second relative displacement provided in an embodiment of the present disclosure;

[0030] Figure 6B Another exemplary schematic diagram of determining a second relative displacement provided in an embodiment of the present disclosure;

[0031] Figure 7 A schematic diagram of an implementation flow of a motion state determination method provided in an embodiment of the present disclosure;

[0032] Figure 8 A schematic diagram of an implementation flow of a motion state determination method provided in an embodiment of the present disclosure;

[0033] Fig. 9 A schematic diagram of an implementation flow of a motion state determination method provided in an embodiment of the present disclosure;

[0034] Fig.10 A schematic diagram of an implementation flow of a motion state determination method provided in an embodiment of the present disclosure;

[0035] Fig.11 A schematic diagram of an implementation flow of a motion state determination method provided in an embodiment of the present disclosure;

[0036] Fig.12 A schematic diagram of an exemplary implementation flow of determining a target pose based on 2D-2D valid matching points provided in an embodiment of the present disclosure;

[0037] Fig.13 A schematic diagram of the structure of a motion state determination device provided by an embodiment of the present disclosure;

[0038] Fig.14 A hardware entity schematic diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present disclosure. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0040] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0041] The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first / second / third" can be interchanged with a specific order or sequence where permitted so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing the present disclosure and are not intended to limit the present disclosure.

[0043] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described first. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations.

[0044] 1) Two-dimensional, the full English name is two-dimension, abbreviated as 2D or 2d. Graphics or images on a plane are usually two-dimensional.

[0045] 2) The essential matrix is ​​a 3*3 matrix containing the epipolar geometric relationship, which reflects the relationship between the image point of a point P in space and its representation in the camera coordinate system under different viewing angles.

[0046] In order to better understand the motion state determination method provided by the embodiment of the present disclosure, the abnormal motion detection method used in the related art is first described below.

[0047] In the related art, anomaly detection based on scene structure information requires first constructing scene structure information (i.e., scene map information), and then detecting abnormal motion based on the constructed scene map information. This anomaly detection method is not flexible enough on the one hand; on the other hand, it leads to the problem that small errors may be magnified in complex situations; for example, if abnormal motion occurs when scene map information is not constructed and the scene map information is being constructed, the abnormal motion cannot be discovered in time, resulting in errors in the constructed map information itself. When abnormal motion detection is subsequently performed based on the erroneous map information, it is easy to cause misdetection or missed detection.

[0048] The embodiment of the present disclosure provides a method for determining a motion state, which can be executed by a processor of a computer device. The computer device can be a mobile robot, or a device with data processing capabilities such as a laptop, a tablet computer, a desktop computer, and a server; and can also be implemented as a server, wherein the server can be a server connected to the mobile robot, or an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. The following will take the computer device as an example of a mobile robot to illustrate the technical solution of the embodiment of the present disclosure.

[0049] Figure 1 A schematic diagram of an implementation flow of a motion state determination method provided by an embodiment of the present disclosure, such as Figure 2 As shown, the method includes S101 to S105.

[0050] S101: Acquire a current image of an area at a preset orientation of a device.

[0051] In the disclosed embodiment, the mobile robot can collect images of its own preset area in real time through its own image acquisition device, or the mobile robot can also obtain images of its own preset area in real time from other devices connected to itself.

[0052] In some embodiments, the mobile robot can obtain a first current image of the upper area of ​​itself through a first image acquisition device deployed in a first preset area of ​​itself. The first preset area can be any area of ​​the mobile robot, as long as the image of the upper area of ​​itself can be acquired by the first image acquisition device deployed in the first preset area. The first preset area is not specifically limited in the embodiment of the present disclosure. Exemplarily, the first preset area can be the top area of ​​the mobile robot, so that the mobile robot can acquire the image of the upper area of ​​itself through the first image acquisition device deployed in the top area of ​​itself. For example, in the case where the mobile robot is a sweeping robot, the image of the ceiling area above the sweeping robot can be acquired by a camera installed on the top of the sweeping robot. Here, since the upper area of ​​the mobile robot itself is less affected by the dynamic environment and has fewer obstructions, the image of the upper area of ​​the mobile robot itself is used to determine the first current motion feature, which can improve the accuracy of the obtained first current motion feature.

[0053] In some embodiments, the mobile robot can obtain a second current image of the area in front of itself along its direction of movement through a second image acquisition device deployed in its second preset area. Similarly, the second preset area can be any area of ​​the mobile robot, as long as the image of the area in front of the mobile robot along the direction of movement of the mobile robot can be acquired through the second image acquisition device deployed in the second preset area. The second preset area is not specifically limited in the embodiments of the present disclosure. Exemplarily, the second preset area can be the front area of ​​the mobile robot, so that the computer device can acquire the image of the area in front of the mobile robot along the direction of movement of the mobile robot through the image acquisition device deployed in the front area of ​​the mobile robot. For example, in the case where the mobile robot is a food delivery robot, the image of the area in front of the food delivery robot along the direction of movement of the food delivery robot can be acquired through a camera installed in the front area of ​​the body of the food delivery robot.

[0054] S102: According to the acquired first key pixel point of the previous frame image, determine a second key pixel point matching the first key pixel point from the current image.

[0055] In the disclosed embodiment, when the mobile robot is collecting images of an area at a preset position of itself in real time, each time a frame of image is collected, a second key pixel point matching the first key pixel point can be determined from the current frame of image based on the first key pixel point of the previous frame of image acquired.

[0056] In some embodiments, the mobile robot can also obtain a video stream by extracting a video frame from the video stream as the current image, and when the current image is obtained, continuing to extract the previous frame image or the previous N-th frame image of the current image from the video stream to obtain the previous frame image of the current image frame, and then, based on the first key pixel point of the previous frame image of the current image obtained, determine the second key pixel point that matches the first key pixel point from the current image.

[0057] Here, in the case where the current image is the first frame of the acquired image, the mobile robot can perform key point extraction processing on the image through a key point extraction algorithm, thereby extracting the first key pixel point of the image.

[0058] In some embodiments, the mobile robot can use the Fast (Features from Accelerated Segments Test, Fast) algorithm to extract feature points (first key pixel points) in the image, so as to quickly obtain the first key pixel points in the image. Through the Fast algorithm, the mobile robot can select any point p in the image, obtain its pixel (brightness) value Ip, and draw a circle with point p as the center and a radius of 3 to obtain 16 pixels on the circle; define a threshold, calculate the pixel difference between p1 and p9 of the 16 pixels and the center p, if their absolute values ​​are all less than the threshold, then point p cannot be a feature point and is directly discarded; otherwise, point p is regarded as a candidate point for the next step of judgment; if p is a candidate point, calculate the pixel difference between p1, p5, p9, p13 of the 16 pixels and the center p, if at least 3 of their absolute values ​​exceed the threshold, then they are regarded as candidate points for the next step of judgment; otherwise, they are directly discarded; if p is a candidate point point, calculate the pixel difference between the 16 points p1 to p16 and the center p. If at least 9 of them exceed the threshold, they are feature points; otherwise, they are directly discarded; then, determine whether there are multiple feature points in a neighborhood (such as 3x3 or 5x5) centered on the feature point p. If there are multiple feature points, calculate the FAST score of each feature point respectively (that is, calculate the s value, and the s value is the sum of the absolute values ​​of the differences between these 16 points and the center p). If p has the largest s value among all the feature points in the neighborhood, retain this feature point, otherwise discard it; if there is only one feature point in the neighborhood, retain this feature point and do not participate in other calculations; in this way, the first key pixel point of the image can be obtained.

[0059] In some embodiments, the mobile robot may also use an Oriented Fast and Rotated Brief (ORB) algorithm to extract feature points in the image, thereby accurately obtaining the first key pixel in the image and the descriptor of each first key pixel. When the mobile robot uses the ORB algorithm, it may first obtain the first key pixel in the image through the Fast algorithm, then calculate the rotation value corresponding to each first key pixel, and then calculate the 256-dimensional BRIEF descriptor of each first key pixel based on the rotation value.

[0060] Here, the mobile robot can obtain multiple groups of mutually matching pixel pairs after determining the second key pixel point that matches the first key pixel point in the current image based on the first key pixel point of the previous frame image acquired, and each group of mutually matching pixel pairs includes a first key pixel point and a second key pixel point.

[0061] In some embodiments, the mobile robot can also identify all objects in the previous frame of the image through its own target recognition module, and use the pixel points corresponding to the area where all objects are located as the first key pixel points.

[0062] In the disclosed embodiment, when the mobile robot obtains a first key pixel point from a previous frame of image, it can use a feature tracking algorithm to determine a second key pixel point that matches the first key pixel point from the current image.

[0063] In some embodiments, the mobile robot can use the optical flow method to determine the second key pixel point that matches the first key pixel point from the current image. The optical flow method is a method that uses the change of pixels in the time domain in the image sequence and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, thereby calculating the motion information of the object between adjacent frames. Through the optical flow method, the mobile robot can use the first key pixel point as the feature point to be tracked, and find the pixel point corresponding to the first key pixel point in the current image, so as to obtain the second key pixel point of the current image.

[0064] In some embodiments, the mobile robot can use a k-nearest neighbor matching algorithm to determine a second key pixel point that matches the first key pixel point from the second image. Through k-nearest neighbor matching, the mobile robot can select k (usually k=2) points that are most similar to each first key pixel point from the second image, calculate the Euclidean distance between the descriptor of the first key pixel point and the descriptor of the most similar similar point among the two most similar points, and calculate the Euclidean distance between the descriptor of the first key pixel point and the descriptor of another similar point, and when the ratio between the two Euclidean distances is greater than a threshold, select the most similar point as the matching point of the first key pixel point; in this way, the second key pixel point of the current image is obtained.

[0065] S103: Determine a first current motion feature of the device based on the first key pixel and the second key pixel.

[0066] In some embodiments, the mobile robot can obtain an intrinsic matrix for characterizing the epipolar geometric relationship between the first key pixel and the second key pixel based on the first key pixel and the second key pixel, obtain multiple groups of decomposition values ​​by performing matrix decomposition on the intrinsic matrix, and obtain a target decomposition value based on the multiple groups of decomposition values, and then obtain the first current motion feature of the mobile robot corresponding to the time period of the previous frame image and the current image according to the target decomposition value, wherein the first current motion feature characterizes the first current motion state of the mobile robot corresponding to this time period. For example, the acquisition time of the previous frame image is T1, and the acquisition time of the current image is T2, so the first current motion feature of the mobile robot corresponding to the time period from T1 to T2 can be determined based on the first key pixel and the second key pixel.

[0067] In some embodiments, the first key pixel point is a pixel point corresponding to the area where all objects in the previous frame image are located; the second key pixel point is a pixel point corresponding to the area where all objects in the current image are located. The mobile robot can identify the target object in the area corresponding to the first key pixel point through its own target recognition module, and construct an image coordinate system, and use the coordinates of the center of the area where the identified target object is located as the first position corresponding to the first time for shooting the previous frame image, and identify the target object in the area corresponding to the second key pixel point through its own target recognition module, and construct an image coordinate system, and use the coordinates of the center of the area where the identified target object is located as the second position corresponding to the second time for shooting the current image; then, the mobile robot can establish a spatial three-dimensional coordinate system corresponding to the robot, and through the conversion relationship between the image coordinate system and the spatial three-dimensional coordinate system, convert the first position and the second position into position coordinates in the spatial three-dimensional coordinate system; then, according to the two position coordinates in the spatial three-dimensional coordinate system, calculate at least one of the moving displacement and the moving direction of the mobile robot in the time period from the first time to the second time, so as to obtain the first current motion feature of the mobile robot.

[0068] Here, the target object may generally be an object with a fixed position, such as a lamp on the ceiling, a hanging item on the wall, etc., or may be other objects with small position changes, which is not limited in the embodiments of the present disclosure.

[0069] In some embodiments, the first current motion feature may be a first relative displacement; thus, the movable robot may obtain a moving distance and a moving direction of itself within the time period between capturing the previous frame image and the current image.

[0070] In some embodiments, the first current motion feature may be a moving distance, so that the movable robot can obtain a moving distance of itself within the time period between capturing the last frame image and the current image.

[0071] In some embodiments, the first current motion feature may be a moving direction, so that the movable robot can obtain a moving direction of itself within the time period between capturing the last frame of image and the current image.

[0072] S104: Acquire the speed of a driving component of the device, and determine a second current motion characteristic of the device based on the speed.

[0073] In the disclosed embodiment, the driving component may be a component that can drive the movable robot to move, for example, a rotatable component such as a wheel or a movable component such as a track. Exemplarily, the movable robot has at least two wheels and moves by rotating the wheels. The movable robot can obtain the speed of each wheel of the movable robot in real time.

[0074] In some embodiments, the speed of the driving component may be an angular velocity or a linear velocity, etc.

[0075] In some embodiments, the mobile robot can detect the speed of each wheel of the mobile robot in real time through a wheel speed meter installed on the mobile robot.

[0076] In some embodiments, the mobile robot can detect the speed of movable parts such as tracks through self-installed or external sensors or other measuring components; or calculate the speed of movable parts such as tracks through parameters of components such as motors that drive the tracks. In the disclosed embodiment, the mobile robot can obtain the speed within the time period between the previous frame image and the current image according to each of its own driving components, and determine the position change information of the mobile robot according to the speed and the time between the previous frame image and the current image, and determine the second current motion feature corresponding to the time period according to the position change information, wherein the second current motion feature represents the second current motion state of the mobile robot corresponding to the time period.

[0077] In some embodiments, the second current motion feature may be a second relative displacement; thereby, the movable robot may obtain another moving distance and another moving direction of itself within the time period between capturing the previous image and the current image, and may subsequently determine the current motion state of the device based on the first relative displacement and the second relative displacement when the first current motion feature is the second relative displacement.

[0078] In some embodiments, the second current motion feature may be a moving distance, so that the movable robot can obtain another moving distance of itself within the time period between capturing the previous image and the current image, and when the first current motion feature is a moving distance, the current motion state of the device can be subsequently determined based on the two moving distances.

[0079] In some embodiments, the second current motion feature may be a moving direction, so that the movable robot can obtain another moving direction of itself within the time period between capturing the previous image and the current image, and can subsequently determine the current motion state of the device based on the two moving directions when the first current motion feature is the moving direction.

[0080] S105. Determine a current motion state of the device based on the first current motion feature and the second current motion feature.

[0081] In the disclosed embodiment, the mobile robot can comprehensively determine its motion state during the time period between capturing the previous image and the current image based on the obtained first current motion feature and the second current motion feature; for example, whether it is in a normal motion state or an abnormal motion state.

[0082] Here, when the current motion feature is relative displacement, the movable robot can determine its current motion state based on the two obtained moving distances and two moving directions.

[0083] In some embodiments, the movable robot may determine that it is currently in a normal motion state when it determines that the first current motion feature and the second current motion feature are consistent.

[0084] Here, the normal motion state may be a normal moving state or a normal paused moving state, etc.

[0085] In some embodiments, the movable robot may determine that it is currently in an abnormal motion state when it determines that the first current motion feature and the second current motion feature are inconsistent.

[0086] Here, the abnormal motion state may be a state such as a collision, wheel spinning, or slipping along an obstacle.

[0087] In the embodiment of the present disclosure, by acquiring a current image of an area of ​​a preset orientation of the device itself, a second key pixel matching the first key pixel is determined from the current image according to the first key pixel of the previous frame of the acquired current image, and a first current motion feature of the device is obtained according to the obtained first key pixel and second key pixel that match each other; in this way, the first current motion feature of the device can be obtained without constructing map information of the area, and then the first current motion state represented by the first current motion feature can be obtained; by acquiring the speed of the driving component of the device, the second current motion feature of the device is determined according to the speed; in this way, the second current motion feature of the device can be obtained, and then the second current motion state represented by the second current motion feature can be obtained; and by determining the current motion state of the device according to the first current motion feature and the second current motion feature, the current correct motion state of the device can be finally determined according to the first current motion state and the second current motion state of the device, so that the real-time motion state of the device can be accurately determined without constructing map information of the area and without increasing the volume of the device and the complexity of the system; finally, the flexibility and accuracy in judging the motion state of the device are improved.

[0088] In some embodiments, Figure 2 As shown, the above S103 can be implemented by S1031-S1032. Figure 2 Take this as an example to illustrate.

[0089] S1031 , obtaining a first target pixel and a second target pixel by filtering the first key pixel and the second key pixel.

[0090] S1032: Determine a first current motion feature of the device based on the first target pixel and the second target pixel.

[0091] In some embodiments, the movable robot can obtain an essential matrix for characterizing the epipolar geometric relationship between the first target pixel point and the second target pixel point based on the first target pixel point and the second target pixel point, obtain multiple groups of decomposition values ​​by performing matrix decomposition on the essential matrix, and obtain a group of target grading values ​​based on the multiple groups of decomposition values, and obtain its own first current motion feature corresponding to the time period between the previous frame image and the current image based on the target decomposition value.

[0092] In some embodiments, the first target pixel point is a pixel point corresponding to the area where all objects in the previous frame image are located; the second target pixel point is a pixel point corresponding to the area where all objects in the current image are located. The mobile robot can identify the target object in the area corresponding to the first target pixel point through its own target recognition module, and construct an image coordinate system, and use the coordinates of the center of the area where the identified target object is located as the first position corresponding to the first time for shooting the previous frame image, and identify the target object in the area corresponding to the second target pixel point through its own target recognition module, and construct an image coordinate system, and use the coordinates of the center of the area where the identified target object is located as the second position corresponding to the second time for shooting the current image; then, the mobile robot can establish a spatial three-dimensional coordinate system corresponding to the robot, and through the conversion relationship between the image coordinate system and the spatial three-dimensional coordinate system, convert the first position and the second position into position coordinates in the spatial three-dimensional coordinate system; then, according to the two position coordinates in the spatial three-dimensional coordinate system, calculate at least one of the moving displacement and the moving direction of the mobile robot in the time period from the first time to the second time, so as to obtain the first current motion feature of the mobile robot.

[0093] Here, the movable robot can filter the first key pixel point and the second key pixel point obtained by a feature filtering algorithm, and use the first key pixel point obtained after filtering as the first target pixel point, and use the second key pixel point obtained after filtering as the second target key pixel point.

[0094] In some embodiments, the mobile robot may use a random sample consensus (RANSAC) algorithm to filter the obtained first key pixel points and second key pixel points. For example, the mobile robot may randomly select a sample subset from all first key pixel points and second key pixel points that match each other through the algorithm, and use the selected sample subset to calculate the model parameters through the least square method, and then calculate the deviation between each key pixel point and the model parameter among the first key pixel points and the second key pixel points that match each other, and compare the obtained deviation with a preset threshold value, when the deviation is less than the threshold value, the pixel point belongs to the inliers of the model, otherwise, the pixel point belongs to the outliers of the model, and multiple inliers may be obtained by this method, and the model parameters are recalculated using the obtained multiple inliers, and the above method is continued to be executed until after a certain number of iterations, the model parameter with the largest number of inliers is used as the final model parameter, and the first key pixel point belonging to the model parameter is used as the first target pixel point, and the second key pixel point belonging to the model parameter is used as the second target pixel point.

[0095] In the embodiment of the present disclosure, by filtering the first key pixel points and the second key pixel points to obtain the first target pixel points and the second target pixel points, the outlier first key pixel points and the second key pixel points can be filtered out, and the valid first key pixel points and the second key pixel points can be retained, thereby improving the accuracy of the obtained first target pixel points and the second target pixel points, which is ultimately beneficial to improving the accuracy of the first current motion feature obtained based on the first target pixel points and the second target pixel points.

[0096] In some embodiments, Figure 3 As shown, the above S103 can also be implemented through S1033 to S1035. Figure 3 Take this as an example to illustrate.

[0097] S1033. Obtain an essential matrix based on the first key pixel and the second key pixel; the essential matrix is ​​used to characterize the epipolar geometric relationship between the first key pixel and the second key pixel.

[0098] In the disclosed embodiment, the mobile robot may calculate an essential matrix for characterizing the epipolar geometric relationship between the first key pixel and the second key pixel based on the first key pixel and the second key pixel.

[0099] In some embodiments, the mobile robot may use a graph optimization method to obtain an essential matrix according to the first key pixel point and the second key pixel point. When the graph optimization method is used to obtain the essential matrix, the mobile robot can pre-construct an optimization model with the essential matrix and the direction vector of the relative displacement as unknowns. For each pair of matching first key pixel points and second key pixel points, an epipolar constraint is used (i.e., in a pair of matching first key pixel points and second key pixel points, the second key pixel point is on the epipolar line relative to the first key pixel point; that is, from the first camera perspective used to shoot the previous frame image where the first key pixel point is located, the line of sight along the first key pixel point is projected on the second camera perspective used to shoot the current image where the second key pixel point is located, which will cover the second key pixel point) as a solution condition of the optimization model; and, multiplying the essential matrix by the transpose of the essential matrix, which is equal to multiplying the antisymmetric matrix corresponding to the direction vector of the relative displacement by the transpose of the antisymmetric matrix corresponding to the direction vector of the relative displacement, as another solution condition, and taking the direction vector of the relative displacement as a unit vector as another solution condition; according to these three solution conditions, an optimal essential matrix that meets each condition under a certain error can be solved, and the optimal essential matrix is ​​used as the obtained essential matrix.

[0100] S1034. Perform matrix decomposition processing on the essential matrix to obtain multiple groups of different decomposition values; each group of decomposition values ​​includes a relative displacement and a relative rotation angle.

[0101] In the disclosed embodiment, the mobile robot can decompose the obtained essential matrix through the singular value decomposition (SVD) method, and obtain 4 groups of different decomposition values, and each group of decomposition values ​​includes a relative displacement and a relative rotation angle.

[0102] S1035. Determine a first relative displacement according to relative displacements and relative rotation angles included in multiple groups of different decomposition values; the first current motion feature includes: a first relative displacement.

[0103] In the embodiment of the present disclosure, when a plurality of different groups of decomposition values ​​are obtained, the movable robot may determine the first relative displacement according to the relative displacement and the relative rotation angle included in the decomposition values.

[0104] In some embodiments, the movable robot may select a set of target decomposition values ​​from multiple sets of decomposition values, and use the relative displacement included in the target decomposition values ​​as the first relative displacement.

[0105] In the disclosed embodiment, the above method can estimate the first relative displacement of the movable robot through the key points of the mutually matched adjacent images, thereby improving the intelligence of the movable robot.

[0106] In some embodiments, the above S1032 may also be implemented by the methods described in the above S1033 to S1035, wherein the first key pixel corresponds to the first target pixel, and the second key pixel corresponds to the second target pixel.

[0107] In some embodiments, Figure 4 As shown, the above S1035 can also be implemented through S301 to S305. Figure 4 Take this as an example to illustrate.

[0108] S301. Select a set of target decomposition values ​​from multiple sets of decomposition values, and use the direction of the relative displacement included in the target decomposition values ​​as the first direction; the first current motion feature includes: a first relative displacement; the first relative displacement includes a first direction and a first distance.

[0109] In some embodiments, in order to improve the selection efficiency, the movable robot may randomly select a set of decomposition values ​​from multiple sets of decomposition values ​​as the target decomposition values.

[0110] In some embodiments, in order to improve selection efficiency, the mobile robot can also arbitrarily select a first key pixel point from all the first key pixel points of the previous frame image, and select a set of target decomposition values ​​from multiple groups of decomposition values ​​based on the selected first key pixel point and the relative rotation angle included in each group of decomposition values.

[0111] In some embodiments, in order to improve the accuracy of the selected target decomposition value, the mobile robot can arbitrarily select multiple first key pixel points from all the first key pixel points of the previous frame image, and select a group of target decomposition values ​​from the multiple groups of decomposition values ​​based on the selected multiple first key pixel points and the relative rotation angle included in each group of decomposition values.

[0112] S302. Rotate each first key pixel point using the relative rotation angle included in the target decomposition value to obtain a first rotated pixel point corresponding to each first key pixel point; the first rotated pixel point corresponds to a second key pixel point that matches each first key pixel point.

[0113] In the disclosed embodiment, for each first key pixel point, the movable robot can rotate the first key pixel point in the previous frame image to the current image according to the relative rotation angle in the target decomposition value, and use the rotated position as the first rotated pixel point corresponding to the first key pixel point; thereby, the first rotated pixel point corresponding to each first key pixel point can be obtained, and, for each first key pixel point, the first rotated pixel point corresponding to the first key pixel point also corresponds to the second key pixel point matching the first key pixel point.

[0114] S303: Obtain a first vector according to a preset reference point and each first rotated pixel point, and obtain a second vector according to a second key pixel point corresponding to the preset reference point and each first rotated pixel point.

[0115] S304 , performing a cross multiplication process on the first vector and the second vector to obtain a scalar value corresponding to each first rotated pixel point.

[0116] In the embodiment of the present disclosure, the preset reference point may be the origin of the camera coordinate system, and the origin of the camera coordinate system may be the optical center (center) of the camera.

[0117] In the disclosed embodiment, for each first rotated pixel point, the movable robot can obtain a first vector pointing from the preset reference point to the first rotated pixel point based on the preset reference point and the first rotated pixel point, and obtain a second vector pointing from the preset reference point to the second key pixel point based on the preset reference point and the second key pixel point corresponding to the first rotated pixel point, and cross-multiply the first vector with the second vector to obtain a scalar value corresponding to the first rotated pixel point.

[0118] S305 . Sum the scalar values ​​corresponding to the first rotated pixel points to obtain a first distance.

[0119] In the disclosed embodiment, after obtaining the scalar value corresponding to each first rotated pixel point, the movable robot can calculate the sum of the scalar values ​​corresponding to all the first rotated pixel points, and use the sum of the obtained scalar values ​​as the first distance; in this way, a first relative displacement with the first direction and the modulus being the first distance is obtained.

[0120] In the disclosed embodiment, the above calculation method can improve the accuracy of the first distance of the first relative displacement, thereby facilitating improving the accuracy of the current motion state of the movable robot determined according to the first relative displacement.

[0121] In some embodiments, the mobile robot can use the following method to select the target decomposition value: select any first key pixel point as the first pixel point; select the second key pixel point that matches any first key pixel point as the second pixel point; for the current group decomposition value, the current group decomposition value is any group of decomposition values ​​in multiple groups of decomposition values, according to the first pixel point and the relative rotation angle included in the current group decomposition value, obtain the second rotated pixel point corresponding to the first pixel point; perform cross multiplication processing on the second rotated pixel point and the relative displacement included in the current group decomposition value to obtain a third vector; perform cross multiplication processing on the second pixel point and the relative displacement included in the current group decomposition value The relative displacement included is cross-multiplied to obtain a fourth vector; the first direction vector is obtained according to the center position of the first camera angle of view of the previous frame image and the first pixel point; the second direction vector is obtained according to the first direction vector and the relative rotation angle included in the current group of decomposition values, and the center vector is determined according to the first direction vector and the second direction vector; when the direction of the third vector is the same as the direction of the fourth vector (same direction condition), and the angle between the second direction vector and the center vector is less than the outer angle of the angle between the first direction vector and the center vector (angle relationship condition), the current group of decomposition values ​​is determined to be the target decomposition value. Through this method, the mobile robot can select a group of decomposition values ​​that meet the above-mentioned same direction condition and angle relationship condition from multiple groups of decomposition values ​​as the target decomposition value.

[0122] In some embodiments, the mobile robot can also use the following method to select the target decomposition value: from the first key pixel point of the previous frame image, select multiple first key pixel points as multiple first pixel points, and use the second key pixel point matching each first pixel point as the second pixel point; for the current group decomposition value, according to the current first pixel point and the relative rotation angle included in the current group decomposition value, obtain the second rotated pixel point corresponding to the current first pixel point; the current group decomposition value is any one group of decomposition values ​​in the multiple groups of decomposition values; the current first pixel point is any one of the multiple first pixel points; cross multiply the second rotated pixel point and the relative displacement included in the current group decomposition value to obtain a third vector; cross multiply the current second pixel point matching the current first pixel point and the relative displacement included in the current group decomposition value The fourth vector is obtained by multiplication processing; the current second pixel point is the second pixel point that matches the current first pixel point; the first direction vector is obtained according to the center position of the first camera angle of view that shot the previous frame of the image and the first pixel point; the second direction vector is obtained according to the first direction vector and the relative rotation angle included in the current group decomposition value, and the center vector is determined according to the first direction vector and the second direction vector; when the direction of the third vector is the same as the direction of the fourth vector, and the angle between the second direction vector and the center vector is less than the outer angle of the angle between the first direction vector and the center vector, the current first pixel point and the current second pixel point are used as a pair of matching points corresponding to the current group decomposition value, thereby obtaining a pair of matching points corresponding to each group of decomposition values; the group of decomposition values ​​with the largest number of corresponding matching point pairs among the multiple groups of decomposition values ​​is determined as the target decomposition value. Through this method, the mobile robot can select a group of decomposition values ​​that meet the above-mentioned same direction conditions and angle relationship conditions from multiple groups of decomposition values ​​as the target decomposition value.

[0123] Here, for each group of decomposition values ​​and each first pixel point of a plurality of first pixels randomly selected from all first key pixels of the previous frame image, on the one hand, the mobile robot can rotate the first pixel point to the current image according to the relative rotation angle in the group of decompositions, and use the rotated position as the second rotated pixel point corresponding to the first pixel point. After that, the second rotated pixel point can be cross-producted with the relative displacement in the group of decomposition values ​​to obtain a third vector; at the same time, the second pixel point matching the first pixel point can be cross-producted with the relative displacement in the group of decomposition values ​​to obtain a fourth vector; on the other hand, for the group of decomposition values, the mobile robot can use the relative rotation angle R in the group of decomposition values ​​to transform the direction vector (hereinafter referred to as the first direction vector) from the center position of the first camera viewing angle used to shoot the previous frame image (the coordinates can be (0,0)) to the first pixel point, and obtain the direction vector (hereinafter referred to as the first direction vector) from the center position of the second camera viewing angle used to shoot the current image to the second pixel point matching the first pixel point. The second direction vector, the vector pointing from the center position of the first camera's viewing angle to the center position of the second camera's viewing angle (hereinafter referred to as the center vector), and the first direction vector constitute a triangle, wherein the triangle has a first interior angle (the angle between the second direction vector and the center vector) and a second interior angle (the angle between the first direction vector and the center vector); thereafter, the mobile robot determines whether the directions of the third vector and the fourth vector are the same, and determines whether the first interior angle is smaller than the exterior angle of the second interior angle. When the directions of the third vector and the fourth vector are the same, and the first interior angle is smaller than the exterior angle of the second interior angle, the first pixel point and the second pixel point matching the first pixel point are determined to be a pair of matching points of the group of decomposition values. In this way, all matching point pairs corresponding to each group of decomposition values ​​in the multiple groups of decomposition values ​​can be obtained. Thereafter, the mobile robot can sort the multiple groups of decomposition values ​​according to the number of corresponding matching point pairs, and select a group of decomposition values ​​with the largest number of corresponding matching point pairs as the target decomposition value.

[0124] In the disclosed embodiment, the obtained multiple groups of decomposition values ​​include a group of decomposition values ​​that can make the depth values ​​of the first key pixel points or the second key pixel points with the largest number in the first camera perspective and the second camera perspective both positive, and also include decomposition values ​​that can make the depth values ​​of the first key pixel points or the second key pixel points with the largest number in the first camera perspective and the second camera perspective negative. Through the above-mentioned same direction condition, multiple groups of decomposition values ​​including the optimal relative rotation angle can be first selected. Through the above-mentioned angle relationship condition, a group of decomposition values ​​including the optimal relative displacement can be selected from the multiple groups of decomposition values ​​including the optimal relative rotation angle. Among them, the group of decomposition values ​​that simultaneously include the optimal relative rotation angle and the optimal relative displacement is a group of decomposition values ​​in which the depth values ​​of the first key pixel points or the second key pixel points with the largest number in the first camera perspective and the second camera perspective are both positive. Compared with randomly selecting the target decomposition value from the multiple groups of decomposition values, the accuracy of the obtained target decomposition value can be improved, thereby facilitating obtaining an accurate first relative displacement according to the target decomposition value.

[0125] In some embodiments, the second current motion feature includes: a second relative displacement; the second relative displacement includes: a second direction and a second distance; Figure 5 As shown, the second current motion feature of the device is determined based on the speed in the above S104, which can be implemented by the following S1041 to S1045. Figure 5 Take this as an example to illustrate.

[0126] S1041. Determine a first position of each driving component of the device to obtain multiple first positions.

[0127] S1042: Determine the second position of each driving component according to the speed of each driving component of the device, the first time corresponding to the previous frame image, and the second time corresponding to the current image, to obtain multiple second positions.

[0128] In the embodiment of the present disclosure, when the driving component is a wheel, each wheel has a first position. The movable robot can calculate the moving distance of the wheel based on the speed of the wheel in the time period from the first time to the second time and this time period, and use the position in front of the movement direction of the wheel and the vertical distance between the first position of the wheel as the moving distance as the second position of the wheel.

[0129] In some embodiments, the mobile robot may set an initial position and use the initial position as the first position of each wheel corresponding to the previous frame image. Exemplarily, the coordinates of the initial position may be (0, 0).

[0130] In some embodiments, the mobile robot can set an initial position (for example, (0,0)) when acquiring the first image, and use the initial position as the first position of each wheel corresponding to the first image. After that, according to the speed of each wheel in the time period between the first time corresponding to the first image and the second time corresponding to the second image and the time period, calculate the movement distance S1 of the wheel, and then determine the second position of the wheel (for example, (0,S1)) according to the movement distance S1; and use the first position and the second position to determine the second relative displacement of the mobile robot in the time period from the first time to the second time; then, when the mobile robot acquires the third image, In the case of a previous image (i.e., the second image is the new previous frame image, and the third image is the new current image), the second position obtained above is used as the new first position of the wheel, and the moving distance S2 of the wheel is calculated according to the speed of the wheel in the time period from the second time corresponding to the second image to the third time corresponding to the third image, and then the new second position of the wheel is determined according to the moving distance S2 (for example, (0, S1+S2)); and the second relative displacement of the movable robot in the time period from the second time to the third time is determined by using the new first position and the new second position; by cyclically updating in this way, the first position of each wheel corresponding to each previous frame image can be obtained.

[0131] In some embodiments, when the speed of each wheel is a linear speed, the movable robot can directly use the linear speed of each wheel to calculate the moving distance; in some embodiments, when the speed of each wheel is an angular speed, the movable robot can determine the linear speed of each wheel based on the angular speed and the radius of each wheel, and then calculate the moving distance based on the obtained linear speed of each wheel.

[0132] In some embodiments, the mobile robot may construct scene map information of the area, and use its own position in the scene map information at the first time as the first position.

[0133] S1043. Determine a center position of a first line segment formed by a plurality of first positions, or a center position of a first area formed by a plurality of first positions, to obtain a first center position.

[0134] S1044. Determine the center position of a second line segment formed by multiple second positions, or the center position of a second area formed by multiple second positions, to obtain a second center position.

[0135] S1045. Use the distance between the first center position and the second center position as the second distance, and use the orientation information of the second center position relative to the first center position as the second direction; the second current motion feature includes: a second relative displacement; the second relative displacement includes: a second direction and a second distance.

[0136] In the disclosed embodiment, the movable robot can determine the center position of the line segment or area formed by the first positions of all its wheels (hereinafter referred to as the first center position) when obtaining the first position of each of its wheels; and, the movable robot can determine the center position of the line segment or area formed by all its wheels (hereinafter referred to as the second center position) when obtaining the second position of each of its wheels; thereafter, the distance between the first center position and the second center position can be used as the second distance, and the orientation information of the second center position relative to the first center position can be used as the second direction.

[0137] In some embodiments, the center position of the first area or the second area may be the location of the center point, center of gravity or incenter of the first area or the second area. For example, when the first area or the second area is an irregular triangular area, the center position of the first area or the second area may be the location of the center of gravity or incenter of the triangular area; when the first area or the second area is an irregular quadrilateral area, the center position of the first area or the second area may be the location of the center of gravity of the quadrilateral area; for another example, when the first area or the second area is an equilateral triangular area or a regular polygonal area, the center position of the first area or the second area may be the location of the center point of the equilateral triangular area or the regular polygonal area.

[0138] For example, Fig. 6A As shown, the movable robot has two wheels, and the first position of the first wheel is w1, and the second position is w2, the first position of the second wheel is w3, and the second position is w4, the center position of the first line segment formed by w1 and w3 is w11, and the center position of the second line segment formed by w2 and w4 is w22. According to w11 and w22, the second relative displacement t1 of the movable robot can be obtained.

[0139] For example, Figure 6BAs shown, the movable robot has 4 wheels, and the first position of the first wheel is w1, and the second position is w1', the first position of the second wheel is w2, and the second position is w2', the first position of the third wheel is w3, and the second position is w3', the first position of the fourth wheel is w4, and the second position is w4'; wherein, the center position of the first area formed by w1, w2, w3 and w4 is w11, and the center position of the second area formed by w1', w2', w3' and w4' is w22. According to w11 and w22, the second relative displacement t1 of the movable robot can be obtained.

[0140] In the disclosed embodiment, the above method can be used to obtain the relative displacement of the device within a period of time without relying on images of the scene around the device, thereby improving the flexibility in determining the relative displacement of the device.

[0141] In some embodiments, when the first current motion feature and the second current motion feature are both moving distances, and the first current motion feature is the first distance of the first relative displacement, and the second current motion feature is the second distance of the second relative displacement, the movable robot may compare the first distance with the first preset threshold, and compare the second distance with the second preset threshold; when the first distance is greater than the first preset threshold and the second distance is greater than the second preset threshold, or when the first distance is less than or equal to the first preset threshold and the second distance is less than or equal to the second preset threshold, it is determined that the device is currently in a normal motion state; and when the first distance is greater than the first preset threshold and the second distance is less than or equal to the second preset threshold, or when the first distance is less than or equal to the first preset threshold and the second distance is greater than the second preset threshold, it is determined that the device is currently in an abnormal motion state.

[0142] In some embodiments, the first preset threshold value may be 0.01 or other values, and the embodiments of the present disclosure do not limit this. Here, the first preset threshold value is set to 0.01, and based on the size relationship between the first distance and 0.01, it can be clearly known whether the movable robot has moved during the time period of collecting the previous frame image and the current image, that is, when the first distance is less than or equal to 0.01, it is visually determined that the movable robot has not moved during the time period of collecting the previous frame image and the current image, that is, it is in a stationary or pure rotation state; and when the first distance is greater than 0.01, it is visually determined that the movable robot has moved during the time period of collecting the previous frame image and the current image.

[0143] In some embodiments, the second preset threshold value may be 1 cm or other values, which are not limited in the embodiments of the present disclosure. Here, since the value of 1 cm is relatively small, when the motion state is judged by using this preset threshold value, even if the judgment is wrong, the impact on the judgment of the motion state in the subsequent time period is relatively small, that is, it will not affect the motion state judgment in the subsequent time period, thereby improving the error tolerance rate when detecting the motion state.

[0144] In the above method, the current motion state of the device can be quickly determined through two moving distances, thereby improving the efficiency of determining the motion state of the device.

[0145] In some embodiments, when the first current motion feature and the second current motion feature are both moving directions, and the first current motion feature is the first direction of the first relative displacement, and the second current motion feature is the second direction of the second relative displacement, the movable robot may compare the first direction with the second direction; when the first direction is the same as the second direction, it is determined that the device is currently in a normal motion state; and when the first direction is different from the second direction, it is determined that the device is currently in an abnormal motion state.

[0146] In the above method, the current motion state of the device can be quickly determined through two moving directions, thereby improving the efficiency of determining the motion state of the device.

[0147] In some embodiments, the first current motion feature includes: a first relative displacement; the second current motion feature includes: a second relative displacement; and the current motion state includes: a first normal motion state. Figure 7 As shown, the above S105 can be implemented by S1051. Figure 7 Take this as an example to illustrate.

[0148] S1051: When a first distance of the first relative displacement is less than or equal to a first preset threshold and a second distance of the second relative displacement is less than or equal to a second preset threshold, determine that the device is currently in a first normal motion state.

[0149] In the disclosed embodiment, when the first distance of the first relative displacement is less than or equal to the first preset threshold, and the second distance of the second relative displacement is less than or equal to the second preset threshold, it can be considered that the first current motion feature is consistent with the second current motion feature, that is, the current motion feature obtained based on vision and the current motion feature obtained based on the speed of the driving component are consistent, and thus it can be determined that the movable robot is currently in the first normal motion state.

[0150] Here, when the first distance of the first relative displacement is less than or equal to the first preset threshold, and the second distance of the second relative displacement is less than or equal to the second preset threshold, it can be determined that the movable robot is in a normal suspended movement state within the corresponding time period.

[0151] In the disclosed embodiment, the above method can be used to determine the normal motion state of the movable robot, thereby improving the intelligence of motion state detection.

[0152] In some embodiments, the first current motion feature includes: a first relative displacement; the second current motion feature includes: a second relative displacement; and the current motion state includes: a second normal motion state. Figure 8 As shown, the above S105 can also be implemented by S1052, Figure 8 Take this as an example to illustrate.

[0153] S1052. Determine that the device is currently in the second normal motion state when the first distance of the first relative displacement is greater than the first preset threshold, the second distance of the second relative displacement is greater than the second preset threshold, and the first direction of the first relative displacement is the same as the second direction of the second relative displacement; the first current motion feature includes: the first relative displacement; the second current motion feature includes: the second relative displacement; the current motion state includes: the second normal motion state.

[0154] In the embodiment of the present disclosure, when the first distance of the first relative displacement is greater than the first preset threshold, the second distance of the second relative displacement is greater than the second preset threshold, and the first direction of the first relative displacement is the same as the second direction of the second relative displacement, it can be considered that the first current motion feature is consistent with the second current motion feature, that is, the current motion feature obtained based on vision and the current motion feature obtained based on the speed of the driving component are consistent. Therefore, it can be determined that the movable robot is in the second normal motion state during the time period of capturing the previous frame image and the current image.

[0155] Here, when the first distance is greater than the first preset threshold, the second distance is greater than the second preset threshold, and the first direction of the first relative displacement is the same as the second direction of the second relative displacement, the movable robot can be considered to be in a normal moving state.

[0156] In the disclosed embodiment, the above method can be used to determine the normal motion state of the movable robot, thereby improving the intelligence of motion state detection.

[0157] In some embodiments, the first current motion feature includes: a first distance of a first relative displacement; the second current motion feature includes: a second distance of a second relative displacement; and the current motion state includes: a first abnormal motion state. Fig. 9 As shown, the above S105 can be implemented by S1053. Fig. 9 Take this as an example to illustrate.

[0158] S1053. When the first distance is less than or equal to the first preset threshold and the second distance is greater than the second preset threshold, determine that the device is currently in a first abnormal motion state; the first current motion feature includes: a first distance of a first relative displacement; the second current motion feature includes: a second distance of a second relative displacement; the current motion state includes: a first abnormal motion state.

[0159] In the embodiment of the present disclosure, when the first distance of the first relative displacement is less than or equal to the first preset threshold, and the second distance of the second relative displacement is greater than the second preset threshold, it can be considered that the first current motion feature is inconsistent with the second current motion feature; that is, the current motion feature obtained based on vision and the current motion feature obtained based on the speed of the driving component are inconsistent. Therefore, it can be determined that the movable robot is in the first abnormal motion state during the time period of collecting the previous frame image and the current image.

[0160] Here, when the first distance of the first relative displacement is less than or equal to the first preset threshold and the second distance of the second relative displacement is greater than the second preset threshold, it can be considered that the movable robot is currently in an abnormal state such as a collision or wheel idling.

[0161] In the disclosed embodiment, the above method can be used to determine what abnormal motion state the movable robot is currently in, thereby improving the intelligence of motion state detection.

[0162] In some embodiments, the first current motion feature includes: a first relative displacement; the second current motion feature includes: a second relative displacement; and the current motion state includes: a second abnormal motion state. Fig.10 As shown, the above S105 can be implemented by S1054. Fig.10 Take this as an example to illustrate.

[0163] S1054. Determine that the device is currently in a second abnormal motion state when the first distance of the first relative displacement is greater than a first preset threshold, the second distance of the first relative displacement is greater than a second preset threshold, and the first direction of the first relative displacement is different from the second direction of the first relative displacement; the first current motion feature includes: a first relative displacement; the second current motion feature includes: a second relative displacement; the current motion state includes: a second abnormal motion state.

[0164] In the embodiment of the present disclosure, when the first distance of the first relative displacement is greater than the first preset threshold, the second distance of the second relative displacement is greater than the second preset threshold, and the first direction of the first relative displacement is different from the second direction of the second relative displacement, it can be considered that the first current motion feature is inconsistent with the second current motion feature; that is, the current motion feature obtained based on vision and the current motion feature obtained based on the speed of the driving component are inconsistent. Therefore, it can be determined that the movable robot is in the second abnormal motion state during the time period of collecting the previous frame image and the current image.

[0165] Here, when the first distance is greater than the first preset threshold, the second distance is greater than the second preset threshold, and the first direction is different from the second direction, it can be considered that the movable robot is currently in an abnormal state such as slipping along an obstacle.

[0166] In the disclosed embodiment, the above method can be used to determine what abnormal motion state the movable robot is currently in, thereby improving the intelligence of motion state detection.

[0167] Below Fig.11 and Fig.12 Take the example to illustrate the application of the motion state determination method provided by the embodiment of the present disclosure in an actual scenario.

[0168] S1. Perform 2D key point tracking on two adjacent frames of images.

[0169] Here, according to the key points of the previous frame image, the optical flow method is used to obtain the corresponding points in the current image, so as to obtain the 2d-2d matching points of the previous frame image and the current image; the RANSAC method is used to filter the obtained 2d-2d matching points of the two adjacent frames to obtain 2d-2d valid matching points.

[0170] S2. Relative motion estimation.

[0171] Here, if Fig.12 As shown, in step S22, the corresponding essential matrix is ​​calculated according to the 2d-2d effective matching points of the two adjacent frames of images obtained in step S21. In step S23, the candidate postures are obtained by decomposing the essential matrix. In step S24, the target posture is selected from the candidate postures using intersection constraints and isotropic constraints. Then, the first relative displacement is determined according to the target posture; and the second relative displacement is determined according to the speed of each wheel detected by the wheel speed meter.

[0172] Here, the candidate pose is the above-mentioned multiple groups of decomposition values, and the target pose is the above-mentioned target decomposition value. The same direction constraint condition corresponds to the direction of the above-mentioned third vector being the same as the direction of the fourth vector, and the intersection constraint condition corresponds to the angle between the above-mentioned second direction vector and the center vector being smaller than the outer angle between the first direction vector and the center vector.

[0173] S3. Consistency judgment.

[0174] Here, the first relative displacement is compared with the second relative displacement. When the first distance of the first relative displacement is less than or equal to 0.01 and the second distance of the second relative displacement is less than or equal to 1 cm, it is determined that the movable robot is in a normal suspended movement state during the time period corresponding to the two adjacent frames of images; when the first distance is greater than 0.01, the second distance is greater than 1 cm, and the first direction of the first relative displacement is the same as the second direction of the second relative displacement, it is determined that the movable robot is in a normal moving state during the time period corresponding to the two adjacent frames of images; when the first distance of the first relative displacement is less than or equal to 0.01 and the second distance of the second relative displacement is greater than 1 cm, it is determined that the movable robot is in an abnormal state of collision or wheel idling during the time period corresponding to the two adjacent frames of images; when the first distance is greater than 0.01, the second distance is greater than 1 cm, and the first direction is different from the second direction, it is determined that the movable robot is in an abnormal state of slipping along an obstacle during the time period corresponding to the two adjacent frames of images.

[0175] S4. When an abnormal motion state is detected, output an abnormal detection result.

[0176] Here, when an abnormal motion state is detected, the mobile robot may send a prompt message to the corresponding upper-layer application to prompt what kind of abnormality is currently occurring.

[0177] In some embodiments, through the above-mentioned motion state determination method, the movable robot can also mark the location where the abnormality occurs when an abnormality is detected, so that when subsequently constructing map information for the area, the location where the abnormality occurs can be masked to improve the accuracy of the constructed map information of the area; and when positioning itself in the area, the location where the abnormality occurs can be masked to make the positioning in the area more accurate.

[0178] In some embodiments, through the above-mentioned motion state determination method, the movable robot can also mark the location where the abnormality occurs when an abnormality is detected, so that when performing subsequent path planning in the area, the location where the abnormality occurs can be avoided, thereby improving the accuracy of path planning.

[0179] Based on the foregoing embodiments, the embodiments of the present disclosure provide a motion state determination device, which includes the units included and the modules included in the units, and can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0180] Fig.13 A schematic diagram of the structure of a motion state determination device provided by an embodiment of the present disclosure is shown in FIG. Fig.13 As shown, the motion state determination device 600 includes: an acquisition module 610, which is used to acquire a current image of an area of ​​a preset orientation of the device; a determination module 620, which is used to determine a second key pixel point matching the first key pixel point in the current image based on the first key pixel point of the previous frame image acquired; based on the first key pixel point and the second key pixel point, determine the first current motion feature of the device; the acquisition module 610 is also used to acquire the speed of the driving component of the device; the determination module 620 is also used to determine the second current motion feature of the device based on the speed; based on the first current motion feature and the second current motion feature, determine the current motion state of the device.

[0181] In some embodiments, the determination module 620 is also used to obtain a first target pixel and a second target pixel by filtering the first key pixel and the second key pixel; and determine the first current motion feature of the device based on the first target pixel and the second target pixel.

[0182] In some embodiments, the first current motion feature includes: a first relative displacement; the determination module 620 is also used to obtain an intrinsic matrix based on the first key pixel point and the second key pixel point; the intrinsic matrix is ​​used to characterize the epipolar geometric relationship between the first key pixel point and the second key pixel point; by performing matrix decomposition processing on the intrinsic matrix, multiple groups of different decomposition values ​​are obtained; each group of decomposition values ​​includes a relative displacement and a relative rotation angle; the first relative displacement is determined based on the relative displacement and relative rotation angle included in the multiple groups of different decomposition values.

[0183] In some embodiments, the first relative displacement includes: a first direction and a first distance; the determination module 620 is also used to select a set of target decomposition values ​​from the multiple sets of decomposition values ​​based on any first key pixel point, and use the direction of the relative displacement included in the target decomposition value as the first direction; each first key pixel point is rotated according to the relative rotation angle included in the target decomposition value to obtain the first rotated pixel point corresponding to each first key pixel point; the first rotated pixel point corresponds to the second key pixel point matching each first key pixel point; a first vector is obtained according to a preset reference point and each first rotated pixel point; and a second vector is obtained according to the preset reference point and the second key pixel point corresponding to each first rotated pixel point; the first vector and the second vector are cross-multiplied to obtain a scalar value corresponding to each first rotated pixel point; the scalar values ​​corresponding to each first rotated pixel point are summed to obtain the first distance.

[0184] In some embodiments, the determination module 620 is further configured to select a set of target decomposition values ​​from the multiple sets of decomposition values, and use the relative displacement included in the target decomposition values ​​as the first relative displacement.

[0185] In some embodiments, the determination module 620 is further used to select multiple first key pixel points from the first key pixel points of the previous frame image as multiple first pixel points, and use the second key pixel point matching each first pixel point as the second pixel point; for the current group decomposition value, according to the current first pixel point and the relative rotation angle included in the current group decomposition value, obtain the second rotated pixel point corresponding to the current first pixel point; the current group decomposition value is any group of decomposition values ​​in the multiple groups of decomposition values; the current first pixel point is any one of the multiple first pixel points; cross-product processing is performed on the second rotated pixel point and the relative displacement included in the current group decomposition value to obtain a third vector; cross-product processing is performed on the current second pixel point matching the current first pixel point and the relative displacement included in the current group decomposition value, Obtain a fourth vector; obtain a first direction vector according to the center position of the first camera viewing angle that took the previous frame of image and the first pixel point; obtain a second direction vector according to the first direction vector and the relative rotation angle included in the current group decomposition value, and determine a center vector according to the first direction vector and the second direction vector; when the direction of the third vector is the same as the direction of the fourth vector, and the angle between the second direction vector and the center vector is less than the outer angle of the angle between the first direction vector and the center vector, use the current first pixel point and the current second pixel point as a pair of matching points corresponding to the current group decomposition value, thereby obtaining a pair of matching points corresponding to each group of decomposition values; determine a group of decomposition values ​​with the largest number of corresponding matching point pairs among the multiple groups of decomposition values ​​as the target decomposition value.

[0186] In some embodiments, the second current motion feature includes: a second relative displacement; the second relative displacement includes: a second direction and a second distance; the determination module 620 is also used to determine the first position of each driving component of the device to obtain multiple first positions; determine the second position of each driving component according to the speed of each driving component of the device, the first time corresponding to the previous frame image and the second time corresponding to the current image to obtain multiple second positions; determine the center position of the second line segment formed by the multiple second positions, or the center position of the second area formed by the multiple second positions to obtain the second center position; use the distance between the first center position and the second center position as the second distance, and use the orientation information of the second center position relative to the first center position as the second direction.

[0187] In some embodiments, the acquisition module 610 is also used to acquire a first current image of the area above the device through a first image acquisition device deployed in a first preset area of ​​the device; or to acquire a second current image of the area in front of the device along the movement direction of the device through a second image acquisition device deployed in a second preset area of ​​the device.

[0188] In some embodiments, the first current motion feature includes: a first relative displacement; the second current motion feature includes: a second relative displacement; the current motion state includes: a first normal motion state; the determination module 620 is also used to determine that the device is currently in the first normal motion state when the first distance of the first relative displacement is less than or equal to a first preset threshold and the second distance of the second relative displacement is less than or equal to a second preset threshold.

[0189] In some embodiments, the first current motion feature includes: a first relative displacement; the second current motion feature includes: a second relative displacement; the current motion state includes: a second normal motion state; the determination module 620 is also used to determine that the device is currently in the second normal motion state when the first distance of the first relative displacement is greater than a first preset threshold, the second distance of the second relative displacement is greater than a second preset threshold, and the first direction of the first relative displacement is the same as the second direction of the second relative displacement.

[0190] In some embodiments, the first current motion feature includes: a first distance of a first relative displacement; the second current motion feature includes: a second distance of a second relative displacement; the current motion state includes: a first abnormal motion state; the determination module 620 is also used to determine that the device is currently in the first abnormal motion state when the first distance is less than or equal to a first preset threshold and the second distance is greater than a second preset threshold.

[0191] In some embodiments, the first current motion feature includes: a first relative displacement; the second current motion feature includes: a second relative displacement; the current motion state includes: a second abnormal motion state; the determination module 620 is also used to determine that the device is currently in the second abnormal motion state when the first distance of the first relative displacement is greater than a first preset threshold, the second distance of the second relative displacement is greater than a second preset threshold, and the first direction of the first relative displacement is different from the second direction of the second relative displacement.

[0192] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiments of the present disclosure can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present disclosure, please refer to the description of the method embodiment of the present disclosure for understanding.

[0193] It should be noted that in the embodiments of the present disclosure, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present disclosure is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiments of the present disclosure are not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0194] An embodiment of the present disclosure provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0195] The embodiment of the present disclosure provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0196] An embodiment of the present disclosure provides a computer program, including a computer-readable code. When the computer-readable code is executed in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0197] The present disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0198] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product disclosed in the present invention, please refer to the description of the method embodiment disclosed in the present invention for understanding.

[0199] It should be noted that Fig.14 is a schematic diagram of a hardware entity of a computer device in an embodiment of the present disclosure, such as Fig.14 As shown, the hardware entity of the computer device 700 includes: a processor 701, a communication interface 702 and a memory 703, wherein:

[0200] Processor 701 generally controls the overall operation of computer device 700 .

[0201] The communication interface 702 enables the computer device to communicate with other terminals or servers through a network.

[0202] The memory 703 is configured to store instructions and applications executable by the processor 701, and can also cache data to be processed or processed by the processor 701 and each module in the computer device 700 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data transmission can be performed between the processor 701, the communication interface 702, and the memory 703 through the bus 704.

[0203] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present disclosure. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present disclosure, the size of the serial numbers of the above-mentioned steps / processes does not mean the order of execution. The execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The serial numbers of the embodiments of the present disclosure are for description only and do not represent the advantages and disadvantages of the embodiments.

[0204] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0205] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0206] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0207] In addition, all functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may be separately configured as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0208] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0209] Alternatively, if the above-mentioned integrated unit of the present disclosure is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0210] The above description is only an implementation mode of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure.

Claims

1. A method for determining a motion state, characterized in that: include: Obtaining a current image of an area at a preset orientation of the device; Determine, from the current image, a second key pixel point that matches the first key pixel point, according to the first key pixel point of the acquired previous frame image; Determining a first current motion feature of the device based on the first key pixel point and the second key pixel point; The first current motion feature includes: a first relative displacement; the first relative displacement includes: a first direction and a first distance; acquiring a speed of a driving component of the device, and determining a second current motion characteristic of the device based on the speed; determining a current motion state of the device based on the first current motion feature and the second current motion feature; The determining, based on the first key pixel point and the second key pixel point, a first current motion feature of the device includes: Based on the first key pixel point and the second key pixel point, an intrinsic matrix is ​​obtained; the intrinsic matrix is ​​used to characterize the epipolar geometric relationship between the first key pixel point and the second key pixel point; By performing matrix decomposition processing on the essential matrix, a plurality of different decomposition values ​​are obtained; each group of decomposition values ​​includes a relative displacement and a relative rotation angle; Determine a set of target decomposition values ​​from the multiple sets of different decomposition values ​​based on intersection constraints and same-direction constraints, and use the direction of the relative displacement included in the target decomposition values ​​as the first direction; and The first distance is obtained based on the target decomposition value, the first key pixel point and the second key pixel point.

2. The method according to claim 1, characterized in that The determining, based on the first key pixel point and the second key pixel point, a first current motion feature of the device includes: Obtaining a first target pixel point and a second target pixel point by filtering the first key pixel point and the second key pixel point; A first current motion feature of the device is determined based on the first target pixel and the second target pixel.

3. The method according to claim 1, characterized in that: The obtaining the first distance based on the target decomposition value, the first key pixel point and the second key pixel point includes: Rotate each first key pixel point by the relative rotation angle included in the target decomposition value to obtain a first rotated pixel point corresponding to each first key pixel point; the first rotated pixel point corresponds to a second key pixel point matched with each first key pixel point; Obtaining a first vector according to a preset reference point and each first rotated pixel point; and obtaining a second vector according to a second key pixel point corresponding to the preset reference point and each first rotated pixel point; Performing a cross product process on the first vector and the second vector to obtain a scalar value corresponding to each first rotated pixel point; The scalar values ​​corresponding to each first rotated pixel point are summed to obtain the first distance.

4. The method according to claim 1, characterized in that: The method further comprises: A set of target decomposition values ​​is selected from the multiple sets of decomposition values, and the relative displacement included in the target decomposition values ​​is used as the first relative displacement.

5. The method according to claim 3, characterized in that: The step of determining a set of target decomposition values ​​from the plurality of different decomposition values ​​based on the intersection constraint condition and the same direction constraint condition comprises: From the first key pixel points of the previous frame image, select a plurality of first key pixel points as a plurality of first pixel points, and use a second key pixel point matched with each first pixel point as a second pixel point; For a current group decomposition value, according to a current first pixel point and a relative rotation angle included in the current group decomposition value, a second rotated pixel point corresponding to the current first pixel point is obtained; the current group decomposition value is any one of the multiple groups of decomposition values; the current first pixel point is any one of the multiple first pixel points; Performing a cross multiplication process on the second rotated pixel point and the relative displacement included in the current group decomposition value to obtain a third vector; Performing a cross multiplication process on a current second pixel point matched with the current first pixel point and a relative displacement included in the current group decomposition value to obtain a fourth vector; Obtain a first direction vector according to the center position of the viewing angle of a first camera that captured the previous frame of image and the first pixel point; Obtain a second direction vector according to the first direction vector and the relative rotation angle included in the current group decomposition value, and determine a center vector according to the first direction vector and the second direction vector; In the case that the direction of the third vector is the same as the direction of the fourth vector, and the angle between the second direction vector and the center vector is smaller than the outer angle of the angle between the first direction vector and the center vector, the current first pixel point and the current second pixel point are used as a pair of matching points corresponding to the current group of decomposition values, so as to obtain a pair of matching points corresponding to each group of decomposition values; Among the multiple groups of decomposition values, the group of decomposition values ​​corresponding to the largest number of matching point pairs is determined as the target decomposition value.

6. The method according to any one of claims 1 to 5, characterized in that: The second current motion feature includes: a second relative displacement; the second relative displacement includes: a second direction and a second distance; the determining the second current motion feature of the device based on the speed includes: determining a first position of each driving component of the device to obtain a plurality of first positions; Determine the second position of each driving component according to the speed of each driving component of the device, the first time corresponding to the previous frame image and the second time corresponding to the current image, and obtain a plurality of second positions; Determine a center position of a first line segment formed by the multiple first positions, or a center position of a first area formed by the multiple first positions, to obtain a first center position; Determine the center position of a second line segment formed by the plurality of second positions, or the center position of a second area formed by the plurality of second positions, to obtain a second center position; The distance between the first center position and the second center position is taken as the second distance, and the orientation information of the second center position relative to the first center position is taken as the second direction.

7. The method according to any one of claims 1 to 5, characterized in that: The step of acquiring a current image of an area at a preset position of the device includes: Acquire a first current image of an upper area of ​​the device by a first image acquisition device disposed in a first preset area of ​​the device; or, A second current image of an area in front of the device along a moving direction of the device is acquired by a second image acquisition device deployed in a second preset area of ​​the device.

8. The method according to any one of claims 1 to 5, characterized in that: The second current motion feature includes: a second relative displacement; the current motion state includes: a first normal motion state; and determining the current motion state of the device based on the first current motion feature and the second current motion feature includes: When the first distance of the first relative displacement is less than or equal to a first preset threshold, and the second distance of the second relative displacement is less than or equal to a second preset threshold, it is determined that the device is currently in the first normal motion state.

9. The method according to any one of claims 1 to 5, characterized in that: The second current motion feature includes: a second relative displacement; the current motion state includes: a second normal motion state; and determining the current motion state of the device based on the first current motion feature and the second current motion feature includes: When the first distance of the first relative displacement is greater than a first preset threshold, the second distance of the second relative displacement is greater than a second preset threshold, and the first direction of the first relative displacement is the same as the second direction of the second relative displacement, it is determined that the device is currently in the second normal motion state.

10. The method according to any one of claims 1 to 5, characterized in that: The first current motion feature includes: a first distance of a first relative displacement; the second current motion feature includes: a second distance of a second relative displacement; the current motion state includes: a first abnormal motion state; and determining the current motion state of the device based on the first current motion feature and the second current motion feature includes: When the first distance is less than or equal to a first preset threshold and the second distance is greater than a second preset threshold, it is determined that the device is currently in the first abnormal motion state.

11. The method according to any one of claims 1 to 5, characterized in that: The second current motion feature includes: a second relative displacement; the current motion state includes: a second abnormal motion state; and determining the current motion state of the device based on the first current motion feature and the second current motion feature includes: When the first distance of the first relative displacement is greater than a first preset threshold, the second distance of the second relative displacement is greater than a second preset threshold, and the first direction of the first relative displacement is different from the second direction of the second relative displacement, it is determined that the device is currently in the second abnormal motion state.

12. A motion state determination device, characterized in that: include: An acquisition module, used to acquire a current image of an area at a preset orientation of the device; A determination module, configured to determine, from the current image, a second key pixel point that matches the first key pixel point, based on the first key pixel point acquired in the previous frame image; Determining a first current motion feature of the device based on the first key pixel point and the second key pixel point; The first current motion feature includes: a first relative displacement; the first relative displacement includes: a first direction and a first distance; The acquisition module is further used to acquire the speed of the driving component of the device; The determination module is further configured to determine a second current motion feature of the device based on the speed; and determine a current motion state of the device based on the first current motion feature and the second current motion feature; The determination module is also used to obtain an essential matrix based on the first key pixel point and the second key pixel point; the essential matrix is ​​used to characterize the epipolar geometric relationship between the first key pixel point and the second key pixel point; by performing matrix decomposition processing on the essential matrix, multiple groups of different decomposition values ​​are obtained; each group of decomposition values ​​includes relative displacement and relative rotation angle; based on intersection constraints and same-direction constraints, a group of target decomposition values ​​are determined from the multiple groups of different decomposition values, and the direction of the relative displacement included in the target decomposition value is used as the first direction; and based on the target decomposition value, the first key pixel point and the second key pixel point, the first distance is obtained.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, wherein: When the processor executes the program, the steps in the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images