Dynamic region detection method, device and equipment

Abstract

The embodiment of the invention provides a dynamic region detection method, a dynamic region detection device and dynamic region detection equipment. A plurality of images shot by a camera at different moments and pose change information between the moments are fused; by calculating the difference (third optical flow value) between the optical flow (first optical flow value) caused by the movement of the camera and the optical flow (second optical flow value) generated by the actual movement of the image content, the interference generated by the movement of the camera instead of the movement of the object in the scene is effectively eliminated, and the accuracy of the dynamic region detection is further improved. The third optical flow value is used for carrying out clustering analysis on the pixel points, target dynamic areas are effectively distinguished and extracted, the areas directly reflect the motion state in the scene, the method overcomes the misjudgment problem possibly caused by only depending on the difference between the images in a traditional method, and the accuracy of the target dynamic area is improved. The detection efficiency and robustness of the dynamic area in a complex dynamic scene are remarkably improved, and more reliable data support is provided for subsequent applications such as visual analysis, target tracking and behavior recognition.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a dynamic area detection method, device and equipment. Background Art

[0002] In core computer vision tasks such as visual localization and mapping, accurately estimating the motion of the camera is the basis for accurate environment modeling and navigation. However, these tasks are often severely disturbed by dynamic objects in the scene. Dynamic objects not only lead to incorrect associations in the feature matching process, thereby reducing the accuracy of camera pose estimation, but also produce unexpected ghosting or blurring in applications such as dense mapping, which seriously damages the clarity of the map and the effectiveness of subsequent recognition, planning and other tasks. Therefore, effectively identifying and processing dynamic areas in the field of view has become a key challenge to improve the performance of visual localization and mapping systems. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a dynamic area detection method, device and apparatus to improve the accuracy, efficiency and robustness of dynamic area detection. The specific technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides a dynamic area detection method, the method comprising:

[0005] Acquire a first image captured by a camera at a first moment, a second image captured by a camera at a second moment, and a position change of the camera from the first moment to the second moment, wherein the position change is measured by a motion sensor;

[0006] Obtaining, according to the position change, a first optical flow value for representing a relative motion caused by the position change, and obtaining, according to the first image and the second image, a second optical flow value for representing a relative motion between each pixel in the first image and each pixel in the second image;

[0007] Calculating a difference between the first optical flow value and the second optical flow value as a third optical flow value;

[0008] Clustering the pixel points in the second image according to the third optical flow value to obtain target clustering areas;

[0009] A target clustering region including pixels whose third optical flow values ​​satisfy a preset dynamic condition is determined as a target dynamic region.

[0010] In a possible implementation, clustering the pixels in the second image according to the third optical flow value to obtain target clustering regions includes:

[0011] Detecting in the second image a target detection area where each target object is located, wherein the target object is an object capable of moving;

[0012] The pixel points in each of the target detection areas are clustered according to the third optical flow value to obtain target cluster areas.

[0013] In a possible implementation, clustering the pixels in each of the target detection areas according to the third optical flow value to obtain each target clustering area includes:

[0014] Respectively determining the difference between the third optical flow values ​​of each pixel point in each target detection area;

[0015] Determine the target detection area whose difference is greater than a preset difference threshold as a clustering candidate area;

[0016] According to the third optical flow value, the pixel points in each of the cluster candidate areas are clustered to obtain each target cluster area.

[0017] In a possible implementation, clustering the pixels in each of the candidate cluster regions according to the third optical flow value to obtain each target cluster region includes:

[0018] Expanding the cluster candidate regions respectively to obtain target expanded regions;

[0019] The pixel points in each of the target expansion regions are clustered according to the third optical flow value to obtain each target cluster region.

[0020] In a possible implementation, clustering the pixels in each of the target detection areas according to the third optical flow value to obtain each target clustering area includes:

[0021] The pixel points in each target detection area are clustered according to the module length and / or direction of the third optical flow value to obtain each target clustering area.

[0022] In a possible implementation manner, respectively determining the difference between the third optical flow values ​​of each pixel point in each target detection area includes:

[0023] Calculating the variance of the third optical flow values ​​of each pixel point in each target detection area respectively;

[0024] The step of determining the target detection area whose difference is greater than a preset difference threshold as a cluster candidate area includes:

[0025] The target detection area whose variance is greater than a preset variance threshold is determined as a clustering candidate area.

[0026] In a possible implementation, obtaining, according to the posture change, a first optical flow value for representing the relative motion caused by the posture change includes:

[0027] Calculating, according to the posture change, a transformation matrix for representing a posture change amount of the camera from the first moment to the second moment;

[0028] According to the transformation matrix and the camera intrinsic parameters of the camera, each pixel point in the first image, or a spatial point corresponding to each pixel point, is projected into the second image to obtain each projection point;

[0029] Calculating the difference between each of the projection points and each of the pixel points in the first image as a first optical flow value;

[0030] The obtaining, according to the first image and the second image, a second optical flow value for representing a relative motion between each pixel in the first image and each pixel in the second image includes:

[0031] The first image and the second image are input into an optical flow estimation deep neural network to obtain a second optical flow value for representing the relative motion between each pixel in the first image and each pixel in the second image.

[0032] In a possible implementation, the method further includes:

[0033] The motion sensor includes a wheel speed meter, and the transformation matrix is ​​calculated by the following formula:

[0034]

[0035] Where T is the transformation matrix, w k is the angular velocity of the camera in the kth sub-period, v k is the linear velocity of the camera in the kth sub-period, is the noise term of the angular velocity, is the noise term of the linear velocity, Δt is the duration of each sub-period;

[0036] or,

[0037] The motion sensor includes an inertial measurement unit, and the transformation matrix is ​​calculated by the following formula:

[0038]

[0039] Among them, T ij is the transformation matrix, w k is the angular velocity of the camera in the kth sub-period, fk is the acceleration of the camera in the kth sub-period, is the paranoid term of the angular velocity, is the paranoid term of the acceleration, is the noise term of the angular velocity, is the noise term of the acceleration, and Δt is the duration of each sub-period.

[0040] In a second aspect, an embodiment of the present application provides a dynamic area detection device, the device comprising:

[0041] An image acquisition module, used to acquire a first image captured by a camera at a first moment, a second image captured by a camera at a second moment, and a position change of the camera from the first moment to the second moment, wherein the position change is measured by a motion sensor;

[0042] an optical flow value acquisition module, configured to obtain, according to the posture change, a first optical flow value for representing a relative motion caused by the posture change, and to obtain, according to the first image and the second image, a second optical flow value for representing a relative motion between each pixel in the first image and each pixel in the second image;

[0043] a difference calculation module, configured to calculate a difference between the first optical flow value and the second optical flow value as a third optical flow value;

[0044] A pixel clustering module, used for clustering the pixels in the second image according to the third optical flow value to obtain target clustering areas;

[0045] The region determination module is used to determine a target cluster region including pixels whose third optical flow values ​​meet a preset dynamic condition as a target dynamic region.

[0046] In a possible implementation, the pixel clustering module includes:

[0047] A first clustering submodule is used to detect in the second image a target detection area where each target object is located, wherein the target object is an object with the ability to move;

[0048] A second clustering submodule, used for clustering the pixel points in each target detection area according to the third optical flow value to obtain each target clustering area;

[0049] The second clustering submodule comprises:

[0050] A first clustering unit, used to respectively determine the difference between the third optical flow values ​​of each pixel point in each target detection area;

[0051] A second clustering unit is used to determine the target detection area whose difference is greater than a preset difference threshold as a clustering candidate area;

[0052] A third clustering unit, configured to cluster the pixel points in each of the candidate clustering regions according to the third optical flow value to obtain each target clustering region;

[0053] The third clustering unit comprises:

[0054] A first subunit is used to expand the cluster candidate regions respectively to obtain target expanded regions;

[0055] A second subunit is used to cluster the pixel points in each of the target expansion regions according to the third optical flow value to obtain each target cluster region;

[0056] The second subunit comprises:

[0057] A first subcomponent is used to cluster the pixel points in each target detection area according to the modulus and / or direction of the third optical flow value to obtain each target clustering area;

[0058] The first clustering unit comprises:

[0059] A third subunit, used to respectively calculate the variance between the third optical flow values ​​of each pixel point in each target detection area;

[0060] The second clustering unit comprises:

[0061] The fourth subunit is used to determine the target detection area whose variance is greater than a preset variance threshold as a clustering candidate area.

[0062] The optical flow value acquisition module includes:

[0063] A first acquisition submodule, configured to calculate, according to the posture change, a transformation matrix for representing a posture change amount of the camera from the first moment to the second moment;

[0064] A second acquisition submodule, used for projecting each pixel point in the first image, or a space point corresponding to each pixel point, into the second image according to the transformation matrix and the camera intrinsic parameters of the camera, to obtain each projection point;

[0065] A third acquisition submodule, used to calculate the difference between each of the projection points and each of the pixel points in the first image as a first optical flow value;

[0066] The optical flow value acquisition module includes:

[0067] The fourth acquisition submodule is used to input the first image and the second image into the optical flow estimation deep neural network to obtain a second optical flow value used to represent the relative motion between each pixel in the first image and each pixel in the second image.

[0068] The device also includes:

[0069] The motion sensor includes a wheel speed meter, and the transformation matrix is ​​calculated by the following formula:

[0070]

[0071] Where T is the transformation matrix, w k is the angular velocity of the camera in the kth sub-period, v k is the linear velocity of the camera in the kth sub-period, is the noise term of the angular velocity, is the noise term of the linear velocity, Δt is the duration of each sub-period;

[0072] or,

[0073] The motion sensor includes an inertial measurement unit, and the transformation matrix is ​​calculated by the following formula:

[0074]

[0075] Among them, T ij is the transformation matrix, w k is the angular velocity of the camera in the kth sub-period, f k is the acceleration of the camera in the kth sub-period, is the paranoid term of the angular velocity, is the paranoid term of the acceleration, is the noise term of the angular velocity, is the noise term of the acceleration, and Δt is the duration of each sub-period.

[0076] In a third aspect, an embodiment of the present application provides a mobile robot, comprising:

[0077] A camera, used to capture a first image at a first moment and to capture a second image at a second moment;

[0078] A motion sensor, used for detecting the position information of the camera in real time;

[0079] A processor obtains a first image captured by the camera at the first moment, a second image captured at the second moment, and the posture information of the camera detected in real time by the motion sensor, and calculates the posture change of the camera from the first moment to the second moment according to the posture information; obtains a first optical flow value for representing the relative motion caused by the posture change according to the posture change, and obtains a second optical flow value for representing the relative motion between each pixel in the first image and each pixel in the second image according to the first image and the second image; calculates the difference between the first optical flow value and the second optical flow value as a third optical flow value; clusters the pixels in the second image according to the third optical flow value to obtain each target clustering area; and determines the target clustering area in which the third optical flow values ​​of the pixels satisfy a preset dynamic condition as the target dynamic area.

[0080] In a fourth aspect, an embodiment of the present application provides an electronic device, including:

[0081] Memory, used to store computer programs;

[0082] The processor is used to implement any of the above-mentioned dynamic area detection methods when executing the program stored in the memory.

[0083] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements any of the dynamic area detection methods described above.

[0084] In a sixth aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the dynamic area detection methods described above.

[0085] Beneficial effects of the embodiments of the present application:

[0086] The dynamic area detection method provided in the embodiment of the present application, by fusing multiple images taken by the camera at different times and the information of the posture change between the times, and by calculating the difference (third optical flow value) between the optical flow caused by the camera's own motion (first optical flow value) and the optical flow generated by the actual motion of the image content (second optical flow value), effectively eliminates the interference caused by the camera's movement rather than the movement of objects in the scene, and further improves the accuracy of dynamic area detection. Furthermore, the third optical flow value is used to perform cluster analysis on the pixels, effectively distinguishing and extracting the target dynamic areas, which directly reflect the motion state in the scene. This method overcomes the problem of misjudgment that may be caused by relying solely on the difference between images in the traditional method, and significantly improves the detection efficiency and robustness of dynamic areas in complex dynamic scenes, providing more reliable data support for subsequent visual analysis, target tracking, behavior recognition and other applications.

[0087] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0089] Figure 1 A schematic diagram of a first flow chart of a dynamic area detection method provided in an embodiment of the present application;

[0090] Figure 2 A schematic diagram of the layout of a mobile robot provided in an embodiment of the present application;

[0091] Figure 3 A first schematic diagram of dynamic mask generation provided in an embodiment of the present application;

[0092] Figure 4 A second schematic diagram of dynamic mask generation provided in an embodiment of the present application;

[0093] Figure 5 A first schematic diagram of non-detection dynamic mask generation provided in an embodiment of the present application;

[0094] Figure 6 A second schematic diagram of non-detection dynamic mask generation provided in an embodiment of the present application;

[0095] Figure 7 A third schematic diagram of non-detection dynamic mask generation provided in an embodiment of the present application;

[0096] Figure 8 A second flow chart of the dynamic area detection method provided in the embodiment of the present application;

[0097] Fig. 9 A first schematic diagram of dynamic area detection provided in an embodiment of the present application;

[0098] Fig.10 A second schematic diagram of dynamic area detection provided in an embodiment of the present application;

[0099] Fig.11 A third schematic diagram of dynamic area detection provided in an embodiment of the present application;

[0100] Fig.12 A schematic diagram of the structure of a dynamic area detection device provided in an embodiment of the present application;

[0101] Fig.13 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0102] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0103] In the field of computer vision and robotics, the detection of dynamic regions has always been a hot topic and a difficult point of research. In the prior art, when dealing with complex dynamic scenes, a core assumption is often used: most areas in the scene are static. This assumption is used to estimate the camera's pose transformation (T) and then separate the dynamic region. However, this static assumption faces significant limitations in practical applications, especially when facing complex dynamic scenes containing a large proportion of moving objects. Specifically, the widespread presence of moving objects in dynamic scenes will seriously interfere with the accurate estimation of the camera's pose. This is because the camera's pose transformation is usually calculated by comparing images at different time points and assuming that most of the scene content remains unchanged (i.e., static). When there is a lot of motion in the scene, this static assumption no longer holds, resulting in a large error in the calculated camera pose. Since the accurate identification of dynamic regions depends on the effective distinction between camera motion and object motion, incorrect camera pose estimation will cause dynamic information to be misclassified as background noise, or static background to be misjudged as a dynamic region, thereby reducing the accuracy and reliability of dynamic region detection.

[0104] Furthermore, current technologies also attempt to detect dynamic areas by combining semantic detection with geometric constraint information. Although semantic detection technology can identify specific objects in a scene, relying solely on semantic information is inadequate for determining the motion state of an object. This is because semantic information is not sufficient to fully capture dynamics. For example, a person's walking pattern may vary depending on the specific situation, and an originally static object may become dynamic due to external influences. Therefore, dynamic area detection cannot be performed accurately based on semantic information alone.

[0105] Based on this, the present application embodiment provides a dynamic area detection method, see Figure 1 , Figure 1 The first flow chart of the dynamic area detection method provided in the embodiment of the present application includes:

[0106] S101, acquiring a first image captured by a camera at a first moment, a second image captured by a camera at a second moment, and a change in the position and posture of the camera from the first moment to the second moment.

[0107] The posture change is measured by a motion sensor installed on the camera.

[0108] S102, obtaining a first optical flow value for representing the relative motion caused by the posture change according to the posture change, and obtaining a second optical flow value for representing the relative motion between each pixel in the first image and each pixel in the second image according to the first image and the second image.

[0109] S103: Calculate the difference between the first optical flow value and the second optical flow value as the third optical flow value.

[0110] S104: Cluster the pixel points in the second image according to the third optical flow value to obtain target clustering regions.

[0111] S105: Determine a target clustering region including pixels whose third optical flow values ​​satisfy a preset dynamic condition as a target dynamic region.

[0112] By applying the above embodiment, by fusing multiple images taken by the camera at different times and the information of the posture change between the times, and by calculating the difference (i.e., the third optical flow value) between the optical flow caused by the camera's own motion (i.e., the first optical flow value) and the optical flow generated by the actual motion of the image content (i.e., the second optical flow value), the interference caused by the camera's movement rather than the movement of objects in the scene is effectively eliminated, and the accuracy of dynamic area detection is further improved. Furthermore, the third optical flow value is used to perform cluster analysis on the pixels, effectively distinguishing and extracting the target dynamic areas, which directly reflect the motion state in the scene. This method overcomes the problem of misjudgment that may be caused by relying solely on the difference between images in the traditional method, and significantly improves the detection efficiency and robustness of dynamic areas in complex dynamic scenes, providing more reliable data support for subsequent visual analysis, target tracking, behavior recognition and other applications.

[0113] Before describing the above S101-S105, the following will first describe the mobile robot to which the dynamic area detection method provided by the present application is applied. Figure 2 , Figure 2 The figure shows a schematic diagram of the layout of the mobile robot provided by the present application, including:

[0114] The mobile robot 200 includes a camera 201 , a motion sensor 202 , and a processor 203 .

[0115] The processor 203 maintains communication connection with the camera 201 and the motion sensor 202 respectively, and the camera 201 and the motion sensor 202 maintain communication connection, so that the motion sensor 202 can obtain the change in the position of the camera 201 when shooting different images.

[0116] The camera 201 is used to capture a first image at a first moment and to capture a second image at a second moment, that is, to execute the above step S101.

[0117] The motion sensor 202 is used to detect the position information of the camera in real time, that is, to execute the above step S101;

[0118] The processor 203 obtains a first image captured by the camera 201 at a first moment, a second image captured at a second moment, and the posture information of the camera 201 detected in real time by the motion sensor 202, calculates the posture change of the camera 201 from the first moment to the second moment according to the posture information, obtains a first optical flow value for representing the relative motion caused by the posture change according to the posture change, obtains a second optical flow value for representing the relative motion between each pixel in the first image and each pixel in the second image according to the first image and the second image; calculates the difference between the first optical flow value and the second optical flow value as a third optical flow value; clusters the pixels in the second image according to the third optical flow value to obtain each target clustering area; determines the target clustering area whose third optical flow values ​​of the pixels meet the preset dynamic conditions as the target dynamic area, that is, executes the above steps S102-S105.

[0119] Figure 2 This is only an example of a layout diagram of a mobile robot. In other embodiments, the layout diagram of the mobile robot may also be in other styles.

[0120] The following will describe steps S101 to S105 in detail:

[0121] In S101, at a certain time point, which is denoted as the "first moment" in this article, the camera captures a specific scene and obtains an image, which is a two-dimensional projection of the specific scene from the camera's perspective, and contains information about all objects of the specific scene within the camera's field of view at that moment. Subsequently, at another time point, which is denoted as the "second moment" in this article, the camera captures the specific scene again and obtains an image. Due to the movement of the camera or the movement of objects in the specific scene, this image will be different from the first image. The difference between the two images can be used to infer the movement of the camera or the changes of objects in the scene.

[0122] The pose refers to the position and attitude (i.e., orientation) of the camera in space. The pose change of the camera refers to the movement (translation) and rotation of the camera in three-dimensional space from the first moment to the second moment. In order to ensure that the measurement of the camera pose change is not disturbed by visual dynamic changes (such as illumination changes, occlusion, etc.), motion sensors installed on the camera are used to directly measure these changes. These sensors include but are not limited to wheel odometers (i.e., wheel speed meters), IMUs (Inertial Measurement Units), etc. These devices can measure the camera's kinematic physical quantities such as acceleration and angular velocity with high precision, and calculate the camera's precise pose change through corresponding algorithms (such as integration).

[0123] In S102, the pixel motion caused by the movement (translation and rotation) of the camera can be calculated using the camera pose change data obtained from the motion sensor (such as IMU), that is, the scene flow, that is, the first optical flow value. The first optical flow value represents: if all objects in the scene remain stationary, the optical flow value caused only by the camera movement.

[0124] In a possible embodiment, in order to improve the efficiency of optical flow calculation, avoid complex feature point detection and matching processes, and reduce the amount of calculation, a transformation matrix that can be used to represent the change in the camera's posture from the first moment when the first image is captured to the second moment when the second image is captured can be first calculated based on the posture change measured by the motion sensor. According to the transformation matrix and the camera intrinsic parameters of the camera, each pixel point in the first image, or the spatial point corresponding to each pixel point, is projected into the second image to obtain each projection point, and then the difference between each projection point and each pixel point in the first image is calculated as the first optical flow value.

[0125] By applying the above-mentioned embodiment, the posture change information provided by the motion sensor is directly utilized. By calculating the transformation matrix of the posture change and the pixel projection, the efficiency of the optical flow calculation is improved, the complex feature point detection and matching process is avoided, and the amount of calculation is reduced. Secondly, because it is directly based on the posture change of the camera, it is not affected by external factors such as image noise and lighting changes, which enhances the robustness of the optical flow calculation and the accuracy of the calculated optical flow value, thereby improving the detection efficiency, correctness and robustness of the dynamic area.

[0126] In a possible embodiment, the first optical flow value and the second optical flow value may be directly input into a preset function, and then the transformation matrix may be calculated. However, this method may result in a large amount of computation.

[0127] Based on this, in another possible embodiment, the transformation matrix can be split into position change t (i.e., translation matrix) and attitude change R (i.e., rotation matrix), and then the position change t and attitude change R are calculated separately, thereby reducing the amount of calculation. The transformation matrix T can be calculated by the following formula (1):

[0128]

[0129] Among them, R is calculated according to the rotation angle of the camera, and t is calculated according to the displacement of the camera.

[0130] It is understandable that the motion sensor can usually only measure the angular velocity, linear velocity, etc. of the camera at a certain moment, and cannot directly measure the rotation angle and displacement of the camera during the entire period of time.

[0131] Therefore, in a possible embodiment, it can be assumed that the camera moves at a constant speed from the first moment to the second moment, so an angular velocity can be measured arbitrarily in the time period from the first moment to the second moment, and then multiplied by the duration of the time period to obtain the rotation angle, and the displacement can also be obtained by the same method. However, in actual application scenarios, the camera is not transformed at a constant speed, so the rotation angle and displacement measured by this method are less accurate, which in turn affects the accurate detection of dynamic areas.

[0132] Based on this, in another possible embodiment, it can be considered that the camera is not moving at a uniform speed, and the switch from the first moment to the second moment can be divided into multiple sub-periods. Theoretically, if the sub-period is short enough, it can be considered that the camera is moving at a uniform speed in each sub-period. Therefore, the change in each sub-period can be calculated based on the changes in angular velocity, linear velocity, etc. measured in each sub-period, so as to obtain the overall change of the camera in the entire time period from the first moment to the second moment, and then obtain the posture change R (i.e., rotation matrix) and position change t (i.e., translation matrix).

[0133] Exemplarily, in this example, for the convenience of description, the first moment to the second moment is divided into n time periods. Since the first moment can be regarded as the starting moment of the first sub-period, the first moment is recorded as the first moment hereinafter, and the second moment is the ending moment of the nth sub-period, so the second moment is recorded as the nth moment hereinafter. The position change t and the attitude change R can be obtained by the following method:

[0134] Method 1: For a motion sensor including a wheel speed meter, taking a two-wheel differential model as an example, the method for obtaining the position change t and the attitude change R is as follows:

[0135] The linear velocity v and angular velocity w of the camera in the world coordinate system are obtained through the two-wheel differential model. The linear velocity v and angular velocity w of the camera from the 1st moment to the nth moment are integrated to obtain the position change t and attitude change R of the camera from the 1st moment to the nth moment. The final R and t are shown in formula (2):

[0136]

[0137] Among them, w k is the angular velocity of the camera in the kth sub-period from the 1st moment to the nth moment, v k is the linear velocity of the camera in the kth sub-period from the 1st moment to the nth moment, is the noise term of angular velocity, is the noise term of the linear velocity, Exp(·) is the exponential mapping, and Δt is the duration of each sub-interval.

[0138] Method 2: For motion sensors including IMU, taking the six-axis IMU as an example, the method for obtaining the position change t and attitude change R is as follows:

[0139] The angular velocity w and acceleration f of the camera in the world coordinate system are obtained through the six-axis IMU. The angular velocity w and acceleration f of the camera from the i-th moment to the j-th moment are integrated to obtain the position change t and attitude change R of the camera from the 1st moment to the nth moment. The final R and t are shown in formula (3):

[0140]

[0141] Among them, w k is the angular velocity of the camera in the nth sub-period from the i-th moment to the 1st moment, f k is the acceleration of the camera in the kth sub-period from the 1st moment to the nth moment, is the paranoid term of angular velocity, is the paranoid term of acceleration, is the noise term of angular velocity, is the noise term of acceleration, and Δt is the duration of each sub-period.

[0142] If the first moment is the i-th moment and the second moment is the j-th moment, then the first sub-period is from the i-th moment to the i+1-th moment, and the above formula (2) can be expressed by the following formula (4), and the above formula (3) can be expressed by the following formula (5):

[0143]

[0144] For the meaning of each symbol in formula (4) and formula (5), please refer to the above and will not be repeated here.

[0145] Method 3: For a motion sensor that includes both a wheel speed meter and an IMU, the position change t is obtained from the wheel speed meter. 1 and attitude change R 1 , the position change t is obtained by IMU 2 and attitude change R 2 , the position change t can be calculated by some filtering or optimization-based calculation methods 1 and position change t 2 Fusion is performed to obtain the position change t and the attitude change R 1 and attitude change R 2 After fusion, the posture change R is obtained.

[0146] By applying the above embodiment, by accurately integrating the angular velocity, linear velocity (or acceleration) data in each sub-period and considering the influence of noise terms and bias terms, the error accumulation is effectively reduced, so that the estimation of the camera pose is closer to the actual situation, thereby improving the accuracy of dynamic area detection. Secondly, the influence of noise terms and bias terms is explicitly considered, and these interference factors are compensated or corrected through algorithms, thereby enhancing the robustness of dynamic area detection and being able to cope with various complex dynamic environments.

[0147] After the transformation matrix is ​​calculated by the above method, the corresponding relationship between the corresponding pixels in the two images can be found according to the transformation matrix and the camera intrinsic parameters of the camera. For example, taking the first moment as the jth moment and the second moment as the ith moment, the pixel coordinates of the same pixel point in the first image are p j , project it into the second image, the pixel coordinate of the pixel is p i , then we have the following formula (6):

[0148] p i =KR ij K -1 p j +Kt ij ...Formula (6)

[0149] Among them, K is the camera internal parameter, t ij is the position change, R ij is the amount of posture change.

[0150] It can be understood that the pixel point in the first image is the projection of the spatial point in the first image coordinate system, so projecting the pixel point in the first image to the second image is equivalent to projecting the spatial point itself to the second image, and the obtained projection point is the same point. Based on this, in some embodiments that can determine the depth of the pixel point or the three-dimensional map point, in addition to the pixel point according to p j The calculation method is to calculate p i For example, a first image and a second image are obtained by using a depth camera, p j The corresponding three-dimensional map point is denoted as P j , then we have the following formula (7):

[0151] p i =K(R ij P j +t ij )...Formula (7)

[0152] According to the above formula (6) or formula (7), the scene flow It can be calculated using the following formula (8):

[0153]

[0154] The second optical flow value is used to represent the relative motion between each pixel in the first image and each pixel in the second image. In a possible embodiment, the first image and the second image can be input into an optical flow estimation deep neural network, such as FlowNet (an optical flow estimation model), to obtain the second optical flow value used to represent the relative motion between each pixel in the first image and each pixel in the second image. Other deep models can also be used.

[0155] By applying the above embodiment, the transformation matrix is ​​first calculated according to the change in the camera's posture, and the pixel points in the first image are accurately projected into the second image using the matrix and the camera's internal parameters. The first optical flow value is obtained by calculating the difference between the projected point and the actual pixel point of the second image, thereby reducing the error accumulation caused by the dynamic change of the scene and improving the accuracy of the first optical flow value. The second optical flow value obtained by processing the first and second images using the optical flow estimation deep neural network improves the efficiency of obtaining the optical flow value, thereby improving the accuracy and efficiency of dynamic area detection.

[0156] In S103, if F 1 Represents the first optical flow value, expressed as F 2 Represents the second optical flow value, expressed as F 3 represents the third optical flow value, which can be calculated by the following formula (9):

[0157] F 3 =F 2 -F 1 ...Formula (9)

[0158] In S104, in order to accurately distinguish the dynamic area from the static area, it is necessary to cluster the pixels in the second image according to the third optical flow value, that is, to group the pixels in the second image according to the similarity of their motion trends, that is, to cluster the pixels with the same or similar motion characteristics into the same area, thereby defining each target clustering area. These target clustering areas can intuitively reflect the dynamically changing parts in the image.

[0159] In order to improve the accuracy, efficiency and robustness of dynamic area detection, in a possible embodiment, the pixel points in each target detection area can be clustered according to the modulus or direction of the third optical flow value, or the optical flow and direction, to obtain each target clustering area.

[0160] By applying the above-mentioned embodiments, a method of clustering pixel points within the target detection area based on the modulus and / or direction of the third optical flow value is adopted, which can accurately and efficiently identify and separate dynamic targets, and can also maintain high robustness in complex environments, such as coping with challenges such as lighting changes and partial occlusion. It not only improves the accuracy of dynamic area detection, but also improves the efficiency and robustness of dynamic area detection.

[0161] In a possible embodiment, all pixels in the second image may be clustered, but it is understandable that the area that can be detected as a dynamic area is often an area where objects with mobility exist. Mobility here refers to active mobility, such as humans and automatic guided robots, while objects such as books, stools, and trees do not have active mobility. For the sake of convenience, objects with active mobility are referred to as target objects in the following text.

[0162] However, in some application scenarios, the image may only contain a small proportion of the target object. In this case, if all pixels in the entire second image are clustered, it may cause a waste of resources and consume a lot of time, thereby resulting in low efficiency of dynamic area detection.

[0163] Based on this, in order to improve the efficiency of dynamic area detection, improve resource utilization, and reduce the cost of dynamic area detection, in another possible embodiment, the target detection area where each target object is located can be detected in the second image in advance, with the aim of quickly locating the area that is most likely to contain dynamic content. Subsequently, only the pixels within these potential dynamic areas are clustered based on the third optical flow value. Specifically, a target detection algorithm based on the YOLO (You Only Look Once, a real-time target detection algorithm) series can be used to detect the target detection area where each target object is located in the second image.

[0164] For example, see Figure 3 , Figure 3 The first schematic diagram of the dynamic mask generation provided in the embodiment of the present application, in which a pedestrian is used as the target object for illustration. The location of the pedestrian is clearly marked by a black frame 301. The area within the black frame 301 is the target detection area. After clustering the pixels in the area based on the third optical flow value, the target cluster area is obtained. According to the third optical flow value, the target cluster area that meets the preset dynamic conditions is determined as the target dynamic area. This process is to determine the mask map of the pedestrian, see Figure 4 , Figure 4 This is a second schematic diagram of dynamic mask generation provided in an embodiment of the present application, in which the black area is a pedestrian mask.

[0165] By applying the above embodiment, the target detection area where each target object is located is accurately detected and located in the second image. Based on the third optical flow value, only the pixels in these target detection areas are clustered and analyzed, which effectively narrows the scope of subsequent processing and focuses only on objects with mobility, such as people, automatic guided robots, etc., thereby eliminating the interference of a large amount of static background information, reducing unnecessary computing resource consumption, improving the efficiency of dynamic area detection, improving resource utilization, and reducing the cost of dynamic area detection. It can also deeply capture the motion characteristics inside the target object, merge pixels with similar motion trends into the same target clustering area, improve the accuracy of clustering, ensure the refinement of dynamic area recognition, and significantly improve the accuracy of dynamic area recognition.

[0166] The above method for obtaining the target detection area is only a preliminary screening of the target objects in the second image. The obtained target detection area includes not only target objects with the ability to move, but also a small number of non-target objects without the ability to move. There may also be situations where target objects with the ability to move do not move in the scene. Therefore, in order to further improve the accuracy of dynamic area detection, it is necessary to further detect the target objects in the target detection area that actually move in the scene.

[0167] In a possible embodiment, an image segmentation method can be used to distinguish the target object from the background in the target detection area. However, this method still cannot distinguish between the target object that actually moves in the scene and the target object that does not move in the scene, resulting in low accuracy of dynamic area detection.

[0168] Based on this, in another possible embodiment, in order to improve the accuracy of dynamic area detection, the differences between the third optical flow values ​​of each pixel point in each target detection area can be determined respectively, and then according to the calculated difference and the preset difference threshold, the target detection area with a difference greater than the preset difference threshold is used as the cluster candidate area. After obtaining the cluster candidate area, the pixel points in each cluster candidate area are clustered according to the third optical flow value to obtain each target cluster area.

[0169] By applying the above embodiment, by respectively determining the difference between the third optical flow values ​​of each pixel point in each target detection area, the target detection area with a difference greater than a preset difference threshold is determined as a clustering candidate area, and then the pixel points in each clustering candidate area are clustered according to the third optical flow value to obtain each target clustering area, which effectively screens out the area where the target object that actually moves is located, and excludes the area where the non-target objects that are stationary or have a small motion amplitude are located, thereby improving the accuracy and efficiency of clustering, and further improving the accuracy and efficiency of dynamic area detection.

[0170] In a possible embodiment, the difference between the third optical flow values ​​of each pixel point in the above-mentioned target detection area can be obtained by calculating the absolute difference or relative difference of the third optical flow values ​​of each pixel point, but this method requires a large amount of calculation for each pixel point, resulting in a low efficiency of the difference calculation process.

[0171] Based on this, in another possible embodiment, in order to more accurately and efficiently obtain the difference between the third optical flow values ​​of each pixel point in each target detection area, the variance between the third optical flow values ​​of each pixel point in each target detection area can be used as the difference. When determining the cluster candidate area, the target detection area with a variance greater than a preset variance threshold can be used as the cluster candidate area. Exemplarily, taking the first moment as the i-th moment and the second moment as the j-th moment as an example, the variance between the third optical flow values ​​of each pixel point in each target detection area can be calculated by the following formula (10):

[0172]

[0173] in, Represents the third optical flow value of the pixel in the i-th row and j-th column in the target detection area The module length, Represents the mean of the modulus lengths.

[0174] Assume that the preset variance threshold is σ Th , the following formula (11) can be used to determine whether the target detection area to which each pixel belongs is a cluster candidate area:

[0175] σ>σ Th ...Formula (11)

[0176] When formula (11) holds true, the target detection area described by the pixel is the cluster candidate area:

[0177] The preset variance threshold may be set by professional technicians based on their work experience or formulated according to industry regulations, and is not specifically limited here.

[0178] By applying the above embodiment, the motion difference between pixels can be efficiently obtained, and complex point-to-point comparison can be avoided, thereby greatly reducing the computational complexity. Furthermore, by determining the target detection area with a variance greater than a preset variance threshold as the clustering candidate area, the area where the target object actually moves can be accurately screened out, thereby effectively improving the efficiency and accuracy of dynamic area detection.

[0179] Furthermore, considering that among the non-target objects that do not have the ability to move, there may be non-target objects that are affected by the target object and move. For example, the bag itself is a non-target object that does not have the ability to move, but when it is held in the hand of a target object, the bag also moves when the person moves. Based on this, the cluster candidate areas can be expanded respectively to obtain target expansion areas, and then the pixels in each target expansion area are clustered according to the third optical flow value to obtain each target cluster area.

[0180] For a target object that is determined to have the ability to move, the candidate clustering area where the target object is located will be expanded to facilitate the discovery of objects around it that are affected by it and move.

[0181] For example, see Figure 5 , Figure 5 The first schematic diagram of non-detection dynamic mask generation provided in the embodiment of the present application is shown in FIG. Figure 5 In the above figure, the pedestrian is the detected target object, and the object in his hand is a non-target object that cannot be detected. The area marked by the black frame 501 is the cluster candidate area. At this time, the object that moves due to the influence of the pedestrian is not located in the cluster candidate area. For pedestrians who are determined to be dynamic, in order to detect the dynamic area where the object they carry is located, the pedestrian detection frame is expanded to facilitate processing of a larger range of areas, see Figure 6 , Figure 6 The second schematic diagram of the non-detection dynamic mask generation provided in the embodiment of the present application performs expansion processing on the cluster candidate area to obtain the target expansion area indicated by the black frame 601. On this basis, considering that the pedestrian is a non-rigid body movement, the movement amplitude of different parts of the body is inconsistent, and the generated optical flow values ​​are not the same, so according to the third optical flow value, the optical flow values ​​of the pixels in each target expansion area are clustered into multiple categories, which can better realize the generation of the mask and obtain each target cluster area, see Figure 7 , Figure 7 This is a third schematic diagram of non-detection dynamic mask generation provided in an embodiment of the present application. The black area in the figure is the mask of pedestrians and objects, that is, the target dynamic area obtained by detection.

[0182] By applying the above embodiment, by expanding the clustered candidate areas to include non-direct moving objects that are potentially affected by the target object, and then accurately clustering them based on the third optical flow value, it is ensured that the target object and the area within its influence range can be accurately identified and distinguished even in complex dynamic scenes, which not only improves the accuracy of dynamic area detection, but also improves the efficiency and robustness of dynamic area detection.

[0183] In S105, the preset dynamic conditions can be set according to the specific needs of the user. For example, the third optical flow values ​​of the pixels contained in each target clustering area can be sorted first, and then the target clustering area containing the largest third optical flow value can be determined as the target dynamic area. Alternatively, the target clustering areas containing the top three third optical flow values ​​can be determined as the target dynamic area. Alternatively, the target clustering areas containing third optical flow values ​​greater than a specific optical flow value can be determined as the target dynamic area, etc., which are not specifically limited here.

[0184] In order to more clearly illustrate the dynamic area detection method of the present application, the following will be described in detail with reference to a specific flow chart and a specific application scenario:

[0185] See also Figure 8 , Figure 8 A second flow chart of the dynamic area detection method provided in the embodiment of the present application includes:

[0186] S801, motion prediction generates scene flow; the scene flow is the aforementioned first optical flow value, which is equivalent to the first optical flow value obtained according to the posture change in the aforementioned step S102 for representing the relative motion caused by the posture change;

[0187] S802, optical flow tracking generates an optical flow; the optical flow is the aforementioned second optical flow value, which is equivalent to obtaining the second optical flow value for representing the relative motion between each pixel in the first image and each pixel in the second image according to the first image and the second image in the aforementioned step S102;

[0188] S803, scene flow compensation optical flow; that is, the aforementioned step S103, calculating the difference between the first optical flow value and the second optical flow value as the third optical flow value;

[0189] S804, target detection generates an identification frame; that is, the target detection area where each target object is located is detected in the second image in the above step;

[0190] S805, dynamic identification of target regions; that is, the above steps respectively determine the differences between the third optical flow values ​​of each pixel point in each target detection region, and determine the target detection region whose difference is greater than a preset difference threshold as a clustering candidate region;

[0191] S806, dynamic target mask generation; that is, the aforementioned step clusters the pixels in the second image according to the third optical flow value to obtain target clustering regions, and determines the target clustering region whose pixel third optical flow values ​​meet the preset dynamic conditions as the target dynamic region.

[0192] In actual application scenarios, see Fig. 9 , Fig. 9This is a first schematic diagram of dynamic area detection provided in an embodiment of the present application. In the figure, the target object that has been detected to have moved in the scene is marked with a detection frame 901, and the area marked with the detection frame 901 is the target detection area. The target object that has not been detected to have moved in the scene is marked with a detection frame 902.

[0193] In order to detect whether the area where the plastic bags carried by pedestrians are located is a dynamic area, the target detection area is expanded, that is, the detection frame is expanded. Fig. 9 The area marked by the detection box 9011 in the figure is the target expanded area obtained after the target detection area is expanded. The detection box of the target object that has not moved can also be expanded. The area marked by the detection box 9022 in the figure is the expanded area obtained after the area where the target object that has not moved is expanded.

[0194] For the expanded target area after expansion, cluster it according to the third optical flow value, see Fig.10 , Fig.10 A second schematic diagram of dynamic area detection provided in an embodiment of the present application, Fig.10 Different colors represent different target clustering areas.

[0195] According to each target cluster area, the target cluster area whose pixel points' third optical flow values ​​meet the preset dynamic conditions is determined as the target dynamic area, which is the process of generating a dynamic mask. Fig.11 , Fig.11 A third schematic diagram of dynamic area detection provided in an embodiment of the present application, Fig.11 It is the pixel-level mask of the entire dynamic area of ​​the dynamic pedestrian and the items he carries, that is, the target dynamic area is finally determined.

[0196] Corresponding to the above-mentioned dynamic area detection method, the present application embodiment also provides a dynamic area detection device, see Fig.12 , Fig.12 The structural diagram of the dynamic area detection device provided in the embodiment of the present application includes:

[0197] An image acquisition module 1201 is used to acquire a first image captured by a camera at a first moment, a second image captured by a camera at a second moment, and a position change of the camera from the first moment to the second moment, wherein the position change is measured by a motion sensor;

[0198] An optical flow value acquisition module 1202 is used to obtain, according to the posture change, a first optical flow value for representing a relative motion caused by the posture change, and to obtain, according to the first image and the second image, a second optical flow value for representing a relative motion between each pixel in the first image and each pixel in the second image;

[0199] A difference calculation module 1203, configured to calculate a difference between the first optical flow value and the second optical flow value as a third optical flow value;

[0200] A pixel clustering module 1204, configured to cluster the pixels in the second image according to the third optical flow value to obtain target clustering regions;

[0201] The region determination module 1205 is configured to determine a target cluster region including pixels whose third optical flow values ​​satisfy a preset dynamic condition as a target dynamic region.

[0202] By applying the above embodiment, by fusing multiple images taken by the camera at different times and the information of the posture change between the times, and by calculating the difference (third optical flow value) between the optical flow caused by the camera's own motion (first optical flow value) and the optical flow generated by the actual motion of the image content (second optical flow value), the interference caused by the camera movement rather than the movement of objects in the scene is effectively eliminated, and the accuracy of dynamic area detection is further improved. Furthermore, the third optical flow value is used to perform cluster analysis on the pixels, effectively distinguishing and extracting the target dynamic area, which directly reflects the motion state in the scene. This method overcomes the problem of misjudgment that may be caused by relying solely on the difference between images in the traditional method, and significantly improves the detection efficiency and robustness of dynamic areas in complex dynamic scenes, providing more reliable data support for subsequent visual analysis, target tracking, behavior recognition and other applications.

[0203] In a possible implementation, the pixel clustering module includes:

[0204] A first clustering submodule is used to detect in the second image a target detection area where each target object is located, wherein the target object is an object with the ability to move;

[0205] A second clustering submodule, used for clustering the pixel points in each target detection area according to the third optical flow value to obtain each target clustering area;

[0206] The second clustering submodule comprises:

[0207] A first clustering unit, used to respectively determine the difference between the third optical flow values ​​of each pixel point in each target detection area;

[0208] A second clustering unit is used to determine the target detection area whose difference is greater than a preset difference threshold as a clustering candidate area;

[0209] A third clustering unit, configured to cluster the pixel points in each of the candidate clustering regions according to the third optical flow value to obtain each target clustering region;

[0210] The third clustering unit comprises:

[0211] A first subunit is used to expand the cluster candidate regions respectively to obtain target expanded regions;

[0212] A second subunit is used to cluster the pixel points in each of the target expansion regions according to the third optical flow value to obtain each target cluster region;

[0213] The second subunit comprises:

[0214] A first subcomponent is used to cluster the pixel points in each target detection area according to the modulus and / or direction of the third optical flow value to obtain each target clustering area;

[0215] The first clustering unit comprises:

[0216] A third subunit, used to respectively calculate the variance between the third optical flow values ​​of each pixel point in each target detection area;

[0217] The second clustering unit comprises:

[0218] The fourth subunit is used to determine the target detection area whose variance is greater than a preset variance threshold as a clustering candidate area.

[0219] The optical flow value acquisition module includes:

[0220] A first acquisition submodule, configured to calculate, according to the posture change, a transformation matrix for representing a posture change amount of the camera from the first moment to the second moment;

[0221] A second acquisition submodule, used for projecting each pixel point in the first image, or a space point corresponding to each pixel point, into the second image according to the transformation matrix and the camera intrinsic parameters of the camera, to obtain each projection point;

[0222] A third acquisition submodule, used to calculate the difference between each of the projection points and each of the pixel points in the first image as a first optical flow value;

[0223] The optical flow value acquisition module includes:

[0224] The fourth acquisition submodule is used to input the first image and the second image into the optical flow estimation deep neural network to obtain a second optical flow value used to represent the relative motion between each pixel in the first image and each pixel in the second image.

[0225] The device also includes:

[0226] The motion sensor includes a wheel speed meter, and the transformation matrix is ​​calculated by the following formula:

[0227]

[0228] Where T is the transformation matrix, w k is the angular velocity of the camera in the kth sub-period, v k is the linear velocity of the camera in the kth sub-period, is the noise term of the angular velocity, is the noise term of the linear velocity, Δt is the duration of each sub-period;

[0229] or,

[0230] The motion sensor includes an inertial measurement unit, and the transformation matrix is ​​calculated by the following formula:

[0231]

[0232] Among them, T ij is the transformation matrix, w k is the angular velocity of the camera in the kth sub-period, f k is the acceleration of the camera in the kth sub-period, is the paranoid term of the angular velocity, is the paranoid term of the acceleration, is the noise term of the angular velocity, is the noise term of the acceleration, and Δt is the duration of each sub-period.

[0233] The present application also provides an electronic device, such as Fig.13 As shown, including:

[0234] Memory 1301, used for storing computer programs;

[0235] The processor 1302 is used to implement the following steps when executing the program stored in the memory 1301:

[0236] Acquire a first image captured by a camera at a first moment, a second image captured by a camera at a second moment, and a position change of the camera from the first moment to the second moment, wherein the position change is measured by a motion sensor;

[0237] Obtaining, according to the position change, a first optical flow value for representing a relative motion caused by the position change, and obtaining, according to the first image and the second image, a second optical flow value for representing a relative motion between each pixel in the first image and each pixel in the second image;

[0238] Calculating a difference between the first optical flow value and the second optical flow value as a third optical flow value;

[0239] Clustering the pixel points in the second image according to the third optical flow value to obtain target clustering areas;

[0240] A target clustering region including pixels whose third optical flow values ​​satisfy a preset dynamic condition is determined as a target dynamic region.

[0241] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 1302, the communication interface, and the memory 1301 communicate with each other via the communication bus.

[0242] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0243] The communication interface is used for communication between the above electronic device and other devices.

[0244] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0245] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0246] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned dynamic area detection methods are implemented.

[0247] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the dynamic area detection methods in the above embodiments.

[0248] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0249] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0250] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0251] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A dynamic area detection method, It is characterized in that The method comprises: Acquire a first image captured by a camera at a first moment, a second image captured by a camera at a second moment, and a position change of the camera from the first moment to the second moment, wherein the position change is measured by a motion sensor; Obtaining, according to the position change, a first optical flow value for representing a relative motion caused by the position change, and obtaining, according to the first image and the second image, a second optical flow value for representing a relative motion between each pixel in the first image and each pixel in the second image; Calculating a difference between the first optical flow value and the second optical flow value as a third optical flow value; Clustering the pixel points in the second image according to the third optical flow value to obtain target clustering areas; A target clustering region including pixels whose third optical flow values ​​satisfy a preset dynamic condition is determined as a target dynamic region.

2. The method according to claim 1, It is characterized in that The clustering of the pixel points in the second image according to the third optical flow value to obtain target clustering areas includes: Detecting in the second image a target detection area where each target object is located, wherein the target object is an object capable of moving; The pixel points in each of the target detection areas are clustered according to the third optical flow value to obtain target cluster areas.

3. The method according to claim 2, It is characterized in that Clustering the pixel points in each target detection area according to the third optical flow value to obtain each target clustering area includes: Respectively determining the difference between the third optical flow values ​​of each pixel point in each target detection area; Determine the target detection area whose difference is greater than a preset difference threshold as a clustering candidate area; According to the third optical flow value, the pixel points in each of the cluster candidate areas are clustered to obtain each target cluster area.

4. The method according to claim 2, It is characterized in that The step of clustering the pixel points in each of the candidate cluster regions according to the third optical flow value to obtain each target cluster region includes: Expanding the cluster candidate regions respectively to obtain target expanded regions; The pixel points in each of the target expansion regions are clustered according to the third optical flow value to obtain each target cluster region.

5. The method according to claim 2, It is characterized in that Clustering the pixel points in each target detection area according to the third optical flow value to obtain each target clustering area includes: The pixel points in each target detection area are clustered according to the module length and / or direction of the third optical flow value to obtain each target clustering area.

6. The method according to claim 3, It is characterized in that The respectively determining the difference between the third optical flow values ​​of each pixel point in each target detection area comprises: Calculating the variance of the third optical flow values ​​of each pixel point in each target detection area respectively; The step of determining the target detection area whose difference is greater than a preset difference threshold as a cluster candidate area includes: The target detection area whose variance is greater than a preset variance threshold is determined as a clustering candidate area.

7. The method according to claim 1, It is characterized in that The obtaining, according to the posture change, a first optical flow value for representing the relative motion caused by the posture change includes: Calculating, according to the posture change, a transformation matrix for representing a posture change amount of the camera from the first moment to the second moment; According to the transformation matrix and the camera intrinsic parameters of the camera, each pixel point in the first image, or a spatial point corresponding to each pixel point, is projected into the second image to obtain each projection point; Calculate the difference between each of the projection points and each of the pixel points in the first image as a first optical flow value; The obtaining, according to the first image and the second image, a second optical flow value for representing a relative motion between each pixel in the first image and each pixel in the second image includes: The first image and the second image are input into an optical flow estimation deep neural network to obtain a second optical flow value for representing the relative motion between each pixel in the first image and each pixel in the second image.

8. The method according to claim 7, It is characterized in that The method further comprises: The motion sensor includes a wheel speed meter, and the transformation matrix is ​​calculated by the following formula: Where T is the transformation matrix, w k is the angular velocity of the camera in the kth sub-period, v k is the linear velocity of the camera in the kth sub-period, is the noise term of the angular velocity, is the noise term of the linear velocity, Δt is the duration of each sub-period; or, The motion sensor includes an inertial measurement unit, and the transformation matrix is ​​calculated by the following formula: Among them, T ij is the transformation matrix, w k is the angular velocity of the camera in the kth sub-period, f k is the acceleration of the camera in the kth sub-period, is the paranoid term of the angular velocity, is the paranoid term of the acceleration, is the noise term of the angular velocity, is the noise term of the acceleration, and Δt is the duration of each sub-period.

9. A dynamic area detection device, It is characterized in that The device comprises: An image acquisition module, used to acquire a first image captured by a camera at a first moment, a second image captured by a camera at a second moment, and a position change of the camera from the first moment to the second moment, wherein the position change is measured by a motion sensor; an optical flow value acquisition module, configured to obtain, according to the posture change, a first optical flow value for representing a relative motion caused by the posture change, and to obtain, according to the first image and the second image, a second optical flow value for representing a relative motion between each pixel in the first image and each pixel in the second image; a difference calculation module, configured to calculate a difference between the first optical flow value and the second optical flow value as a third optical flow value; A pixel clustering module, used for clustering the pixels in the second image according to the third optical flow value to obtain target clustering areas; The region determination module is used to determine a target cluster region including pixels whose third optical flow values ​​meet a preset dynamic condition as a target dynamic region.

10. The device according to claim 9, It is characterized in that The pixel clustering module comprises: A first clustering submodule is used to detect in the second image a target detection area where each target object is located, wherein the target object is an object with the ability to move; A second clustering submodule, used for clustering the pixel points in each target detection area according to the third optical flow value to obtain each target clustering area; The second clustering submodule comprises: A first clustering unit, used to respectively determine the difference between the third optical flow values ​​of each pixel point in each target detection area; A second clustering unit is used to determine the target detection area whose difference is greater than a preset difference threshold as a clustering candidate area; A third clustering unit, configured to cluster the pixel points in each of the candidate clustering regions according to the third optical flow value to obtain each target clustering region; The third clustering unit comprises: A first subunit is used to expand the cluster candidate regions respectively to obtain target expanded regions; A second subunit is used to cluster the pixel points in each of the target expansion areas according to the third optical flow value to obtain each target clustering area; The second subunit comprises: A first subcomponent is used to cluster the pixel points in each target detection area according to the modulus and / or direction of the third optical flow value to obtain each target clustering area; The first clustering unit comprises: A third subunit, used to respectively calculate the variance between the third optical flow values ​​of each pixel point in each target detection area; The second clustering unit comprises: A fourth subunit is used to determine the target detection area whose variance is greater than a preset variance threshold as a clustering candidate area; The optical flow value acquisition module includes: A first acquisition submodule, configured to calculate, according to the posture change, a transformation matrix for representing a posture change amount of the camera from the first moment to the second moment; A second acquisition submodule, used for projecting each pixel point in the first image, or a space point corresponding to each pixel point, into the second image according to the transformation matrix and the camera intrinsic parameters of the camera, to obtain each projection point; A third acquisition submodule, used to calculate the difference between each of the projection points and each of the pixel points in the first image as a first optical flow value; The optical flow value acquisition module includes: a fourth acquisition submodule, configured to input the first image and the second image into an optical flow estimation deep neural network to obtain a second optical flow value for representing a relative motion between each pixel in the first image and each pixel in the second image; The device also includes: The motion sensor includes a wheel speed meter, and the transformation matrix is ​​calculated by the following formula: Where T is the transformation matrix, w k is the angular velocity of the camera in the kth sub-period, v k is the linear velocity of the camera in the kth sub-period, is the noise term of the angular velocity, is the noise term of the linear velocity, Δt is the duration of each sub-period; or, The motion sensor includes an inertial measurement unit, and the transformation matrix is ​​calculated by the following formula: Among them, T ij is the transformation matrix, w k is the angular velocity of the camera in the kth sub-period, f k is the acceleration of the camera in the kth sub-period, is the paranoid term of the angular velocity, is the paranoid term of the acceleration, is the noise term of the angular velocity, is the noise term of the acceleration, and Δt is the duration of each sub-period.

11. A mobile robot, It is characterized in that include: A camera, used for capturing a first image at a first moment and capturing a second image at a second moment; A motion sensor, used for detecting the position information of the camera in real time; A processor is provided to obtain a first image captured by the camera at the first moment, a second image captured at the second moment, and the posture information of the camera detected in real time by the motion sensor, and calculate the posture change of the camera from the first moment to the second moment according to the posture information; obtain a first optical flow value for representing the relative motion caused by the posture change according to the posture change, and obtain a second optical flow value for representing the relative motion between each pixel in the first image and each pixel in the second image according to the first image and the second image; and calculate the difference between the first optical flow value and the second optical flow value as a third optical flow value; According to the third optical flow value, the pixels in the second image are clustered to obtain target cluster regions; and the target cluster region in which the third optical flow values ​​of the pixels contained meet a preset dynamic condition is determined as a target dynamic region.

12. An electronic device, It is characterized in that include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-8 when executing a program stored in a memory.

13. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images