Target position detection method, system and device and storage medium

Abstract

The invention provides a target position detection method, system and device and a storage medium, and the method comprises the steps: obtaining a conversion relation between a camera coordinate system of a monocular camera and a calibration plate coordinate system, the calibration plate coordinate system being a three-dimensional coordinate system established based on a calibration plate placed on a specified plane; acquiring a detection image shot by the monocular camera; performing depth estimation and target identification on the detection image, and obtaining a three-dimensional coordinate of a target in a camera coordinate system; converting the three-dimensional coordinate of the target in the camera coordinate system into the three-dimensional coordinate of the target in the calibration plate coordinate system based on the conversion relation between the camera coordinate system and the calibration plate coordinate system; and performing target position detection according to the three-dimensional coordinates of the target in the calibration plate coordinate system. According to the method and the device, the target three-dimensional coordinates in the camera coordinate system are converted into the calibration plate coordinate system, so that the visual angle can be converted into the visual angle perpendicular to the specified plane, errors caused by the visual angle are reduced, and the target position detection accuracy is improved.

Description

Background Art

[0002] With the increasing importance of prison security management, how to effectively monitor and identify abnormal behaviors of personnel in prisons is very critical. Effective abnormal behavior monitoring can not only improve the security of prisons, but also provide timely warnings when incidents occur, reducing potential losses. Traditional video surveillance systems mainly rely on manual monitoring, which has many limitations in monitoring capabilities and efficiency. First, manual monitoring is easily affected by factors such as the monitor's distraction and fatigue, resulting in unstable monitoring effects. To ensure comprehensive monitoring, a large amount of human resources are required for continuous monitoring, which is costly. However, manual monitoring cannot achieve efficient monitoring all-weather and without blind spots, and it is easy to have monitoring blind spots. Secondly, since the surveillance video is usually a two-dimensional image and lacks depth information, it is difficult for monitoring personnel to accurately judge the real distance and position of the object, especially in a complex prison environment. The camera is usually shot at a bird's-eye view. The two-dimensional image under this perspective is difficult to provide clear observation information, which increases the difficulty of monitoring. At present, most of the existing behavior detection systems on the market monitor abnormal behavior through image processing such as motion detection and contour extraction. They are still at the two-dimensional image analysis stage and lack effective use of depth information. At the same time, these methods are highly dependent on features and have limited adaptability to complex scenes.

[0003] In recent years, with the development of computer vision technology, especially the progress of monocular depth estimation algorithms, new technical means have been provided to solve the above problems. Monocular depth estimation algorithms can restore the three-dimensional coordinates of objects in the scene by analyzing a single two-dimensional image, thus realizing a three-dimensional understanding of the scene. Using a monocular depth estimation algorithm, the depth information of video frames acquired by different monitoring can be restored. Compared with technologies such as binocular stereo vision or lidar, this method has the advantages of low cost, no need to install and maintain chemical instruments, and easy deployment. It can be widely used in fields such as intelligent monitoring. However, the existing monocular depth estimation algorithm is mainly based on the understanding of two-dimensional images to obtain depth information. It is limited by the camera's viewing angle and will deviate from the actual position of the target due to the influence of the camera's viewing angle, resulting in inaccurate target position detection. Summary of the invention

[0004] In response to the problems in the prior art, the purpose of this application is to provide a target position detection method, system, device and storage medium. By converting the target three-dimensional coordinates in the camera coordinate system into the calibration plate coordinate system, the viewing angle can be converted into a viewing angle perpendicular to the specified plane, thereby reducing the error caused by the viewing angle and improving the accuracy of target position detection.

[0005] The present invention provides a method for detecting a target position, comprising the following steps:

[0006] Obtain the conversion relationship between the camera coordinate system of the monocular camera and the calibration plate coordinate system, where the calibration plate coordinate system is a three-dimensional coordinate system established based on the calibration plate placed on a specified plane;

[0007] Get the detection image taken by the monocular camera;

[0008] Perform depth estimation and target recognition on the detection image to obtain the three-dimensional coordinates of the target in the camera coordinate system;

[0009] Based on the conversion relationship between the camera coordinate system and the calibration plate coordinate system, the three-dimensional coordinates of the target in the camera coordinate system are converted into the three-dimensional coordinates of the target in the calibration plate coordinate system;

[0010] The target position detection is performed according to the three-dimensional coordinates of the target in the calibration plate coordinate system. The target position detection includes detecting the position of the target in the specified plane and / or detecting the height relative to the specified plane.

[0011] By adopting the present invention, firstly, the conversion relationship between the camera coordinate system and the calibration plate coordinate system is obtained through calibration. After the three-dimensional coordinates of the specific target in the detection image in the camera coordinate system are obtained, the three-dimensional coordinates of the target in the camera coordinate system are converted to the calibration plate coordinate system, so that the position coordinates of the target in the viewing angle perpendicular to the specified plane can be obtained, and the error caused by the camera shooting viewing angle is eliminated. The target position is detected based on the three-dimensional coordinates of the target in the calibration coordinate system. Since the position detection of the target has been converted into the position detection in the viewing angle perpendicular to the specified plane, that is, the position detection in the horizontal view relative to the specified plane, when detecting the position of the target in the specified plane or the height of the target relative to the specified target, there will be no deviation in position judgment due to the viewing angle offset, thereby improving the accuracy of target position detection.

[0012] In some embodiments, obtaining the conversion relationship between the camera coordinate system of the monocular camera and the calibration plate coordinate system includes the following steps:

[0013] Obtain the actual three-dimensional coordinates of each calibration plate feature point in the calibration plate placed on the specified plane in the calibration plate coordinate system;

[0014] Obtain the calibration plate image taken by the camera, and determine the pixel coordinates of each calibration plate feature point in the calibration plate image;

[0015] According to the actual three-dimensional coordinates and pixel coordinates of each calibration plate feature point in the calibration plate coordinate system, an external parameter calibration algorithm is used to obtain the rotation matrix from the camera coordinate system of the monocular camera to the calibration plate coordinate system;

[0016] According to the calibration plate image and rotation matrix, calculate the translation vector from the camera coordinate system to the calibration plate coordinate system.

[0017] In some embodiments, calculating the translation vector from the camera coordinate system to the calibration plate coordinate system includes the following steps:

[0018] A monocular depth estimation model is used to estimate the depth of the calibration plate image to obtain the depth information of each pixel in the calibration plate image.

[0019] According to the depth information and the internal parameters of the monocular camera, the detection three-dimensional coordinates of each pixel in the calibration plate image in the camera coordinate system are obtained;

[0020] The translation vector from the camera coordinate system to the calibration plate coordinate system is calculated according to the detected three-dimensional coordinates, the actual three-dimensional coordinates and the rotation matrix of a reference feature point among the feature points of the calibration plate.

[0021] In some embodiments, after calculating the translation vector from the camera coordinate system to the calibration plate coordinate system, the following steps are also included:

[0022] Calculate the verification three-dimensional coordinates of the feature points of the calibration plate in the calibration plate coordinate system according to the three-dimensional coordinates, rotation matrix and translation vector of the feature points of the calibration plate in the camera coordinate system;

[0023] According to the comparison results between the verified 3D coordinates of the feature points of the calibration plate and the actual 3D coordinates, verify whether the rotation matrix and the translation vector meet the calibration requirements;

[0024] If not, repeat the step of obtaining the conversion relationship between the coordinate system of the monocular camera and the coordinate system of the calibration plate.

[0025] In some embodiments, performing depth estimation and target recognition on the detection image to obtain the three-dimensional coordinates of the target in the camera coordinate system includes the following steps:

[0026] Perform depth estimation and coordinate recognition on the detection image to obtain point cloud information of the detection image. The point cloud information includes the three-dimensional coordinates of each pixel point in the detection image in the camera coordinate system.

[0027] Perform target recognition on the detection image to determine the pixel position of the target in the detection image;

[0028] The three-dimensional coordinates of the target in the camera coordinate system are obtained according to the pixel position of the target in the detection image.

[0029] In some embodiments, performing depth estimation and coordinate recognition on the detection image to obtain point cloud information of the detection image includes the following steps:

[0030] A monocular depth estimation model is used to estimate the depth of the detection image to obtain the depth information of each pixel in the detection image;

[0031] According to the intrinsic parameters of the camera and the depth information of each pixel in the detection image, the three-dimensional coordinates of each pixel in the detection image in the camera coordinate system are obtained as point cloud information.

[0032] In some embodiments, the designated plane is a horizontal plane, and the three-dimensional coordinates of the target in the calibration plate coordinate system include a horizontal component and a vertical component; performing target position detection according to the three-dimensional coordinates of the target in the calibration plate coordinate system includes the following steps:

[0033] Determine whether the target exceeds the preset height range according to the vertical component of the target's three-dimensional coordinates in the calibration plate coordinate system;

[0034] The horizontal component of the target's three-dimensional coordinates in the calibration plate coordinate system is used to determine whether the target exceeds the preset horizontal position range.

[0035] The present application also provides a target position detection system, which is applied to the above target position detection method. The system includes:

[0036] A conversion relationship acquisition module is used to obtain the conversion relationship between the camera coordinate system of the monocular camera and the calibration plate coordinate system, where the calibration plate coordinate system is a three-dimensional coordinate system established based on a calibration plate placed on a specified plane;

[0037] An image acquisition module is used to obtain the detection image taken by the monocular camera;

[0038] The target recognition module is used to perform depth estimation and target recognition on the detection image and obtain the three-dimensional coordinates of the target in the camera coordinate system;

[0039] A coordinate conversion module is used to convert the three-dimensional coordinates of the target in the camera coordinate system into the three-dimensional coordinates of the target in the calibration plate coordinate system based on the conversion relationship between the camera coordinate system and the calibration plate coordinate system;

[0040] The position detection module is used to detect the target position according to the three-dimensional coordinates of the target in the calibration plate coordinate system. The target position detection includes detecting the position of the target in the specified plane and / or detecting the height relative to the specified plane.

[0041] The present application also provides a target position detection device, including:

[0042] processor;

[0043] a memory storing executable instructions of the processor;

[0044] The processor is configured to execute the steps of the target position detection method by executing the executable instructions.

[0045] By adopting the target position detection device provided in the present application, the processor executes the target position detection method when executing the executable instructions, thereby obtaining the beneficial effects of the above target position detection method.

[0046] An embodiment of the present application also provides a computer-readable storage medium for storing a program, which implements the steps of the target position detection method when executed by a processor.

[0047] By adopting the computer-readable storage medium provided in the present application, the program stored therein implements the steps of the target position detection method when being executed, thereby obtaining the beneficial effects of the above-mentioned target position detection method. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings.

[0049] Figure 1 is a flow chart of a target position detection method according to an embodiment of the present application;

[0050] Figure 2 is a schematic diagram of coordinate transformation according to an embodiment of the present application;

[0051] Figure 3 It is a flow chart of obtaining the conversion relationship between the camera coordinate system of a monocular camera and the coordinate system of a calibration plate according to an embodiment of the present application;

[0052] Figure 4 is a flow chart of obtaining the three-dimensional coordinates of a target in a camera coordinate system according to an embodiment of the present application;

[0053] Figure 5 is a schematic diagram of a target position detection system according to an embodiment of the present application;

[0054] Figure 6 is a structural schematic diagram of a target position detection device according to an embodiment of the present application;

[0055] Figure 7 It is a schematic diagram of the structure of a computer storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and the concepts of the example embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the figures represent the same or similar structures, and thus their repeated description will be omitted. Although "first" or "second" etc. are used in this specification to represent certain features, they are only used to represent the function and are not used as a limitation on the number and importance of specific features.

[0057] like Figure 1 As shown, in one embodiment, the present application provides a target position detection method, comprising the following steps:

[0058] S100: Acquire a conversion relationship between a camera coordinate system of a monocular camera and a calibration plate coordinate system, where the calibration plate coordinate system is a three-dimensional coordinate system established based on a calibration plate placed on a specified plane;

[0059] The conversion relationship here refers to the conversion relationship of the coordinates in the camera coordinate system of the monocular camera to the coordinates in the calibration plate coordinate system, for example, including the rotation matrix and translation vector that convert the coordinates in the camera coordinate system to the coordinates in the calibration plate coordinate system. The coordinates of a point in the camera coordinate system are multiplied by the rotation matrix and added with the translation vector, and the result is the coordinates of the point in the calibration plate coordinate system.

[0060] By placing the calibration plate on the designated plane to calibrate the conversion relationship between the camera coordinate system and the calibration plate coordinate system, the original shooting angle of the camera can be converted into an angle of view perpendicular to the designated plane; the specific position of the designated plane can be determined according to the dimension of the target position detection, and the target position detection includes the position detection of the target in the designated plane and / or the height detection relative to the designated plane; in order to detect the moving position of the target in the horizontal plane and / or the height of the target relative to the horizontal plane, the horizontal plane is used as the designated plane; or, in order to detect the moving position of the target in the vertical plane perpendicular to the horizontal plane and / or the height of the target relative to the vertical plane, the vertical plane is used as the designated plane; or, in order to detect the moving position of the target in a specific inclined surface and / or the height of the target relative to the inclined surface, the inclined surface is used as the designated plane; the following will take the designated plane as the horizontal plane as an example for explanation, but the present application is not limited to this;

[0061] like Figure 2As shown, when the designated plane is a horizontal plane, the calibration plate is set on the horizontal plane. The three coordinate axes of the camera coordinate system are represented by x, y, and z, and the three coordinate axes of the calibration plate coordinate system are represented by x', y', and z'. In the calibration plate coordinate system, a feature point of the calibration plate is taken as the coordinate origin O' (for example, the upper left corner of the calibration plate is taken as the coordinate origin, but the present application is not limited to this), the direction from the coordinate origin of the calibration plate to another feature point is taken as the x'-axis direction, and another direction perpendicular to the x'-axis direction is determined on the calibration plate as the y'-axis direction, and the direction perpendicular to the calibration plate, that is, the direction perpendicular to the horizontal plane, is the z'-axis direction; the calibration plate may be, for example, a checkerboard calibration plate, or a calibration plate with other graphics may be used; Figure 2 The position of the camera and the placement of the calibration plate are only examples and are not intended to limit the scope of protection of this application;

[0062] S200: Acquire a detection image captured by a monocular camera;

[0063] This application only needs to acquire the detection image through a monocular camera and analyze it, that is, the three-dimensional coordinates of the target in the camera coordinate system can be obtained without using a binocular camera. In order to eliminate the shooting angle deviation of the monocular camera, it will be further converted to the calibration plate coordinate system later;

[0064] The detection image captured by the monocular camera can be a single image or a frame of an image in a video. When applied to the monitoring field, the monocular camera is the camera of the monitoring equipment, which actually captures the monitoring video in real time, and extracts a frame of image from the monitoring video as the detection image. Each frame of image can be used as the detection image for detection analysis, or a frame of image can be selected as the detection image every preset number of frames;

[0065] S300: performing depth estimation and target recognition on the detection image to obtain the three-dimensional coordinates of the target in the camera coordinate system;

[0066] Performing depth estimation on the detection image, including using a monocular depth estimation model to identify pixel depth information; performing target recognition on the detection image, for example, using a trained target recognition model to identify the pixel position of the target in the detection image, and using the depth information estimated by the monocular depth estimation model combined with the camera intrinsic parameters to further obtain the three-dimensional coordinates of the target in the camera coordinate system;

[0067] S400: based on the conversion relationship between the camera coordinate system and the calibration plate coordinate system, convert the three-dimensional coordinates of the target in the camera coordinate system into the three-dimensional coordinates of the target in the calibration plate coordinate system;

[0068] The obtained three-dimensional coordinates of the target in the calibration plate coordinate system are the coordinates used for subsequent target position detection;

[0069] S500: Detect the target position according to the three-dimensional coordinates of the target in the calibration plate coordinate system. The target position detection includes detecting the position of the target in the specified plane and / or detecting the height relative to the specified plane.

[0070] By adopting the present invention, first, the conversion relationship between the camera coordinate system and the calibration plate coordinate system is obtained through calibration through step S100, and then the three-dimensional coordinates of the specific target in the detection image in the camera coordinate system are obtained through steps S200 to S300. Then, the three-dimensional coordinates of the target in the camera coordinate system are converted to the calibration plate coordinate system through step S400, so that the position coordinates of the target in the viewing angle perpendicular to the specified plane can be obtained, and the error caused by the camera shooting angle can be eliminated. The target position is detected based on the three-dimensional coordinates of the target in the calibration coordinate system through step S500. Since the position detection of the target has been converted into the position detection in the viewing angle perpendicular to the specified plane, that is, the position detection in the horizontal view relative to the specified plane, when detecting the position of the target in the specified plane or the height of the target relative to the specified target, there will be no deviation in position judgment due to the viewing angle offset, thereby improving the accuracy of target position detection. By adopting this application, when applied to prison personnel monitoring, the monocular camera can be used to accurately measure the position of personnel, which can greatly improve the monitoring capability and intelligence level of the prison system without adding additional hardware costs, and provide strong technical support for more efficient public safety management. By judging the position and height of personnel on the horizontal plane that have crossed the boundary, the climbing and crossing behaviors of personnel can be quickly identified, the response speed of abnormal events can be improved, and the accuracy and real-time performance of abnormal behavior detection of personnel in prisons can be significantly improved.

[0071] This application can be used not only for detecting the location of people in scenes such as prisons, roads, and squares, but also for detecting the location of specific objects in various scenes, as long as the corresponding target recognition model is selected according to the target type. For example, when used for detecting the location of people in a specific scene, the target recognition model adopts a person recognition model, and sample images of people in the specific scene can be collected, and the person recognition model can be trained after adding the person location label. When used for detecting the location of a specific object, sample images containing the specific object are collected, and the object recognition model is trained after adding the object location label.

[0072] After acquiring the detection image, the depth information of each pixel in the detection image can be estimated through the monocular depth estimation algorithm. Combining the depth information and the camera's internal parameters (camera focal length, principal point coordinates), the depth information can be converted into the three-dimensional coordinates of the target in the camera coordinate system. However, the depth information is obtained by using a machine learning model to predict the depth value of each pixel in the two-dimensional image based on the image features of the detection image. It is not directly obtained through geometric measurement, and there is a certain deviation from the actual position of the actual target in the camera coordinate system. In actual detection scenarios, cameras are usually installed at a higher position with a certain pitch and yaw angle. Although this setting is conducive to covering a larger monitoring range, it also makes it impossible to observe the three-dimensional coordinates of the target directly obtained using the monocular depth estimation model. Due to the influence of the camera height, pitch angle and yaw angle, it is difficult to accurately judge the position of the target and its estimated changes based on the three-dimensional coordinates obtained directly based on the monocular depth estimation model. Therefore, Figure 2 The camera coordinate system xyz in is just a formal representation. It is the estimated camera coordinate system inferred from the detection image by the monocular depth estimation by learning complex feature mapping, which is not completely consistent with the actual camera coordinate system. Because the camera coordinate system estimated by the monocular depth estimation algorithm is not the same as the actual camera coordinate system, if the existing process of using a monocular camera to obtain point cloud information and perform coordinate system conversion, the rotation matrix and translation vector obtained by extrinsic calibration using the calibration plate are reversely projected to obtain the conversion from the camera coordinate system to the world coordinate system, the point cloud information obtained by the monocular camera cannot be accurately converted to the world coordinate system, that is, the reverse projection directly using the calibrated extrinsic parameters cannot be converted to the target coordinate system.

[0073] The present invention converts the three-dimensional coordinates of the target in the camera coordinate system into the three-dimensional coordinates in the calibration plate coordinate system, and standardizes the position information of the target in all directions through the coordinate system conversion to simulate the horizontal viewing angle of the camera optical axis placed upright on the specified plane, thereby realizing the transformation from two-dimensional detection to three-dimensional behavior understanding, thereby ensuring the consistency of three-dimensional coordinate data under different viewing angles, effectively eliminating the measurement errors caused by viewing angle differences, and improving the accuracy and reliability of target detection analysis. The code involved in the present invention can be written in Python3 language or other category languages ​​to realize the automation of target position detection.

[0074] The following takes the position detection of personnel in a prison scene as an example to specifically illustrate the implementation of the target position detection method. It is understandable that the following description is only an example and is not intended to limit the scope of protection of this application. The method can also be applied to other scenes and other types of target position detection.

[0075] In this embodiment, before step S100, the camera is further calibrated with internal parameters, including: providing a calibration plate, and predetermining the calibration plate feature points used for calibration in the calibration plate. The calibration plate is placed on the floor of the prison. A monocular camera is used to take multiple images of the calibration plate, and the camera is calibrated with internal parameters through the feature points in the images to obtain the focal length and principal point coordinates of the camera. The camera internal parameter calibration can adopt the existing internal parameter calibration method. The internal parameter matrix Among them, f x ,f y ,c x ,c y are the focal length of the x and y components and the coordinates of the principal point, respectively.

[0076] like Figure 3 As shown, step S100: obtaining the conversion relationship between the camera coordinate system of the monocular camera and the coordinate system of the calibration plate, includes the following steps:

[0077] S110: Obtain the actual three-dimensional coordinates of each calibration plate feature point in the calibration plate placed on the specified plane in the calibration plate coordinate system; specifically, determine the coordinates of each calibration plate feature point on the x' axis and the y' axis according to the distance between each calibration plate feature point and the feature point as the coordinate origin, and the coordinate of the z' axis is 0, so as to obtain the actual three-dimensional coordinates obj_points in the calibration plate coordinate system;

[0078] S120: Acquire an image of the calibration plate captured by the camera, and determine the pixel coordinates of each feature point of the calibration plate in the image of the calibration plate;

[0079] Specifically, the position of the calibration plate is fixed, and a monocular camera is used to extract frames from the captured video stream to collect an image of the calibration plate. The corner detection algorithm of OpenCV2 (Open Source Computer Vision Library2) is used to determine the pixel coordinates of the feature points. OpenCV 2 provides a series of functions and tools for processing image and video data, helping developers to easily implement computer vision-related applications, such as target detection, image recognition, image segmentation, feature extraction, and many other fields.

[0080] S130: according to the actual three-dimensional coordinates and pixel coordinates of each feature point of the calibration plate in the calibration plate coordinate system, an external parameter calibration algorithm is used to obtain a rotation matrix from the camera coordinate system of the monocular camera to the calibration plate coordinate system;

[0081] Specifically, for each calibration plate feature point, a relationship equation between the three-dimensional coordinates of the calibration plate feature point in the calibration plate coordinate system and the three-dimensional coordinates in the camera coordinate system is constructed, and its basic form includes the rotation matrix R and the translation vector t; if there are n calibration plate feature points, n equations are obtained, forming an overdetermined system of equations, and the unknown quantity includes 9 elements of the rotation matrix R. The optimization algorithm such as the least squares method is used to solve this overdetermined system of equations to find the value of the rotation matrix R that minimizes the error of all equations (for example, it can be solved by calling related functions from OpenCV2) to obtain the rotation matrix R;

[0082] The result of calling the related functions may be a rotation vector rather than a rotation matrix. Therefore, the rotation vector needs to be transformed into a rotation matrix through Rodrigues transformation. The specific transformation process is as follows:

[0083]

[0084] Among them, R 3×3 Represents the rotation matrix. θ=norm(r), r=r / θ, r is the rotation vector, norm(r) represents the modulus of the rotation vector. After determining θ, the rotation vector is normalized by r=r / θ and then inserted into the above formula.

[0085] Then, according to the calibration plate image and the rotation matrix, the translation vector from the camera coordinate system to the calibration plate coordinate system is calculated; specifically, the translation vector is calculated using the following steps:

[0086] S141: Using a monocular depth estimation model to perform depth estimation on the calibration plate image to obtain depth information of each pixel in the calibration plate image;

[0087] S142: Obtain the detection three-dimensional coordinates of each pixel point in the calibration plate image in the camera coordinate system according to the depth information and the internal parameters of the monocular camera;

[0088] Specifically, according to the depth information and the internal parameters of the monocular camera, the three-dimensional coordinates of each pixel in the calibration plate image in the camera coordinate system can be obtained, that is, the point cloud information of the calibration plate image. Here, in order to distinguish it from the actual three-dimensional coordinates in step S110, the three-dimensional coordinates obtained from the calibration plate image analysis are expressed as the detected three-dimensional coordinates;

[0089] S143: Calculate a translation vector from the camera coordinate system to the calibration plate coordinate system according to the detected three-dimensional coordinates, the actual three-dimensional coordinates and the rotation matrix of a reference feature point among the feature points of the calibration plate;

[0090] Calculating the translation vector includes using the rotation matrix to multiply the three-dimensional coordinates of the calibration plate reference feature points in the camera coordinate system by the left hand, and then subtracting the actual three-dimensional coordinates of the calibration plate reference feature points. The calculated translation vector is expressed by the following formula:

[0091]

[0092] Among them, t cam_to_board Represents the translation vector from the camera coordinate system to the calibration plate coordinate system, R 3×3 is the rotation matrix obtained by Rodrigues transformation of the rotation vector obtained when performing external parameter calibration in step S130, ( x′ ,O y′ ,O z′ ) T is the actual three-dimensional coordinate of the reference feature point in the calibration plate coordinate system, ( x ,O y ,O z ) T is the three-dimensional coordinate of the reference feature point in the calibration plate coordinate system; the reference feature point here can be the origin of the calibration plate coordinate system, then ( x′ ,O y′ ,O z′ ) T is (0,0,0) T However, the present application is not limited thereto. The reference feature point may also be another feature point on the calibration plate. Alternatively, in another alternative embodiment, the feature point on the calibration plate is not used to calculate the translation vector. Instead, a specific point on the floor outside the calibration plate is used to replace the reference feature point. Since the actual three-dimensional coordinates of the specific point on the floor in the calibration plate coordinate system can also be actually measured, the detected three-dimensional coordinates of the specific point can also be obtained by detecting and calculating the image containing the specific point. The translation vector can also be calculated based on the actual three-dimensional coordinates of the specific point, the detected three-dimensional coordinates and the rotation matrix.

[0093] After obtaining the rotation matrix and translation vector from the camera coordinate system to the calibration plate coordinate system in step S100, the rotation matrix and translation vector can be used for coordinate conversion in step S400.

[0094] In this embodiment, before the rotation matrix and translation vector obtained in step S100 are applied to the actual detection process, a step of verifying the accuracy of the rotation matrix and translation vector may be included. Specifically, since the actual three-dimensional coordinates of each calibration plate feature point in the calibration plate coordinate system are known, they can be used to verify the accuracy of the coordinate conversion process. After the step S143: calculating the translation vector from the camera coordinate system to the calibration plate coordinate system, the following steps are also included:

[0095] According to the three-dimensional coordinates, rotation matrix and translation vector of the feature points of the calibration plate in the camera coordinate system, the verification three-dimensional coordinates of the feature points of the calibration plate in the calibration plate coordinate system are calculated; the mathematical expression is as follows:

[0096]

[0097] Among them, corners board It represents the verified 3D coordinates of the calibration plate feature points in the calibration plate coordinate system. m and n are the number of columns and rows of the calibration plate feature points arranged in the matrix in the calibration plate. 3×(m-1)×(n-1) The three-dimensional coordinates of the feature points of the calibration plate in the camera coordinate system can be obtained through the pixel coordinates of the feature points of the calibration plate in the calibration plate image and the point cloud information of the calibration plate image.

[0098] According to the comparison results between the verified 3D coordinates of the feature points of the calibration plate and the actual 3D coordinates, verify whether the rotation matrix and the translation vector meet the calibration requirements;

[0099] Here, the root mean square error of the verified three-dimensional coordinates and the actual three-dimensional coordinates can be calculated. The calculation formula is as follows:

[0100]

[0101] Where rmse represents the root mean square error, obj_point represents the actual 3D coordinates of the feature points of the calibration plate. When calculating the root mean square error, the difference between the verified 3D coordinates and the actual 3D coordinates of each feature point of the calibration plate is calculated and squared, and then the squares of the differences corresponding to all feature points of the calibration plate are averaged.

[0102] If not satisfied, repeat the step of obtaining the conversion relationship between the coordinate system of the monocular camera and the coordinate system of the calibration plate. Specifically, if the root mean square error is greater than the preset error threshold, it means that an error occurred during the camera calibration and coordinate system conversion process, and the translation vector of the obtained rotation matrix does not meet the requirements. It is necessary to repeat the step S100 until the root mean square error is less than or equal to the preset error threshold. If the root mean square error is less than or equal to the preset error threshold, the rotation matrix and translation vector are used in the formal target detection process, and steps S200 to S500 are continued. The rotation matrix and translation vector obtained in step S100 are used to transform the three-dimensional coordinates of the target in the camera coordinate system to obtain the three-dimensional coordinates of the target in the calibration plate coordinate system, simulating the three-dimensional point cloud under the normal viewing angle of the camera to ensure the accuracy and consistency of the position information.

[0103] like Figure 4 As shown, the step S300: performing depth estimation and target recognition on the detection image to obtain the three-dimensional coordinates of the target in the camera coordinate system includes the following steps:

[0104] Performing depth estimation and coordinate recognition on the detection image to obtain point cloud information of the detection image, wherein the point cloud information includes the three-dimensional coordinates of each pixel in the detection image in the camera coordinate system; specifically comprising: S311: performing depth estimation on the detection image using a monocular depth estimation model to obtain depth information of each pixel in the detection image; S312: obtaining the three-dimensional coordinates of each pixel in the detection image in the camera coordinate system as point cloud information based on the intrinsic parameters of the camera and the depth information of each pixel in the detection image;

[0105] S320: Perform target recognition on the detection image to determine the pixel position of the target in the detection image; for example, use the YOLOv5 target detection algorithm to obtain a rectangular box [x min ,y min ,x max ,y max ], where x min ,y min ,x max ,y max Represents the pixel coordinate values ​​of the corner points of the rectangular frame in the detection image; therefore, obtaining the pixel position of the target here is to obtain the pixel coordinate range of the target in the detection image;

[0106] S330: Obtaining the three-dimensional coordinates of the target in the camera coordinate system according to the pixel position of the target in the detection image;

[0107] Among them, after obtaining the three-dimensional coordinates of each pixel point in the rectangular box, since the rectangular box is often relatively large and will contain some background information, the rectangular box is first reduced (reduced according to a set multiple, the set multiple is, for example, 10%, 20%, etc.), and then the average of the three-dimensional coordinates of all pixel points in the reduced rectangular box is calculated as the spatial three-dimensional coordinates of the target at the camera's top-down angle, that is, the three-dimensional coordinates of the target in the camera coordinate system.

[0108] The step S400: based on the conversion relationship between the camera coordinate system and the calibration plate coordinate system, converting the three-dimensional coordinates of the target in the camera coordinate system into the three-dimensional coordinates of the target in the calibration plate coordinate system, comprises the following steps:

[0109] While keeping the origin of the calibration plate coordinate system unchanged, the rotation matrix can be directly multiplied by the three-dimensional coordinates of the target in the camera coordinate system, and then added to the translation vector to obtain the three-dimensional coordinates of the target in the calibration plate coordinate system.

[0110] In actual detection, the calibration plate has been removed from the floor. In order to facilitate intelligent monitoring, if the position of the origin of the calibration plate coordinate system is not marked on the floor, the pixel coordinates of the origin of the calibration plate coordinate system in the scene can be re-determined. For example, another feature point on the floor (such as the corner of the floor tile) is selected as the new origin of the calibration plate coordinate system to replace the original calibration plate origin coordinates. At this time, since the position of the origin of the calibration plate coordinate system has changed, the previous translation vector can no longer be used. In step S400, the three-dimensional coordinates of the target in the calibration plate coordinate system are obtained using the following steps:

[0111] The pixel coordinates of the origin of the calibration plate coordinate system are re-determined according to the detection image, and the three-dimensional coordinates of the origin of the calibration plate coordinate system in the camera coordinate system are obtained based on the pixel coordinates of the origin of the calibration plate coordinate system and the point cloud information of the detection image (P x ,P y ,P z ) T , combining the three-dimensional coordinates of the target in the camera coordinate system, the three-dimensional coordinates of the origin of the calibration plate coordinate system in the camera coordinate system, the rotation matrix and the translation vector to obtain the three-dimensional coordinates of the target in the calibration plate coordinate system as follows:

[0112]

[0113] Among them, (x t ′,y t ′,z t ′) T is the three-dimensional coordinate of the target in the calibration plate coordinate system, (x t ,y t ,z t ) T is the three-dimensional coordinate of the target in the camera coordinate system, (O x ,O y ,O z ) T is the three-dimensional coordinate of the origin of the calibration plate coordinate system in the calibration plate coordinate system, that is, (O x′ ,O y′ ,O z′ ) T is (0,0,0) T This formula can be understood as first changing the target point (x t ,y t ,z t ) T The coordinates of the origin of the calibration plate coordinate system in the camera coordinate system (P x ,P y ,P z ) TTranslate it to make it relative to the origin of the new calibration plate coordinate system, and then because these relative points are already aligned with the origin of the calibration plate, apply the rotation matrix to rotate these relative points. Finally, add back the origin of the calibration plate coordinate system in its own coordinate system to get the correct position of the target relative to the calibration plate coordinate system.

[0114] After step S400, if the correct position of the target in the calibration plate coordinate system is obtained for multiple image frames of a video respectively, the video stream output is synthesized based on the three-dimensional coordinates of the target in the calibration plate coordinate system obtained in step S400 for each frame.

[0115] In this embodiment, the designated plane is a horizontal plane, and the three-dimensional coordinates of the target in the calibration plate coordinate system include a horizontal component and a vertical component. The step S500: detecting the target position according to the three-dimensional coordinates of the target in the calibration plate coordinate system includes the following steps.

[0116] Determine whether the target exceeds the preset height range based on the vertical component of the target's three-dimensional coordinates in the calibration plate coordinate system; that is, when the person's z'-axis coordinate value is greater than the preset height range, it is determined that the person has climbed abnormally high;

[0117] Whether the target exceeds the preset horizontal position range is determined based on the horizontal component of the target's three-dimensional coordinates in the calibration plate coordinate system; that is, when the person's x'-axis coordinate value and y'-axis coordinate value exceed the x'-axis coordinate value and y'-axis coordinate value range of the specified area, it is judged as an abnormal crossing of the person.

[0118] After detecting abnormal climbing and / or abnormal crossing of boundaries, a warning can be issued in time and the management staff can be notified in time. Therefore, this application can greatly improve the monitoring capability and intelligence level of the prison without increasing the additional hardware cost, and provide strong technical support for more efficient public safety management.

[0119] like Figure 5 As shown, the embodiment of the present application also provides a target position detection system, which is applied to the target position detection method, and the system includes:

[0120] The conversion relationship acquisition module M100 is used to obtain the conversion relationship between the camera coordinate system of the monocular camera and the calibration plate coordinate system, where the calibration plate coordinate system is a three-dimensional coordinate system established based on the calibration plate placed on a specified plane;

[0121] Image acquisition module M200, used to obtain the detection image taken by the monocular camera;

[0122] The target recognition module M300 is used to perform depth estimation and target recognition on the detection image and obtain the three-dimensional coordinates of the target in the camera coordinate system;

[0123] The coordinate conversion module M400 is used to convert the three-dimensional coordinates of the target in the camera coordinate system into the three-dimensional coordinates of the target in the calibration plate coordinate system based on the conversion relationship between the camera coordinate system and the calibration plate coordinate system;

[0124] The position detection module M500 is used to detect the target position according to the three-dimensional coordinates of the target in the calibration plate coordinate system. The target position detection includes detecting the position of the target in the specified plane and / or detecting the height relative to the specified plane.

[0125] By adopting the present invention, the conversion relationship between the camera coordinate system and the calibration plate coordinate system is firstly obtained by calibration through the conversion relationship acquisition module M100, and then the three-dimensional coordinates of the specific target in the detection image in the camera coordinate system are obtained through the image acquisition module M200 and the target recognition module M300. Then, the three-dimensional coordinates of the target in the camera coordinate system are converted into the calibration plate coordinate system through the coordinate conversion module M400, so that the position coordinates of the target in the viewing angle perpendicular to the specified plane can be obtained, and the error caused by the camera shooting viewing angle can be eliminated. The target position is detected based on the three-dimensional coordinates of the target in the calibration coordinate system through the position detection module M500. Since the position detection of the target has been converted into the position detection in the viewing angle perpendicular to the specified plane, that is, the position detection in the horizontal view relative to the specified plane, when detecting the position of the target in the specified plane or the height of the target relative to the specified target, the deviation of the position judgment will not be caused by the viewing angle offset, thereby improving the accuracy of the target position detection.

[0126] In the target position detection system of the present application, the functions of each module can be implemented by using the specific implementation methods of each step in the above-mentioned target position detection method, which will not be described in detail here.

[0127] An embodiment of the present application also provides a target position detection device, comprising a processor; a memory, in which executable instructions of the processor are stored; wherein the processor is configured to perform the steps of the target position detection method by executing the executable instructions.

[0128] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0129] Refer to the following Figure 6 The electronic device 600 according to this embodiment of the present application is described. Figure 6 The electronic device 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0130] like Figure 6 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0131] The storage unit stores a program code, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps of various exemplary embodiments of the present application described in the above target position detection method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0132] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0133] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0134] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0135] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0136] By adopting the target position detection device provided in the present application, the processor executes the target position detection method when executing the executable instructions, thereby obtaining the beneficial effects of the above target position detection method.

[0137] The embodiment of the present application also provides a computer-readable storage medium for storing a program, and when the program is executed by a processor, the steps of the target position detection method are implemented. In some possible implementations, various aspects of the present application can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps of various exemplary implementations of the present application described in the above target position detection method section of this specification.

[0138] refer to Figure 7 As shown, a program product 800 for implementing the above method according to an embodiment of the present application is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0139] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0140] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0141] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or cluster. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0142] By adopting the computer-readable storage medium provided in the present application, the program stored therein implements the steps of the target position detection method when being executed, thereby obtaining the beneficial effects of the above-mentioned target position detection method.

[0143] The above content is a further detailed description of the present application in combination with specific preferred implementation methods, and it cannot be determined that the specific implementation of the present application is limited to these descriptions. For ordinary technicians in the technical field to which the present application belongs, several simple deductions or substitutions can be made without departing from the concept of the present application, which should be deemed to fall within the scope of protection of the present application.

Claims

1. A target position detection method, characterized in that: The steps include: Obtaining a conversion relationship between a camera coordinate system of a monocular camera and a calibration plate coordinate system, wherein the calibration plate coordinate system is a three-dimensional coordinate system established based on a calibration plate placed on a specified plane; Get the detection image taken by the monocular camera; Performing depth estimation and target recognition on the detection image to obtain the three-dimensional coordinates of the target in the camera coordinate system; Based on the conversion relationship between the camera coordinate system and the calibration plate coordinate system, converting the three-dimensional coordinates of the target in the camera coordinate system into the three-dimensional coordinates of the target in the calibration plate coordinate system; The target position detection is performed according to the three-dimensional coordinates of the target in the calibration plate coordinate system, and the target position detection includes detecting the position of the target in the specified plane and / or detecting the height relative to the specified plane.

2. The target position detection method according to claim 1, characterized in that: The step of obtaining the conversion relationship between the camera coordinate system of the monocular camera and the coordinate system of the calibration plate comprises the following steps: Obtain the actual three-dimensional coordinates of each calibration plate feature point in the calibration plate placed on the specified plane in the calibration plate coordinate system; Acquire a calibration plate image captured by a camera, and determine the pixel coordinates of each calibration plate feature point in the calibration plate image; According to the actual three-dimensional coordinates and pixel coordinates of each calibration plate feature point in the calibration plate coordinate system, an external parameter calibration algorithm is used to obtain a rotation matrix from the camera coordinate system of the monocular camera to the calibration plate coordinate system; A translation vector from the camera coordinate system to the calibration plate coordinate system is calculated according to the calibration plate image and the rotation matrix.

3. The target position detection method according to claim 2, characterized in that: Calculating the translation vector from the camera coordinate system to the calibration plate coordinate system includes the following steps: Using a monocular depth estimation model to perform depth estimation on the calibration plate image to obtain depth information of each pixel in the calibration plate image; According to the depth information and the internal parameters of the monocular camera, the detection three-dimensional coordinates of each pixel point in the calibration plate image in the camera coordinate system are obtained; The translation vector from the camera coordinate system to the calibration plate coordinate system is calculated according to the detected three-dimensional coordinates, the actual three-dimensional coordinates and the rotation matrix of a reference feature point among the calibration plate feature points.

4. The target position detection method according to claim 2, characterized in that: After calculating the translation vector from the camera coordinate system to the calibration plate coordinate system, the following steps are also included: Calculate the verification three-dimensional coordinates of the feature point of the calibration plate in the calibration plate coordinate system according to the three-dimensional coordinates of the feature point of the calibration plate in the camera coordinate system, the rotation matrix and the translation vector; Verify whether the rotation matrix and the translation vector meet the calibration requirements according to the comparison result between the verified three-dimensional coordinates and the actual three-dimensional coordinates of the feature points of the calibration plate; If not, repeat the step of obtaining the conversion relationship between the coordinate system of the monocular camera and the coordinate system of the calibration plate.

5. The target position detection method according to claim 1, characterized in that: Performing depth estimation and target recognition on the detection image to obtain the three-dimensional coordinates of the target in the camera coordinate system includes the following steps: Performing depth estimation and coordinate recognition on the detection image to obtain point cloud information of the detection image, wherein the point cloud information includes the three-dimensional coordinates of each pixel point in the detection image in the camera coordinate system; Performing target recognition on the detection image to determine the pixel position of the target in the detection image; The three-dimensional coordinates of the target in the camera coordinate system are obtained according to the pixel position of the target in the detection image.

6. The target position detection method according to claim 5, characterized in that: Performing depth estimation and coordinate recognition on the detection image to obtain point cloud information of the detection image includes the following steps: Using a monocular depth estimation model to perform depth estimation on the detection image to obtain depth information of each pixel in the detection image; According to the intrinsic parameters of the camera and the depth information of each pixel in the detection image, the three-dimensional coordinates of each pixel in the detection image in the camera coordinate system are obtained as the point cloud information.

7. The target position detection method according to claim 1, characterized in that: The designated plane is a horizontal plane, and the three-dimensional coordinates of the target in the calibration plate coordinate system include a horizontal component and a vertical component; performing target position detection according to the three-dimensional coordinates of the target in the calibration plate coordinate system includes the following steps: Determining whether the target exceeds a preset height range according to the vertical component of the three-dimensional coordinates of the target in the calibration plate coordinate system; It is determined whether the target exceeds a preset horizontal position range according to the horizontal component of the three-dimensional coordinates of the target in the calibration plate coordinate system.

8. A target position detection system, characterized in that: The target position detection method applied to any one of claims 1 to 7, wherein the system comprises: A conversion relationship acquisition module, used to acquire the conversion relationship between the camera coordinate system of the monocular camera and the calibration plate coordinate system, wherein the calibration plate coordinate system is a three-dimensional coordinate system established based on a calibration plate placed on a specified plane; An image acquisition module is used to obtain the detection image taken by the monocular camera; A target recognition module is used to perform target recognition on the detection image and obtain the three-dimensional coordinates of the target in the camera coordinate system; A coordinate conversion module, used for converting the three-dimensional coordinates of the target in the camera coordinate system into the three-dimensional coordinates of the target in the calibration plate coordinate system based on the conversion relationship between the camera coordinate system and the calibration plate coordinate system; The position detection module is used to detect the target position according to the three-dimensional coordinates of the target in the calibration plate coordinate system, wherein the target position detection includes the position detection of the target in the specified plane and / or the height detection relative to the specified plane.

9. A target position detection device, characterized in that: include: processor; a memory storing executable instructions of the processor; The processor is configured to execute the steps of the target position detection method according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the target position detection method according to any one of claims 1 to 7 are implemented.

Images