Plant station personnel positioning method and system based on visual identification

By installing video cameras in the factory site and using algorithms to identify and coordinate conversion, the problems of personnel positioning and inspection trajectory review in unmanned production environments are solved, and the accurate identification and positioning of inspection personnel and accidental intruders are achieved. , improve the quantifiability and safety of inspection work.

CN119068056AInactive Publication Date: 2024-12-03WANGSHUZHI TECH (HANGZHOU) CO LTD

Patent Information

Application Number
CN202411554321.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-04
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In a production environment without a man or with few people, such as water plants, power plants, and gas stations, it is difficult to accurately conduct personnel positioning and inspection trajectory review. The existing technology has problems such as equipment dependence, negligence in manual monitoring and inaccurate indoor positioning.

Method used

By installing a video camera in the factory site, using algorithms to identify and coordinate conversion, combining camera parameters and factory site coordinate system, personnel positioning and trajectory tracking are realized, and Kalman filtering algorithm is used to improve positioning accuracy.

Benefits of technology

Accurate identification and positioning of inspection personnel and accidental intruders is achieved, negligence of manual surveillance is avoided, the quantifiability and safety of inspection work is improved, and the dependence on the strength of GIS signals is reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068056A_ABST
    Figure CN119068056A_ABST
Patent Text Reader

Abstract

The invention discloses a station personnel positioning method and system based on visual identification, and belongs to the technical field of positioning. The method comprises the following steps: acquiring camera parameters; collecting an image through a camera; performing personnel identification on the image to obtain a positioning frame coordinate; obtaining personnel image information according to the positioning frame coordinates; comparing the personnel image information with a visual database to obtain the identity of a worker; obtaining personnel positioning point coordinates according to the camera parameters, the positioning frame coordinates and the plant station coordinate system conversion matrix; and according to the plurality of personnel positioning point coordinates, obtaining a personnel movement track. According to the invention, based on visual identification, personnel positioning and track tracking are realized without being influenced by indoor and outdoor GIS signal intensity; a plant station coordinate system is constructed, personnel positioning and trajectory tracking are performed on working personnel in a plant station, and quantification of inspection work is realized; when people enter a dangerous area in a plant station, identification, positioning and tracking are automatically carried out, and danger of people is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of positioning technology, and in particular to a plant personnel positioning method and system based on visual recognition. Background Art

[0002] For production environments with few or no people, such as water plants, power stations, gas stations, and industrial parks, regular inspections are required. However, it is difficult to accurately review whether the inspection personnel are in place and whether the inspection scope is comprehensive. The difficulty lies in the inability to accurately locate personnel. The existing technical solution is to install NFC or QR code check-in points at key points in the plant and station. After the personnel inspect the key points, they use mobile devices combined with corresponding applications to check in to achieve personnel positioning; for outdoor inspections of the plant and station, GIS positioning technology can be used to locate personnel; multiple cameras are generally installed in the plant and station for video surveillance, but on-duty personnel are required to conduct manual monitoring and judgment. At the same time, for the accidental intrusion of external personnel, the current situation generally relies on on-duty personnel to identify and locate through manual monitoring.

[0003] The method of setting up punch-in points can only ensure that personnel pass through key points, but cannot clearly determine whether key areas are patrolled, and the personnel trajectory is not accurate enough; the GIS positioning method can achieve relatively accurate personnel positioning outdoors, but it is difficult to accurately locate indoors or even underground; video surveillance requires manual identification and judgment, and there are situations such as personnel slacking off and negligence, which results in untimely handling of accidents. Summary of the invention

[0004] The purpose of the present invention is to provide a plant and station personnel positioning method based on visual recognition. The present invention uses a number of video cameras installed in the plant and station, and through the video content collected by them, the algorithm is used to identify personnel, including patrol personnel and accidental intruders, and then combined with the internal parameters and installation parameters of the camera, the coordinates are converted through the algorithm to determine the coordinate position of the personnel, so as to realize the positioning of the personnel in the plant and station and the tracking of the trajectory. It realizes the effective review of patrol personnel and the rapid identification and positioning of accidental intruders.

[0005] In order to solve the above technical problems, the present invention provides a plant personnel positioning method based on visual recognition, which is characterized by comprising the following steps: Get camera parameters; Collect images through a camera; Perform person recognition on the image and obtain the positioning frame coordinates; According to the positioning frame coordinates, the person image information is obtained; Compare the personnel image information with the visual database to obtain the staff identity; According to the camera parameters, positioning frame coordinates and plant station coordinate system conversion matrix, the coordinates of the personnel positioning point are obtained; According to the coordinates of multiple personnel positioning points, the personnel movement trajectory is obtained.

[0006] Preferably, obtaining camera parameters specifically includes the following steps: Obtain several standard calibration plate images taken by the camera; According to the standard calibration plate image, the intrinsic and extrinsic parameters of the camera are obtained; Use a laser rangefinder or a tape measure to measure and obtain the installation height H; Use the RTK device to mark points in the CGCS2000 coordinate system to obtain the camera coordinates (x0, y0); Find a point on the ground corresponding to the vertical midline of the camera surveillance video as the calibration point, and use the RTK device to obtain the corresponding coordinates (x1, y1). θ represents the angle between the camera installation direction and the true north direction. As the horizontal angle, the clockwise direction is positive, and the horizontal angle in the first quadrant is ; The distance from the calibration point to the camera projection point is obtained through the calibration point coordinates. , with the camera's positive projection point as the origin and the optical axis's ground projection direction as the Y c Axis direction, with the ground and Y c The vertical direction of the axis is X c The camera coordinate system is established by the axis, and the camera pitch angle α=γ-β, where , so the coordinates of the calibration point on the image are (w / 2,v0), then , thus the camera pitch angle α can be obtained, where h is the image height pixel resolution, is the normalized focal length on the Y axis.

[0007] Preferably, obtaining the intrinsic and extrinsic parameters of the camera according to the standard calibration plate image specifically includes the following steps: According to the basic parameters of the calibration plate, the number of squares, the interval width and the image of the standard calibration plate, the Zhang Zhengyou calibration method in the opencv algorithm library is used to obtain the intrinsic and extrinsic parameters of the camera.

[0008] Preferably, the camera intrinsic parameters are: , camera external parameters: ; in: f x = f / d x 、f y = f / d y , respectively represent the normalized focal length on the X-axis and Y-axis, where f represents the focal length, d x d y Respectively represent the physical size of the pixel in the X-axis and Y-axis directions; u, v, represent the horizontal and vertical coordinates of the center pixel of the standard calibration plate image. The coordinates of the standard calibration plate image take the upper left corner as the origin, so u=w / 2, v=h / 2, where w and h are the width and height pixel resolutions of the standard calibration plate image; R represents the rotation matrix (3×3); T represents the offset vector (3×1).

[0009] Preferably, according to the camera parameters, the positioning frame coordinates and the plant station coordinate system conversion matrix, the coordinates of the personnel positioning point in the camera coordinate system are obtained, which specifically includes the following steps: The y coordinate of the personnel positioning point in the camera coordinate system , γ=α+β, tanβ can be obtained according to the image plane coordinate relationship , and then we can get , and then the x coordinate of the personnel positioning point in the camera coordinate system can be obtained ,in , so ; In summary, the coordinates (x, y) of the personnel positioning point in the camera coordinate system can be obtained; After obtaining the personnel coordinates in the camera coordinate system, the plant station coordinate system is established. According to the camera position coordinates (x0, y0) and its horizontal angle θ, the coordinate system transformation matrix can be obtained as follows: , and then the coordinate system is transformed to obtain the single camera personnel positioning coordinates (x w ,y w ) = .

[0010] Preferably, obtaining the movement trajectory of a person according to the coordinates of a plurality of person positioning points specifically includes the following steps: Extract image feature points and feature descriptors of each image through feature extraction algorithm; Use a brute force matching algorithm to match image feature points in different images pairwise and calculate the distance between their feature descriptors; According to the distance between the feature descriptors, the matching relationship between the feature points in different camera graphics is determined; According to the matching relationship, the homography matrix Homography is obtained; According to the homography matrix, the pixel recognition coordinates of different people in one image are transformed, and the transformation results are matched with the pixel recognition coordinates of different people in another image. The distance is calculated, and the shortest distance principle is used to determine whether the people identified in different images are the same person, and the coordinate group data of each person under the monitoring of different cameras is obtained; The coordinate group data of each person monitored by different cameras is processed by the Kalman filter algorithm to obtain the person's positioning coordinates; By calculating the personnel positioning results in different time frames, the personnel movement trajectory is obtained.

[0011] Preferably, the coordinate group data of each person under the monitoring of different cameras is processed by a Kalman filter algorithm to obtain the most accurate personnel positioning coordinates, which specifically includes the following steps: a) Equation of state: ; where x k is the system state vector, [x, y, vx, vy], x and y are position coordinates, vx and vy are relative velocity components; A is the state transfer matrix, B is the control input matrix, u k is the control input, w k is the process noise, assumed to be Gaussian noise, with a covariance matrix of Q; b) Observation equation:

[0012] where z k,i is the observation value of the i-th camera, H i is the corresponding observation matrix, v k,i is the observation noise, assumed to be Gaussian noise, and the covariance matrix is ​​R i ; c) Prediction step, update the system state and covariance matrix: Prediction status:

[0013] Prediction covariance:

[0014] d) Update step: Calculate the Kalman gain: Calculate the gain for each camera:

[0015] Combined gain of all cameras:

[0016] Where H and R are the weighted averages of the observation matrix and observation noise of all cameras, respectively; Update the state estimate:

[0017] Update the covariance matrix:

[0018] Finally, we get the accurate personnel location coordinates (X k ,Y k ); By calculating the personnel positioning results in different time frames, the personnel movement trajectory is obtained.

[0019] Preferably, the feature extraction algorithm is SIFT algorithm, SURF algorithm, FAST algorithm, ORB algorithm or neural network HomographyNet.

[0020] Preferably, performing person recognition on the image to obtain the positioning frame coordinates specifically includes the following steps: Through the personnel recognition algorithm and personnel recognition model, the image is recognized and the positioning frame coordinates of the person in the image are obtained (u l ,v l ,u r ,v r )。u l ,v l ,u r ,v r They represent the u / v pixel coordinates of the upper left corner and the lower right corner of the identification positioning frame respectively. The coordinates of the personnel positioning are on the ground plane. The positioning point is located at the intersection of the vertical center line of the positioning frame and the ground. The pixel coordinates of the personnel positioning are (u0, v0) = ((u l -u r ) / 2 ,v r ); The personnel recognition algorithm is a Yolo algorithm, an R-CNN algorithm, a Fast R-CNN algorithm, an SSD algorithm, a RetinaNet algorithm or an EfficientDet algorithm; The Yolo algorithm adopts the Yolov8 algorithm.

[0021] The present invention provides a plant personnel positioning system based on visual recognition, comprising: Acquisition module, used to obtain camera parameters; A collection module, used for collecting images through a camera; A positioning frame coordinate recognition module is used to perform person recognition on the image and obtain positioning frame coordinates; A personnel image module is used to obtain personnel image information according to the positioning frame coordinates; The identity recognition module is used to compare the personnel image information with the visual database to obtain the identity of the staff member; The personnel positioning module is used to obtain the coordinates of the personnel positioning point according to the camera parameters, the positioning frame coordinates and the plant station coordinate system conversion matrix; The personnel motion trajectory generation module is used to obtain the personnel motion trajectory according to the coordinates of multiple personnel positioning points.

[0022] Compared with the prior art, the present invention has the following beneficial effects: 1. Use existing installed cameras without adding additional equipment or devices; 2. Based on visual recognition, personnel positioning and trajectory tracking are achieved, which is not affected by the strength of indoor and outdoor GIS signals; 3. Construct the plant station coordinate system to locate and track the personnel in the plant station, so as to make the inspection work quantifiable; 4. Automatically identify, locate and track personnel entering dangerous areas within the plant to avoid danger to personnel; 5. It also has the ability to identify and locate people who accidentally break into the plant and station, assisting in the rapid implementation of security work; BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0024] Figure 1 It is a flow chart of a method for locating plant personnel based on visual recognition according to the present invention; Figure 2 It is a flowchart of camera calibration and measurement; Figure 3 is a schematic diagram of the determination of the horizontal angle θ; Figure 4 Schematic diagram of measuring the pitch angle α of the camera; Figure 5 It is a schematic diagram of the process of coordinate positioning of monocular camera personnel; Figure 6 It is a schematic diagram of the corresponding relationship between the image plane coordinate system and the camera coordinate system; Figure 7 It is a schematic diagram of multi-camera coordinate filtering positioning; Figure 8 It is a module diagram of the system and device; Fig. 9 It is a schematic diagram of the distribution position of the calibration plate in the image; Fig.10 This is a schematic diagram of pixel coordinates of ground points where personnel are located in Example 1; Fig.11 This is a schematic diagram of an example of the trajectory of two persons in Example 1; Fig.12 is a schematic diagram of the correspondence between two images at different angles in Example 1; Fig.13 It is a schematic diagram of the personnel positioning points obtained by monitoring two cameras respectively, and the personnel trajectory line obtained after the Kalman filter algorithm. DETAILED DESCRIPTION

[0025] Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention, so the present invention is not limited to the specific implementation disclosed below.

[0026] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0027] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0028] The present invention is further described in detail below in conjunction with the accompanying drawings: The present invention provides a plant personnel positioning method based on visual recognition, comprising: 1) Overall technical process, such as Figure 1 As shown: In order to realize personnel coordinate positioning through video surveillance, the basic parameters of the camera must first be calibrated and measured, including the camera's internal and external parameters, installation height, pitch angle, installation coordinates, and horizontal angle.

[0029] For surveillance videos or screenshots of a single camera, the Yolov8 algorithm is used to identify people, determine the pixel coordinates of people on the image, and combine the measured camera parameters with the coordinate conversion algorithm to calculate the coordinates of people in the camera coordinate system. Then, the plant station coordinate system is established, and the coordinate system conversion is performed in combination with the camera's measured coordinates and horizontal angles. Finally, the coordinate position of people under a single camera in the plant station coordinate system is obtained. At the same time, the personnel identification results of the Yolov8 algorithm can be further matched with the registered personnel visual database to achieve the identification of specific staff and intruders.

[0030] Furthermore, for areas monitored by multiple cameras, based on the recognition and positioning results of each camera, multiple recognition results are integrated through image feature extraction and matching to obtain multi-coordinate positioning results of a single object. The multiple positioning results are filtered through the Kalman filtering algorithm to achieve precise positioning of personnel.

[0031] 2) Camera calibration and measurement, such as Figure 2 As shown; Camera calibration: Take multiple images of the standard calibration plate at different positions and angles within the camera's field of view, and use the standard Zhang Zhengyou calibration method in the OpenCV algorithm library to calibrate the camera and obtain the camera's intrinsic and extrinsic parameters.

[0032] Camera internal parameters: , camera external parameters: .

[0033] in: f x = f / d x 、f y = f / d y , respectively represent the normalized focal length on the X-axis and Y-axis, where f represents the focal length, d x d y Respectively represent the physical size of the pixel in the X-axis and Y-axis directions.

[0034] u, v represent the horizontal and vertical coordinates of the center pixel of the image. The image coordinates take the upper left corner as the origin, so u=w / 2,v=h / 2, where w and h are the width and height pixel resolutions of the image.

[0035] R, represents the rotation matrix (3×3).

[0036] T, represents the offset vector (3×1).

[0037] The camera installation parameter measurements include: installation height, coordinates, horizontal angle, and pitch angle.

[0038] The installation height H can be directly measured by a laser rangefinder or a tape measure.

[0039] The camera coordinates (x0, y0) can be obtained by using RTK equipment in the CGCS2000 coordinate system.

[0040] The horizontal angle θ can be measured by a compass or other equipment. You can also use a camera to monitor the video, find a point on the vertical midline of the screen that corresponds to the ground position as a calibration point, and use an RTK device to mark the point to obtain the corresponding coordinates (x1, y1). θ represents the angle between the camera installation direction and the true north direction. If the clockwise direction is positive, the direction in the first quadrant is , the solution for other quadrants is similar, but attention should be paid to period conversion, such as Figure 3 shown.

[0041] The camera pitch angle α can be measured by instruments such as a slope meter. It can also be obtained by calculation. The distance from the calibration point to the camera projection point can be obtained through the coordinates of the calibration point. The camera's positive projection point is taken as the origin, and the ground projection direction of the optical axis is taken as the Y c Axis direction, with the ground and Y c The vertical direction of the axis is X c Axis establishes the camera coordinate system, such as Figure 4 As shown, the pitch angle α=γ-β, where β can be obtained through pixel coordinates. Since the calibration point is on the vertical bisector, the coordinates of the calibration point on the image are (w / 2, v0), then , thus the pitch angle α value can be obtained, where h is the image height pixel resolution, is the normalized focal length on the Y axis.

[0042] 3) Monocular camera personnel coordinate positioning, such as Figure 5 As shown; On the premise that the basic parameters of the camera have been calibrated and measured, the process of personnel positioning is as follows: First, obtain the video / screenshot captured by the camera, use the Yolov8 algorithm and the trained personnel recognition model to identify the person in the image, and obtain the positioning frame coordinates (u l ,v l ,u r ,v r ), u l ,v l ,u r ,v r They represent the u / v pixel coordinates of the upper left corner and lower right corner of the identification positioning frame respectively. The personnel positioning is the coordinate on the ground plane, so the positioning point is located at the intersection of the vertical center line of the positioning frame and the ground, and the personnel positioning pixel coordinates are (u0, v0) = ((u l -u r ) / 2 ,v r ).

[0043] Furthermore, the personnel image information is obtained according to the positioning frame coordinates, and compared with the information in the existing staff visual database to determine the specific identity of the staff.

[0044] like Figure 6 As shown in the figure, from the corresponding relationship between the image plane coordinate system and the camera coordinate system, it can be seen that the y coordinate of the personnel positioning point in the camera coordinate system is , γ=α+β, where the camera pitch angle α has been measured, and tanβ can be obtained according to the image plane coordinate relationship , and then we can get Then we can get the x coordinate of the personnel positioning point in the camera coordinate system ,in , so In summary, the coordinates (x, y) of the personnel positioning point in the camera coordinate system can be obtained.

[0045] After obtaining the personnel coordinates in the camera coordinate system, the plant coordinate system needs to be established. The CGCS2000 standard coordinate system is generally used, and a local coordinate system can also be established as needed. According to the camera position coordinates (x0, y0) and its horizontal angle θ, the coordinate system conversion matrix can be obtained as follows: , and then the coordinate system is converted to obtain the single camera personnel positioning coordinates (x w ,y w ) = .

[0046] 4) Multi-camera coordinate filtering positioning, such as Figure 7 As shown; For areas monitored by multiple cameras at multiple angles, based on the above-mentioned monocular camera personnel coordinate positioning method, each camera can obtain a set of personnel identification and personnel positioning coordinate results.

[0047] For each camera to collect images, the SIFT algorithm is used to extract the feature points and feature descriptors of each image. The image feature points in different graphics are matched in pairs using a brute force matching algorithm, and the distance between their feature descriptors is calculated. According to the distance, the matching relationship of the feature points in different camera graphics is determined, and the homography matrix Homography is obtained based on the matching relationship. Then, according to the homography matrix Homography, the pixel recognition coordinates of different people in one image are transformed, that is, the coordinate transformation is performed between the two images based on the same world coordinate system, and the transformation results are matched with the pixel recognition coordinates of different people in another image. The distance is calculated, and the person identified in different images is determined according to the principle of the shortest distance. That is, the images of the same person in different camera images are matched, and the coordinate group data of each person under the monitoring of different cameras can be obtained.

[0048] The monitoring results of different cameras have problems such as errors and noise. The Kalman filter algorithm is used to obtain more accurate position coordinates. The process of Kalman filter algorithm processing multi-camera coordinate group data is as follows: a) Equation of state:

[0049] where x kis the system state vector, [x, y, vx, vy], x and y are position coordinates, vx and vy are relative velocity components; A is the state transfer matrix, B is the control input matrix, u k is the control input, w k is the process noise, assumed to be Gaussian noise, with a covariance matrix of Q; b) Observation equation:

[0050] where z k,i is the observation value of the i-th camera, H i is the corresponding observation matrix, v k,i is the observation noise, assumed to be Gaussian noise, and the covariance matrix is ​​R i ; c) Prediction step, update the system state and covariance matrix: Prediction status:

[0051] Prediction covariance:

[0052] d) Update step: Calculate the Kalman gain: Calculate the gain for each camera:

[0053] Combined gain of all cameras:

[0054] Where H and R are the weighted averages of the observation matrix and observation noise of all cameras, respectively.

[0055] Update the state estimate:

[0056] Update the covariance matrix:

[0057] Finally, more accurate personnel positioning coordinates (X k ,Y k ). Furthermore, according to the above method, by calculating the personnel positioning results of different time frames, the movement trajectory of the personnel can be obtained.

[0058] 5) Systems and devices, such as Figure 8 As shown; In order to realize the above-mentioned personnel positioning method, a matching system and device are required.

[0059] According to application needs, the media center obtains collected image data from surveillance cameras and stores it in storage devices. The media center extracts image data from the storage device and transmits it to the inference platform for personnel positioning calculation. After the calculation is completed, the inference platform transmits the personnel positioning results back to the media center, which pushes them to the application system after integrating various types of data to realize various system applications, including: inspection personnel positioning, personnel intrusion positioning, and personnel tracking and protection in dangerous areas.

[0060] In order to better illustrate the technical effect of the present invention, the present invention provides the following specific embodiments to illustrate the above technical process: Embodiment 1: A method for locating plant personnel based on visual recognition, comprising the following steps: To calibrate the camera, use a standard calibration plate and take pictures at different positions and angles on the camera screen, usually no less than 15 pictures. The distribution of the calibration plate in the image can be referenced Fig. 9 , you can adjust the front and rear angles appropriately and take multiple photos at each position, try to ensure that the calibration board grid is horizontal or vertical with the third line of the picture, and the calibration board falls entirely within the picture. Fig. 9 As shown; According to the basic parameters of the calibration board, the number of squares, the interval width, and the calibration graphics taken above, the Zhang Zhengyou calibration method in the opencv algorithm library is used to obtain the intrinsic and extrinsic parameters of the camera.

[0061] Camera internal parameters: , camera external parameters: .

[0062] Use a tape measure or laser rangefinder to directly measure the camera installation height H.

[0063] Use RTK equipment to mark points in the CGCS2000 coordinate system to obtain the coordinates of the camera's orthographic projection point (x0, y0). You can also establish a coordinate system at the factory station and use a level, theodolite, tape measure and other equipment to measure the coordinates of the camera's orthographic projection point.

[0064] The horizontal angle of the camera, θ, can be measured by a compass or other equipment. You can also use the camera to monitor the screen, find a point on the vertical midline of the screen that corresponds to the ground position as a calibration point, and use the RTK equipment to mark the point to obtain the corresponding coordinates (x1, y1). θ represents the angle between the camera installation direction and the true north direction, with the clockwise direction being positive, then .

[0065] The camera pitch angle α can be measured by equipment such as a horizontal slope meter. It can also be obtained by calculation. The distance from the calibration point to the camera projection point can be obtained by the coordinates of the calibration point determined during the horizontal angle measurement. The camera coordinate system is established with the camera's orthographic projection point as the origin, the ground projection direction of the optical axis as the Yc axis, and the direction perpendicular to the ground and the Yc axis as the Xc axis. The pitch angle α=γ-β, where β can be obtained through pixel coordinates. Since the calibration point is on the vertical bisector, the coordinates of the calibration point on the image are (w / 2, v0), then , thus the pitch angle α value can be obtained.

[0066] Get the image captured by the camera, and use the Yolov8 algorithm and the trained personnel recognition model to identify the person in the image. Get the positioning frame coordinates of the person in the image (u l ,v l ,u r ,v r ), the pixel coordinates of the ground point where the person is located (u0, v0) = ( (u l -u r ) / 2, such as Fig.10 shown.

[0067] From the correspondence between the plane coordinate system and the camera coordinate system, we can know that the y coordinate of the personnel positioning point in the camera coordinate system is , γ=α+β, where α has been measured, and tanβ can be obtained according to the image plane coordinate relationship , and then we can get Then we can get the x coordinate of the personnel positioning point in the camera coordinate system ,in , so In summary, the coordinates (x, y) of the personnel positioning point in the camera coordinate system can be obtained.

[0068] According to the camera position coordinates (x0, y0) and its horizontal angle θ, the coordinate system transformation matrix can be obtained as follows: , and use this to convert the coordinate system. The positioning coordinates in the camera coordinate system are used to obtain the single camera personnel positioning coordinates in the factory station coordinate system (x w ,y w ) = .

[0069] By locating people in surveillance videos over a period of time, we can obtain the location trajectory of people, such as Fig.11 Shown, are examples of the trajectories of two persons; For areas monitored by multiple cameras, the results of multiple camera monitoring can be used to accurately locate personnel. Each camera collects images, and the SIFT algorithm is used to extract the feature points and feature descriptors of each image. The image feature points in different graphics are matched in pairs using a brute force matching algorithm, and the distance between their feature descriptors is calculated. According to the distance, the matching relationship of the feature points in different camera graphics is determined, and the homography matrix Homography is obtained based on the matching relationship. Then, according to the homography matrix Homography, the pixel recognition coordinates of different personnel in one image are transformed, that is, the coordinate transformation is performed between the two images based on the same world coordinate system, and the transformation results are matched with the pixel recognition coordinates of different personnel in another image. The distance is calculated, and the shortest distance principle is used to determine whether the personnel identified in different images are the same person, that is, the images of the same person in different camera images are matched, and the coordinate group data of each person under the monitoring of different cameras can be obtained.

[0070] like Fig.12 As shown, it is the correspondence between two images at different angles in the same scene.

[0071] The monitoring results of different cameras have problems such as errors and noise. The Kalman filter algorithm is used to obtain more accurate position coordinates. The process of Kalman filter algorithm processing multi-camera coordinate group data is as follows: Equation of state: ; Where xk is the system state vector, [x,y,vx,vy], x and y are the position coordinates, vx and vy are the relative velocity components; A is the state transfer matrix, B is the control input matrix, uk is the control input, wk is the process noise, assumed to be Gaussian noise, and the covariance matrix is ​​Q.

[0072] Observation equation: ; Where zk,i is the observation value of the i-th camera, Hi is the corresponding observation matrix, vk,i is the observation noise, which is assumed to be Gaussian noise, and the covariance matrix is ​​Ri; Prediction step, update the system state and covariance matrix: a) Equation of state:

[0073] b) Observation equation:

[0074] where z k,i is the observation value of the i-th camera, H i is the corresponding observation matrix, v k,i is the observation noise, assumed to be Gaussian noise, and the covariance matrix is ​​R i ; c) Prediction step, update the system state and covariance matrix: Prediction status:

[0075] Prediction covariance:

[0076] d) Update step: Calculate the Kalman gain: Calculate the gain for each camera:

[0077] Combined gain of all cameras:

[0078] Where H and R are the weighted averages of the observation matrix and observation noise of all cameras, respectively.

[0079] Update the state estimate:

[0080] Update the covariance matrix:

[0081] Finally, more accurate personnel positioning coordinates (X k ,Y k ). Furthermore, according to the above method, by calculating the personnel positioning results of different time frames, the movement trajectory of the personnel can be obtained.

[0082] Furthermore, according to the above method, by calculating the personnel positioning results in different time frames, the movement trajectory of the personnel can be obtained. Fig.13 As shown in the figure, the personnel positioning points obtained by monitoring two cameras respectively, and the personnel trajectory lines obtained after the Kalman filter algorithm.

[0083] In addition to the Yolov8 algorithm mentioned above, other Yolo algorithms and R-CNN, Fast R-CNN, SSD, RetinaNet, EfficientDet and other algorithms can also be used for personnel recognition algorithms.

[0084] In the process of person image matching, in addition to the SIFT algorithm, SURF, FAST, ORB and neural network HomographyNet can also be used to extract image features.

[0085] In addition to using the Kalman filter algorithm to refine personnel coordinates, other methods such as weighted averaging, Bayesian weighting, and particle filtering can also be used.

[0086] In addition to using the standard CGCS2000 to establish the plant station coordinate system, other coordinate systems can also be used according to actual needs, or a separate coordinate system can be established based on the plant station range.

[0087] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules, modules or units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units, modules or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0088] The units may or may not be physically separated, and the components displayed as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0089] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0090] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by the central processing unit (CPU), the above-mentioned functions defined in the method of the present invention are executed. It should be noted that the above-mentioned computer-readable medium of the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of an electrical, magnetic, optical, electromagnetic, infrared segment, or semiconductor, or any combination of the above.

[0091] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0092] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A plant personnel positioning method based on visual recognition, characterized in that: The following steps are involved: Get camera parameters; Collect images through a camera; Perform person recognition on the image and obtain the positioning frame coordinates; According to the camera parameters, positioning frame coordinates and plant station coordinate system conversion matrix, the coordinates of the personnel positioning point are obtained; According to the coordinates of multiple personnel positioning points, the personnel movement trajectory is obtained.

2. The plant personnel positioning method based on visual recognition according to claim 1 is characterized in that: Get the camera parameters, including the following steps: Obtain several standard calibration plate images taken by the camera; According to the standard calibration plate image, the intrinsic and extrinsic parameters of the camera are obtained; Use a laser rangefinder or a tape measure to measure and obtain the installation height H; Use the RTK device to mark points in the CGCS2000 coordinate system to obtain the camera coordinates (x0, y0); Find a point on the ground corresponding to the vertical midline of the camera surveillance video as the calibration point, and use the RTK device to obtain the corresponding coordinates (x1, y1). θ represents the angle between the camera installation direction and the true north direction. As the horizontal angle, the clockwise direction is positive, and the horizontal angle in the first quadrant is ; The distance from the calibration point to the camera projection point is obtained through the calibration point coordinates. , with the camera's positive projection point as the origin and the optical axis's ground projection direction as the Y c Axis direction, with the ground and Y c The vertical direction of the axis is X c The camera coordinate system is established by the axis, and the camera pitch angle α=γ-β, where , so the coordinates of the calibration point on the image are (w / 2,v0), then , thus the camera pitch angle α can be obtained, where h is the image height pixel resolution, is the normalized focal length on the Y axis.

3. The plant personnel positioning method based on visual recognition according to claim 2 is characterized in that: According to the standard calibration plate image, the intrinsic and extrinsic parameters of the camera are obtained, which specifically includes the following steps: According to the basic parameters of the calibration plate, the number of squares, the interval width and the image of the standard calibration plate, the Zhang Zhengyou calibration method in the opencv algorithm library is used to obtain the intrinsic and extrinsic parameters of the camera.

4. The plant personnel positioning method based on visual recognition according to claim 3 is characterized in that: Camera internal parameters: , camera external parameters: ; in: f x = f / d x 、f y = f / d y , respectively represent the normalized focal length on the X-axis and Y-axis, where f represents the focal length, d x d y Respectively represent the physical size of the pixel in the X-axis and Y-axis directions; u, v, represent the horizontal and vertical coordinates of the center pixel of the standard calibration plate image. The coordinates of the standard calibration plate image take the upper left corner as the origin, so u=w / 2, v=h / 2, where w and h are the width and height pixel resolutions of the standard calibration plate image; R represents the rotation matrix (3×3); T represents the offset vector (3×1).

5. The plant personnel positioning method based on visual recognition according to claim 4 is characterized in that: According to the camera parameters, the positioning frame coordinates and the plant station coordinate system conversion matrix, the coordinates of the personnel positioning point in the camera coordinate system are obtained, which specifically includes the following steps: The y coordinate of the personnel positioning point in the camera coordinate system , γ=α+β, tanβ can be obtained according to the image plane coordinate relationship , and then we can get , and then the x coordinate of the personnel positioning point in the camera coordinate system can be obtained ,in , so ; In summary, the coordinates (x, y) of the personnel positioning point in the camera coordinate system can be obtained; After obtaining the personnel coordinates in the camera coordinate system, the plant station coordinate system is established. According to the camera position coordinates (x0, y0) and its horizontal angle θ, the coordinate system transformation matrix can be obtained as follows: , and then the coordinate system is transformed to obtain the single camera personnel positioning coordinates (x w ,y w ) = .

6. The plant personnel positioning method based on visual recognition according to claim 5 is characterized in that: According to the coordinates of multiple personnel positioning points, the movement trajectory of the personnel is obtained, which specifically includes the following steps: Extract image feature points and feature descriptors of each image through feature extraction algorithm; Use a brute force matching algorithm to match image feature points in different images pairwise and calculate the distance between their feature descriptors; According to the distance between the feature descriptors, the matching relationship between the feature points in different camera graphics is determined; According to the matching relationship, the homography matrix Homography is obtained; According to the homography matrix, the pixel recognition coordinates of different people in one image are transformed, and the transformation results are matched with the pixel recognition coordinates of different people in another image. The distance is calculated, and the shortest distance principle is used to determine whether the people identified in different images are the same person, and the coordinate group data of each person under the monitoring of different cameras is obtained; The coordinate group data of each person monitored by different cameras is processed by the Kalman filter algorithm to obtain the person's positioning coordinates; By calculating the personnel positioning results in different time frames, the personnel movement trajectory is obtained.

7. The plant personnel positioning method based on visual recognition according to claim 6 is characterized in that: The coordinate group data of each person under different camera monitoring is processed by the Kalman filter algorithm to obtain the most accurate person positioning coordinates, which specifically includes the following steps: a) Equation of state: ; where x k is the system state vector, [x, y, vx, vy], x and y are position coordinates, vx and vy are relative velocity components; A is the state transfer matrix, B is the control input matrix, u k is the control input, w k is the process noise, assumed to be Gaussian noise, with a covariance matrix of Q; b) Observation equation: ; where z k,i is the observation value of the i-th camera, H i is the corresponding observation matrix, v k,i is the observation noise, assumed to be Gaussian noise, and the covariance matrix is ​​R i ; c) Prediction step, update the system state and covariance matrix: Prediction status: ; Prediction covariance: ; d) Update step: Calculate the Kalman gain: Calculate the gain for each camera: ; Combined gain of all cameras: ; Where H and R are the weighted averages of the observation matrix and observation noise of all cameras, respectively; Update the state estimate: ; Update the covariance matrix: ; Finally, we get the accurate personnel location coordinates (X k ,Y k ); By calculating the personnel positioning results in different time frames, the personnel movement trajectory is obtained.

8. The plant personnel positioning method based on visual recognition according to claim 7 is characterized in that: The following steps are also included: According to the positioning frame coordinates, the person image information is obtained; Compare the personnel image information with the visual database to obtain the staff identity.

9. The plant personnel positioning method based on visual recognition according to claim 8 is characterized in that: Perform person recognition on the image and obtain the positioning frame coordinates, which specifically includes the following steps: Through the personnel recognition algorithm and personnel recognition model, the image is recognized and the positioning frame coordinates of the person in the image are obtained (u l ,v l ,u r ,v r )。u l ,v l ,u r ,v r They represent the u / v pixel coordinates of the upper left corner and the lower right corner of the identification positioning frame respectively. The coordinates of the personnel positioning are on the ground plane. The positioning point is located at the intersection of the vertical center line of the positioning frame and the ground. The pixel coordinates of the personnel positioning are (u0, v0) = ((u l -u r ) / 2 ,v r ); The personnel recognition algorithm is a Yolo algorithm, an R-CNN algorithm, a Fast R-CNN algorithm, an SSD algorithm, a RetinaNet algorithm or an EfficientDet algorithm; The Yolo algorithm adopts the Yolov8 algorithm.

10. A plant personnel positioning system based on visual recognition, used to implement the plant personnel positioning method based on visual recognition as claimed in any one of claims 1 to 9, characterized in that: include: Acquisition module, used to obtain camera parameters; A collection module, used for collecting images through a camera; A positioning frame coordinate recognition module is used to perform person recognition on the image and obtain positioning frame coordinates; A personnel image module is used to obtain personnel image information according to the positioning frame coordinates; The identity recognition module is used to compare the personnel image information with the visual database to obtain the identity of the staff member; The personnel positioning module is used to obtain the coordinates of the personnel positioning point according to the camera parameters, the positioning frame coordinates and the plant station coordinate system conversion matrix; The personnel motion trajectory generation module is used to obtain the personnel motion trajectory according to the coordinates of multiple personnel positioning points.

Citation Information

Patent Citations

  • Agricultural machines navigation hierarchical positioning process and system

    CN101285686A

  • Intelligent video monitoring method and system thereof

    CN101635835A

  • Target detection recognition algorithm based on 3D (Three-dimensional) trajectory analysis

    CN106127137A

  • A multi-camera data fusion method based on a spatial coordinate system

    CN109190508A

  • Method for dynamically tracking and positioning indoor personnel in large-scale place

    CN111080679A

Cited By

  • Method and system for quickly searching appointed personnel and tracking and shooting based on positioning

    CN119342349A

  • Passive vision positioning method and device based on multi-constraint measurement-free calibration

    CN120194711A

  • Production park personnel positioning method and system

    CN120321587A

  • A production park personnel positioning method and system

    CN120321587B

  • System, method for real-time color coding of object in video image, and machine-readable medium

    RU2859858C1