Short-range low-altitude photoelectric blind compensation monitoring method, system, equipment and terminal

Abstract

The invention belongs to but not limited to the technical field of short-range low-altitude monitoring management, and particularly relates to a short-range low-altitude photoelectric blind compensation monitoring method. According to the method, the system precision is improved in the monitoring process by calibrating the photoelectric equipment, including calibration of parameters, positions, optical axis azimuth angles and pitch angles, and sensing the posture change of the variable-focus equipment in real time. In the image information extraction process, a marine target can be automatically identified and positioned, the three-dimensional coordinates of the target are calculated through coordinate conversion and joint solution, and then the target is accurately positioned and tracked by using variable-focus equipment. In addition, the system can autonomously allocate monitoring tasks of a plurality of targets, and parameter correction and system state monitoring are carried out in combination with user interaction. The historical video data management function further improves the efficiency and accuracy of blind compensation monitoring, and is suitable for accurate photoelectric monitoring in a multi-target dynamic environment.

Description

Technical Field

[0001] The present invention belongs to but is not limited to the technical field of short-range and low-altitude monitoring management, and in particular relates to a short-range and low-altitude photoelectric blind spot monitoring method, system, equipment and terminal. Background Art

[0002] Comprehensive and accurate perception of the surrounding environment is the premise for maintaining one's own safety, preventing enemy infiltration and accidental collision. With the rapid development and application of drone technology, the demand for detection and defense of low-altitude small drones has become increasingly strong. Only by accurately obtaining the location of the drone can such low, slow and small targets be controlled. The current means of monitoring the surrounding air environment mainly include photoelectric detection technology, radar detection technology and sound cross-positioning technology, which respectively collect and extract information from optical, electromagnetic and sound signals to monitor aerial targets around the platform. Among them, the photoelectric detection technology monitors the surrounding environment passively by receiving environmental information and does not actively transmit signals, which can avoid electromagnetic interference and expose its own position. In addition, photoelectric information conforms to the identification habits of personnel, and can better meet the needs of distinguishing the nature of target types. Moreover, its deployment requirements are relatively low, and it can fill in the blind spots of short-range low-altitude areas.

[0003] When the existing photoelectric detection system and video surveillance system are directly used to monitor low-altitude small targets, the following problems are faced: first, they can only distinguish the type of target, lack depth information, and cannot obtain the accurate position of the target; second, due to the movement of the platform or the target, the position of the target in the image changes, making it difficult to track and distinguish stably; third, the viewing angle of a single photoelectric device is limited. The longer the focal length, the clearer the target, but the smaller the viewing angle, the smaller the observed range. Therefore, in order to meet the monitoring needs of the platform's surrounding environment, it is urgent to design a monitoring system that can use photoelectric equipment to identify, locate and track short-range low-altitude targets around the platform.

[0004] In view of the above analysis, the technical problems that need to be solved urgently in the prior art are:

[0005] (1) When a single photoelectric detection system senses the surrounding environment, it can only distinguish the type of target but cannot obtain the target's position due to the lack of depth information. When using other equipment to achieve distance detection, it is usually necessary to actively emit signals, which will expose the position. The distance information obtained also needs to be aligned with the target's position, which is a complicated process and difficult to operate.

[0006] (2) When the existing monitoring system on land is directly used to monitor low-altitude targets, the position of the target in the image changes due to the movement of the platform or the target, making it difficult to track and identify the target stably.

[0007] (3) The viewing angle of a single optoelectronic device is limited. The longer the focal length, the clearer the target, but the smaller the viewing angle, the smaller the observed range. It is difficult to achieve full-angle coverage of the sea area around the platform, and monitoring loopholes are prone to occur.

[0008] (4) Radar equipment can locate the target, but its electromagnetic radiation is strong, which will affect other electromagnetic equipment and cause certain harm to the human body, and it cannot determine the attributes of the target. Summary of the invention

[0009] In response to the problems existing in the prior art, the present invention provides a short-range, low-altitude optoelectronic blind spot monitoring method, system, equipment and terminal. The system includes 3 or more fixed optoelectronic devices and 3 variable-focus optoelectronic devices. The fixed optoelectronic devices have a large viewing angle and are responsible for full-time monitoring of the overhead airspace. The variable-focus optoelectronic devices are responsible for tracking and identifying targets discovered by the fixed optoelectronic devices. When the system is built, the internal parameters of each optoelectronic device are calibrated in advance using the calibration board through the optoelectronic device calibration module; after selecting a reference point, the azimuth, distance and height of each optoelectronic device relative to the reference point are calibrated. For fixed optoelectronic devices, the azimuth and pitch angles of the optical axis should also be calibrated, and for variable-focus optoelectronic devices, the reference azimuth and pitch angles of the optical axis need to be calibrated. During use, the attitude perception module is used to perceive the changes in the focal length, optical axis pitch angle and optical axis azimuth of the variable-focus optoelectronic device in real time. If the position of the optoelectronic device changes relative to the reference point, the position of the optoelectronic device relative to the reference point is perceived in real time; the image information extraction module is used to detect the target in each image and distinguish its category and relative position; the data processing module is used to perform continuous frame fusion, target position solution and target motion trajectory tracking; the user interaction module is used to input parameters, monitor and control, display monitoring information, pay attention to key targets and warn high-threat targets; the data management module is used to save information, query historical information and import external information.

[0010] The present invention is achieved by a short-range low-altitude photoelectric blind spot monitoring method, comprising:

[0011] S1, pre-preparation, optoelectronic equipment calibration: before the equipment is deployed, the internal parameters, position, height, optical axis azimuth, and optical axis pitch angle of the optoelectronic equipment are calibrated one by one using the calibration board; the relevant parameters of each optoelectronic equipment are saved and managed through the data management module; the internal parameters of each optoelectronic equipment are calibrated using the calibration board through the optoelectronic equipment calibration module, and after selecting a reference point, the azimuth, distance, and height of each optoelectronic equipment relative to the reference point are calibrated. For fixed optoelectronic equipment, the azimuth and pitch angle of the optical axis need to be calibrated, and for variable-focus optoelectronic equipment, the optical axis reference azimuth and pitch angle need to be calibrated. During use, the attitude sensing module is used to sense the changes in the focal length, optical axis pitch angle, and optical axis azimuth of the variable-focus optoelectronic equipment in real time. If the position of the optoelectronic equipment changes relative to the reference point, the position of the optoelectronic equipment relative to the reference point is sensed in real time;

[0012] S2, image information extraction, identifies the aerial targets of interest in the image sequence data, detects their categories and locates their positions in the image;

[0013] S3, coordinate transformation: According to the imaging principle, the position of the target is transformed into the orientation relative to the optoelectronic device.

[0014] S4, joint solution: using the orientation information of the target relative to two or more fixed optoelectronic devices and the relative positions of the optoelectronic devices, solve the three-coordinate position of the target relative to the reference point and convert it into an approximate position.

[0015] S5, improving positioning accuracy: using the position of the target obtained in step 3, guiding the variable-focus optoelectronic device to find the target, and re-solving the precise position of the target to determine the target attributes.

[0016] S6, autonomous target allocation: For multiple targets in the air at the same time, variable-focus optoelectronic devices are automatically allocated to locate them separately.

[0017] S7, target tracking: Use the target tracking algorithm to track the position of the target and drive the servo of the variable-focus optoelectronic device to ensure continuous control of the target.

[0018] S8, user interaction: manually adjust parameters, correct accuracy, control system, display tracking status, focus on high-threat targets, and feedback the working status of the blind spot monitoring system;

[0019] S9, Data Management: Save surveillance video information, query, replay, delete or import historical data from other devices.

[0020] Further, S1 specifically includes: in the preliminary preparation work, mainly calibrating the internal and external parameters of the optoelectronic equipment;

[0021] The internal parameter matrix includes the focal length f of the optoelectronic device, the width d of the pixel unit x , the height of the pixel unity , and the width w0 and height h0 of the CCD photosensitive element. The distortion mainly includes radial distortion (k1, k2, k3) and tangential distortion (p1, p2). The intrinsic parameter matrix and distortion matrix of the optoelectronic device are calibrated by Zhang Zhengyou calibration method;

[0022] The external parameters of the optoelectronic device mainly include the position of the optoelectronic device relative to the reference point (X0, Y0, Z0) and the angle of the optical axis of the optoelectronic device. (The variable-focus optoelectronic device is the reference angle). The center of the platform (set by yourself) is used as the origin of the horizontal coordinate system, the north direction is the Y axis, the east direction is the X axis, and the vertical horizontal plane is the Z axis. (X0, Y0) represents the position of the optoelectronic device relative to the reference point, and Z0 represents the height of the optoelectronic device above sea level. It indicates the azimuth of the optical axis of the optoelectronic device in the horizontal direction, that is, the angle between the optical axis of the optoelectronic device and the Y-axis. The clockwise direction is positive and the value range is 90°~90°. θ0 indicates the pitch angle of the optical axis of the optoelectronic device, that is, the angle between the optical axis of the optoelectronic device and the horizontal plane. The upward direction is positive and the value range is 0°~90°.

[0023] S2 specifically includes: using image processing technology to extract information about low-altitude drones and other targets of interest from optoelectronic equipment images. Using a target detection network model to train the neural network using massive data, automatically fitting target recognition features, and realizing target detection and recognition.

[0024] Collect low-altitude video images through the Internet or on-site, annotate and name the targets of interest, and convert them into data sets for neural network training. The network weights are obtained through multiple batches of training, and are ultimately used to detect targets in the image. The results of the target detection network are expressed as:

[0025] t=T(x,y,w,h,class,conf);

[0026] In the formula, (x, y) represents the coordinates of the upper left corner of the target, (w, h) represents the width and height of the target, class represents the category of the target; conf represents the confidence that the target is of a certain category, which is used to adjust the sensitivity of the target.

[0027] Since the sky background is relatively simple, the edge detection method can be used to identify the basic shape of the target, and the target's position reference point (x0, y0) in the image can be determined by calculating the center of mass.

[0028] Further, S3 specifically includes: using the position of the target relative to the center of the image and the focal length, calculating the azimuth angle and the pitch angle of the target relative to the optical axis of the optoelectronic device.

[0029]

[0030] Then the azimuth and elevation angles of the target relative to the optoelectronic device are

[0031]

[0032] Further, S4 specifically includes: using the positions of multiple fixed optoelectronic devices and the orientation of the target relative to the fixed optoelectronic devices to jointly solve the approximate position of the target.

[0033] The positions of the two photoelectric devices P1 (X1, Y1, Z1), P2 (X2, Y2, Z2) obtained in the preparatory work and the orientation of the same target relative to the two photoelectric devices

[0034] From the definition of the symmetric equation of a space line, we know that a space line can be represented by the direction of the line and a point on the line. The vectors of the two lines can be represented as

[0035] Thus, two straight lines connecting the target and the two optoelectronic devices can be obtained.

[0036]

[0037] If the two lines intersect, the intersection is the location of the target.

[0038] The joint solution can obtain the position of the target (X t ,Y t ,Z t )

[0039] In the actual solution process, since the centroids of the target at different angles do not necessarily coincide and there are errors in the coordinate conversion process, the two straight lines often cannot intersect. At this time, the midpoint of the common perpendicular segment of the two straight lines is the position of the target.

[0040] Assume that the intersection points of the common perpendicular line and the two straight lines are P3(X3, Y3, Z3) and P4(X4, Y4, Z4) respectively. According to the representation method of points on the straight line, they can be expressed as:

[0041]

[0042] And the perpendicular bisector is perpendicular to the two straight lines respectively, then

[0043]

[0044] From this we can solve t1 and t2, and then we can obtain P3(X3,Y3,Z3) and P4(X4,Y4,Z4).

[0045] The midpoint coordinates can be expressed as

[0046] From this, we can find the position of the target (X t ,Y t ,Z t ).

[0047] According to the performance of optoelectronic equipment, fixed optoelectronic equipment has a wide field of view but is affected by focal length and has low accuracy. Variable-focus optoelectronic equipment can accurately identify the target and lock the target position by changing the focal length. Therefore, the error of the target position is relatively large at this time, and it is an approximate target position.

[0048] Further, S5 specifically includes: according to the approximate position of the target, guiding the variable-focus optoelectronic device to track the target, calculating the precise position of the target and performing fine-grained detection on the properties of the target to improve the accuracy of target detection.

[0049] Using the target approximate location (X t ,Y t ,Z t ), calculate its azimuth and elevation angle relative to each variable-focus optoelectronic device When the zoom photoelectric device is in small focal length mode, it drives the servo to adjust the optical axis of the photoelectric device to azimuth and pitch angles.

[0050] Repeat S3 to S5 until the focal length of the variable-focus optoelectronic device is adjusted to a maximum value or accurate characteristic information of the target is obtained.

[0051] S6 specifically includes: when multiple optoelectronic devices grasp a target at the same time, multiple positions of the same target can be obtained. By weighted averaging each position point, the error can be minimized and the accuracy of the solution can be improved. When there are multiple aerial targets in the fixed optoelectronic device, the principle of prioritizing the nearest target is adopted to allocate the zoom optoelectronic device. That is, according to the nature and position of the target identified by the fixed optoelectronic device, the importance is sorted, and the important targets are tracked first. If there are multiple important targets or the number of zoom optoelectronic devices is greater than the number of targets, the principle of proximity is adopted for tracking.

[0052] Furthermore, S7 specifically includes: since aerial targets usually move faster and their trajectories may cross each other, the positions of the targets at multiple moments are used to predict the position of the target at the next moment using a Kalman filter, and the actual position of the target is compared using Bayesian inference to minimize the mean square of the prediction error, and predictions are performed and the covariance matrix is ​​updated in a loop to ultimately achieve stable target tracking.

[0053] S8 specifically includes: the manual input parameters of the user interaction part include the location of the optoelectronic device, the accuracy requirement threshold and the internal parameters of the optoelectronic device; the control functions include completing the manual / automatic selection of the optoelectronic device, information source reading, information fusion method, key target selection, detection and tracking status; the tracking status display includes the video information of each optoelectronic device, target category information, location information, motion information and threat level judgment status.

[0054] S9 specifically includes: a data management part, which is mainly used for recording the location information, pitch angle information, video image information, target detection and tracking conditions, and equipment operation conditions of each optoelectronic device and managing related data.

[0055] Another object of the present invention is to provide a short-range low-altitude photoelectric blindness compensation system using the short-range low-altitude photoelectric blindness compensation monitoring method, comprising:

[0056] Image information extraction module, used for object detection in images;

[0057] Photoelectric equipment detection module, used for detecting and transmitting various posture parameters of photoelectric equipment;

[0058] An optoelectronic device control module, used to control the rotation, pitch and focal length change of the variable-focus optoelectronic device;

[0059] Data processing module, used for continuous frame fusion, target position calculation and target trajectory tracking;

[0060] User interaction module, used for parameter input, monitoring control and monitoring information display;

[0061] The data management module is used to save information, query historical information and import external information.

[0062] Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the short-range and low-altitude optoelectronic blind spot monitoring method.

[0063] Another object of the present invention is to provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the short-range and low-altitude optoelectronic blind spot monitoring method.

[0064] Another object of the present invention is to provide an information data processing terminal, which includes the short-range and low-altitude optoelectronic blind spot monitoring system.

[0065] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0066] First, through the comprehensive use of various types of optoelectronic equipment, the detection and monitoring of low-altitude small targets can be realized. Through the combined application of existing optoelectronic equipment and the systematic processing of the information obtained, low-cost, high-efficiency, all-round and concealed monitoring of low-altitude targets can be achieved.

[0067] Enhances low altitude short range electro-optical surveillance by solving technical problems of existing systems. The following is an overview of key features and technical advances:

[0068] 1. Calibration of optoelectronic equipment (S1):

[0069] The method first calibrates the optoelectronic device to ensure accurate calibration of parameters such as position, height and optical axis azimuth. This step uses a calibration board and saves the device parameters through a data management module.

[0070] 2. Image information extraction (S2):

[0071] The system extracts information from image sequences to identify and locate aerial targets of interest. This process involves training a neural network using massive amounts of data to achieve automatic detection and recognition of targets, and adjusting sensitivity through confidence levels.

[0072] 3. Coordinate transformation (S3):

[0073] The method uses the azimuth and elevation angles of the optical axis of the optoelectronic device to convert the two-dimensional position of the identified target in the image into a three-dimensional real position relative to the optoelectronic device.

[0074] 4. Jointly solve the target approximate position (S4):

[0075] The method uses the relative position information of two or more fixed optoelectronic devices to calculate the approximate position of the target. This involves determining the target position by solving the intersection of the line of sight or the midpoint of the common perpendicular line from each device to the target.

[0076] 5. Improve positioning accuracy (S5):

[0077] In order to improve the detection accuracy of the target, the method uses the information obtained by the fixed optoelectronic device to guide the variable-focus optoelectronic device to further lock the target position, solve the target's more accurate position, and determine the target's attributes at the same time.

[0078] 6. Autonomous allocation target (S6):

[0079] For multiple aerial targets existing at the same time, the system can autonomously allocate variable-focus optoelectronic equipment to locate each target separately.

[0080] 7. Target tracking (S7):

[0081] The method adopts the target tracking algorithm to continuously track the target position and control the azimuth and elevation angles of the optoelectronic device to ensure continuous monitoring of the target.

[0082] 8. User interaction (S8):

[0083] Users can manually adjust parameters, correct accuracy, control the system, and pay special attention to high-threat targets, while obtaining feedback on the working status of the blind spot monitoring system.

[0084] 9. Data Management (S9):

[0085] The system saves surveillance video information and supports query, playback, deletion or import of historical data from other devices.

[0086] Technological advancements:

[0087] This method solves the accuracy and efficiency problems of existing monitoring systems through precise calibration, advanced image processing, neural network-based target recognition, and multi-device systematic collaborative tracking related technologies.

[0088] The target position is calculated by the vector intersection point and the midpoint of the common perpendicular line, which improves the accuracy of target positioning.

[0089] The system's ability to automatically assign tracking devices and make real-time adjustments represents a significant technological advancement in low-altitude surveillance.

[0090] This approach provides a comprehensive solution for improving the surveillance of low- and medium-altitude targets such as drones, shuttles, or balloons.

[0091] Second, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0092] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are:

[0093] Mature optoelectronic monitoring equipment can be used to deploy drone monitoring systems in key locations to achieve control over low-flying, slow-moving, and small drones.

[0094] (2) The technical solution of the present invention solves a technical problem that people have long been eager to solve but have never been able to solve: low-cost monitoring and management of unmanned equipment.

[0095] Third, the technical solution of the present invention solves the following technical problems in the prior art in industrial applications and has achieved significant technical progress:

[0096] Technical issues solved:

[0097] 1. Accuracy issues of low-altitude target detection:

[0098] In the prior art, the detection and positioning accuracy of low-altitude targets (such as shuttles or balloons) is limited, especially in complex marine environments, where targets are easily affected by background noise, resulting in large detection errors. This invention effectively solves the problem of low low-altitude target detection accuracy through precise calibration of optoelectronic equipment, efficient extraction of image information, and coordinate conversion.

[0099] 2. Efficiency issues of multi-target tracking and allocation:

[0100] Traditional systems lack effective allocation and tracking mechanisms when dealing with multiple low-altitude targets that appear simultaneously, which easily leads to waste of monitoring equipment resources or target omissions. The present invention ensures the efficiency of simultaneous multi-target monitoring through autonomous target allocation and efficient target tracking algorithms, solving the problem of unreasonable equipment resource allocation in the prior art.

[0101] 3. Error problem of target positioning:

[0102] In the prior art, due to the limitation of the field of view and focal length of the optoelectronic device, the positioning accuracy of the fixed device is low and the target cannot be accurately locked. The present invention solves the problem of insufficient positioning accuracy by jointly solving the orientation information of multiple fixed devices and using the variable-focus optoelectronic device to accurately lock the target, which significantly reduces the target positioning error.

[0103] Technological advancements:

[0104] 1. Improved the accuracy of low-altitude target detection and positioning:

[0105] Through precise equipment calibration and advanced image processing technology, this invention significantly improves the detection accuracy of low-altitude targets, especially in marine environments, and effectively addresses the challenges of complex backgrounds and small targets.

[0106] 2. Enhanced the system's multi-target processing capabilities:

[0107] This invention improves the system's responsiveness and resource utilization efficiency when processing multiple low-altitude targets simultaneously through autonomous allocation and efficient tracking, overcoming the limitations of traditional systems in multi-target processing.

[0108] 3. Accurate target positioning and attribute recognition are achieved:

[0109] The present invention not only improves the initial positioning accuracy of the target, but also realizes the determination of the target attributes through the precise positioning of the variable-focus optoelectronic device, so that the system can more accurately identify and track high-threat targets.

[0110] 4. Improved the flexibility of user interaction:

[0111] This technical solution allows users to manually adjust and control the monitoring system, providing greater flexibility and accuracy, especially in monitoring high-threat targets.

[0112] In general, the technical solution of the present invention has achieved significant technological progress in the field of low-altitude optoelectronic blind spot monitoring, can effectively solve multiple key problems in the existing technology in actual industrial applications, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0113] Figure 1 It is a flow chart of a short-range low-altitude photoelectric blind spot monitoring method provided by an embodiment of the present invention;

[0114] Figure 2 is a schematic diagram of a coordinate system provided by an embodiment of the present invention;

[0115] Figure 3 It is a structural diagram of a short-range and low-altitude optoelectronic blind spot monitoring system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0116] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. They include:

[0117] like Figure 1 As shown, the short-range low-altitude optoelectronic blind spot monitoring method provided by the embodiment of the present invention includes:

[0118] S1, pre-preparation, optoelectronic equipment calibration: before the equipment is deployed, the optoelectronic equipment parameters, position, height, optical axis azimuth, and optical axis pitch angle are calibrated one by one using the calibration board; the relevant parameters of each optoelectronic equipment are saved and managed through the data management module; the optoelectronic equipment calibration module uses the calibration board to calibrate the internal parameters of each optoelectronic equipment, and after selecting a reference point, the azimuth, distance, and height of each optoelectronic equipment relative to the reference point are calibrated. For fixed optoelectronic equipment, the azimuth and pitch angle of the optical axis need to be calibrated, and for variable-focus optoelectronic equipment, the optical axis reference azimuth and pitch angle need to be calibrated. During use, the attitude sensing module is used to sense the changes in the focal length, optical axis pitch angle, and optical axis azimuth of the variable-focus optoelectronic equipment in real time. If the position of the optoelectronic equipment changes relative to the reference point, the position of the optoelectronic equipment relative to the reference point is sensed in real time;

[0119] S2, image information extraction, identifies the marine targets of interest in the image sequence data and locates their positions in the images;

[0120] S3, coordinate conversion: according to the imaging principle, the position of the target is converted into the orientation relative to the optoelectronic device;

[0121] S4, joint solution: using the orientation information of the target relative to two or more fixed optoelectronic devices and the relative positions of the optoelectronic devices, solve the three-coordinate position of the target relative to the reference point and convert it into an approximate position;

[0122] S5, improving positioning accuracy: using the position of the target obtained in step 3, guiding the variable-focus optoelectronic device to find the target, and re-solving the precise position of the target to determine the target attributes;

[0123] S6, autonomous target allocation: for multiple targets in the air at the same time, the variable-focus optoelectronic equipment is automatically allocated to locate them separately;

[0124] S7, target tracking: Use the target tracking algorithm to track the position of the target and drive the servo of the variable-focus optoelectronic device to ensure continuous grasp of the target;

[0125] S8, user interaction: manually adjust parameters, correct accuracy, control system, display tracking status, focus on high-threat targets, and feedback the working status of the blind spot monitoring system;

[0126] S9, Data Management: Save surveillance video information, query, replay, delete or import historical data from other devices.

[0127] S1 specifically includes: in the preliminary preparation work, mainly calibrating the internal and external parameters of the optoelectronic equipment;

[0128] like Figure 1 As shown, the internal parameter matrix includes the focal length f of the optoelectronic device, the width d of the pixel unit x , the height of the pixel unit y , and the width w0 and height h0 of the CCD photosensitive element; the distortion mainly includes radial distortion (k1, k2, k3) and tangential distortion (p1, p2); the intrinsic parameter matrix and distortion matrix of the optoelectronic device are calibrated by Zhang Zhengyou calibration method;

[0129] The external parameters of the optoelectronic device mainly include the position of the optoelectronic device (X0, Y0, Z0) and the angle of the optical axis of the optoelectronic device. The center of the platform (set by yourself) is used as the origin of the horizontal coordinate system, the north direction is the Y axis, the east direction is the X axis, and the vertical horizontal plane is the Z axis; (X0, Y0) represents the position of the optoelectronic device relative to the reference point, and Z0 represents the height of the optoelectronic device above sea level. It indicates the azimuth of the optical axis of the optoelectronic device in the horizontal direction, that is, the angle between the optical axis of the optoelectronic device and the Y axis. The clockwise direction is positive and the value range is 90°~90°. θ0 indicates the elevation angle of the optical axis of the optoelectronic device, that is, the angle between the optical axis of the optoelectronic device and the horizontal plane. The upward direction is positive and the value range is 0°~90°.

[0130] S2 specifically includes: using image processing technology to extract information about targets of interest such as low-altitude drones from optoelectronic equipment images; using a target detection network model to train the neural network using massive data, automatically fitting target recognition features, and realizing target detection and recognition;

[0131] Collect low-altitude video images through the Internet or on-site, annotate and name the targets of interest, and convert them into data sets for neural network training; obtain network weights through multiple batches of training, and finally use them to detect targets in the image; the results detected by the target detection network are expressed as:

[0132] t=T(x,y,w,h,class,conf);

[0133] In the formula, (x, y) represents the coordinates of the upper left corner of the target, (w, h) represents the width and height of the target, class represents the category of the target; conf represents the confidence that the target is a certain category, which is used to adjust the sensitivity of the target;

[0134] Since the sky background is relatively simple, the edge detection method can be used to identify the basic shape of the target, and the target's position reference point (x0, y0) in the image can be determined by calculating the center of mass.

[0135] S3 specifically includes: using the position of the target relative to the center of the image and the focal length, calculating the azimuth angle and the pitch angle of the target relative to the optical axis of the optoelectronic device;

[0136]

[0137] Then the azimuth and elevation angles of the target relative to the optoelectronic device are

[0138]

[0139] S4 specifically includes: using the positions of multiple fixed optoelectronic devices and the orientation of the target relative to the fixed optoelectronic devices to jointly solve the approximate position of the target;

[0140] The positions of the two photoelectric devices P1 (X1, Y1, Z1), P2 (X2, Y2, Z2) obtained in the preparatory work and the orientation of the same target relative to the two photoelectric devices

[0141] From the definition of the symmetric equation of a space line, we know that a space line can be represented by the direction of the line and a point on the line; the vectors of two lines can be represented as

[0142] Thus, two straight lines connecting the target and the two photoelectric devices can be obtained;

[0143]

[0144] If the two lines intersect, the intersection is the location of the target;

[0145] The joint solution can obtain the position of the target (X t ,Y t ,Z t )

[0146] In the actual solution process, since the centroids of the target at different angles do not necessarily coincide, and there are errors in the coordinate conversion process, the two straight lines often cannot intersect. In this case, the midpoint of the common perpendicular line segment of the two straight lines is the position of the target.

[0147] Assume that the intersection points of the common perpendicular line and the two straight lines are P3(X3, Y3, Z3) and P4(X4, Y4, Z4) respectively. According to the representation method of points on the straight line, they can be expressed as:

[0148]

[0149] And the perpendicular bisector is perpendicular to the two straight lines respectively, then

[0150]

[0151] From this, we can solve for t1 and t2, and then we can obtain P3 (X3, Y3, Z3) and P4 (X4, Y4, Z4);

[0152] The midpoint coordinates can be expressed as

[0153] From this, we can find the position of the target (X t ,Y t ,Z t );

[0154] According to the performance of optoelectronic equipment, fixed optoelectronic equipment has a wide field of view but its accuracy is low due to the influence of focal length. Variable-focus optoelectronic equipment can accurately identify the target and lock the target position by changing the focal length. Therefore, the error of the target position required at this time is large and it is the approximate position of the target.

[0155] S5 specifically includes: according to the approximate position of the target, guiding the variable-focus optoelectronic device to track the target, calculating the precise position of the target and performing fine-grained detection of the properties of the target to improve the accuracy of target detection;

[0156] Using the target approximate location (X t ,Y t ,Z t ), calculate its azimuth and elevation angle relative to each variable-focus optoelectronic device When the zoom photoelectric device is in small focal length mode, it drives the servo to adjust the optical axis of the photoelectric device to azimuth and pitch angles.

[0157] Repeat S3 to S5 until the focal length of the variable-focus optoelectronic device is adjusted to a maximum value or accurate characteristic information of the target is obtained;

[0158] S6 specifically includes: when multiple optoelectronic devices grasp a target at the same time, multiple positions of the same target can be obtained. By taking a weighted average of each position point, the error can be minimized and the accuracy of the solution can be improved; when there are multiple aerial targets in the fixed optoelectronic device, the zoom optoelectronic device is allocated based on the principle of prioritizing the nearest target; that is, according to the nature and position of the target identified by the fixed optoelectronic device, the importance is ranked, and important targets are tracked first. If there are multiple important targets or the number of zoom optoelectronic devices is greater than the number of targets, tracking is carried out based on the principle of proximity.

[0159] S7 specifically includes: since aerial targets usually move fast and their trajectories intersect with each other, the positions of the targets at multiple moments are used to predict the position of the target at the next moment using a Kalman filter, and the actual position of the target is compared to minimize the mean square of the prediction error using Bayesian inference, and the prediction and covariance matrix are cyclically performed to finally achieve stable target tracking;

[0160] S8 specifically includes: the manual input parameters of the user interaction part include the location of the optoelectronic device, the accuracy requirement threshold and the internal parameters of the optoelectronic device; the control function includes completing the selection of optoelectronic devices, information source reading, information fusion mode, key target selection, detection and tracking; the tracking status display includes the video information of each optoelectronic device, target category information, location information, motion information and threat level judgment;

[0161] S9 specifically includes: a data management part, which is mainly used for recording the posture information, video image information, target detection and tracking conditions, and equipment operation conditions of each optoelectronic device and managing related data.

[0162] like Figure 3 As shown, the short-range low-altitude optoelectronic blind spot monitoring system provided by the embodiment of the present invention is characterized by comprising:

[0163] Image information extraction module, used for object detection in images;

[0164] Photoelectric equipment detection module, used for detecting and transmitting various posture parameters of photoelectric equipment;

[0165] An optoelectronic device control module, used to control the rotation, pitch and focal length change of the variable-focus optoelectronic device;

[0166] Data processing module, used for continuous frame fusion, target position calculation and target trajectory tracking;

[0167] User interaction module, used for parameter input, monitoring control and monitoring information display;

[0168] The data management module is used to save information, query historical information and import external information.

[0169] An application embodiment of the present invention provides a computer device, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the steps of a short-range and low-altitude optoelectronic blind spot monitoring method.

[0170] An application embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of a short-range and low-altitude optoelectronic blind spot monitoring method.

[0171] An application embodiment of the present invention provides an information data processing terminal, which includes a short-range low-altitude optoelectronic blind spot monitoring system.

[0172] The working principle of the short-range low-altitude photoelectric blind spot monitoring system of the present invention can be described in detail through the following six steps:

[0173] 1. Image information extraction and target detection:

[0174] The image information extraction module in the system first extracts image information from the image sequence acquired by the optoelectronic device. Using advanced image processing algorithms and pre-trained neural network models, the module can accurately identify low-altitude targets in the image, such as drones or small aircraft. The detected target information, including its location and category, is transmitted to the data processing module for further processing.

[0175] 2. Photoelectric equipment attitude parameter detection and control:

[0176] The optoelectronic device detection module is responsible for real-time monitoring of the attitude parameters of the optoelectronic device, including position, azimuth angle of the optical axis, pitch angle, etc. These attitude parameters will be transmitted to the optoelectronic device control module to adjust the rotation, pitch and focal length of the device, so as to ensure that the device is always in the best monitoring state and ensure continuous tracking and high-precision positioning of the target.

[0177] 3. Target position calculation and trajectory tracking:

[0178] The data processing module receives data from the image information extraction module and the optoelectronic device detection module, calculates the current position of the target, and improves the calculation accuracy of the target position through multi-frame image fusion technology. The module also uses advanced target tracking algorithms to continuously monitor the target's motion trajectory to ensure stable tracking of the target in complex environments.

[0179] 4. User interaction and system control:

[0180] The user interaction module provides a user-friendly interface for operators, allowing users to input monitoring parameters, manually control optoelectronic equipment, and monitor real-time target tracking information. Users can use this module to fine-tune the system, such as correcting detection accuracy, setting tracking priorities, and especially focusing on monitoring high-threat targets. In addition, this module is also responsible for displaying monitoring information, including real-time video and target trajectory.

[0181] 5.Data management and historical information query:

[0182] The data management module is responsible for storing all monitoring information, including target detection information, trajectory data, and video data. Users can query and replay historical information through this module to analyze previous monitoring records. In addition, the system also supports the import of external information to facilitate comprehensive management and further processing of data.

[0183] 6. Integration and application of information data processing terminals:

[0184] The monitoring system of the present invention is integrated in an information data processing terminal, which includes hardware components such as computer equipment, memory and processor. The computer program stored in the memory realizes the coordinated work of various functional modules of the above-mentioned monitoring system under the execution of the processor, thereby ensuring the efficiency and accuracy of the entire monitoring process. Through this integrated terminal device, the system can be widely used in different industrial environments, such as offshore oil platforms and port cargo areas, to achieve real-time monitoring and blind spot filling of low-altitude targets.

[0185] The following are two examples of industrial applications:

[0186] Example 1: Low-altitude drone monitoring of offshore oil platforms

[0187] Offshore oil platforms are often threatened by illegal intrusions from drones, which pose risks to the safety of the platforms, such as surveillance, damage to equipment, and even carrying dangerous items. Traditional monitoring methods make it difficult to accurately monitor and locate drones flying at low altitudes.

[0188] Photoelectric equipment arrangement: multiple photoelectric monitoring devices are arranged around the offshore oil platform, and the position, optical axis azimuth and pitch angle of each device are accurately calibrated using the calibration method of the present invention.

[0189] Target detection and tracking: When a drone enters the monitoring range, the system can quickly identify and locate the drone through image processing technology and a trained neural network model. The joint solution of multiple optoelectronic devices enables the specific location of the drone to be accurately located.

[0190] Real-time response: The system automatically assigns variable-focus optoelectronic devices to track the drone according to its threat level and continuously monitors its movement trajectory. If a drone is confirmed to be highly threatening, the platform security manager can manually adjust the tracking parameters through the system's user interaction module and take appropriate countermeasures.

[0191] Application effect:

[0192] It improves the offshore oil platform's ability to monitor low-altitude drones and reduces potential safety hazards caused by drone intrusions.

[0193] The system can automatically and efficiently manage multiple simultaneous drone targets, reducing the burden of manual monitoring and improving response speed.

[0194] Example 2: Low-altitude monitoring system in port cargo area

[0195] In the busy port cargo area, low-flying drones and other small aircraft are used for illegal activities such as smuggling, illegal filming, etc. Traditional monitoring methods have difficulty covering all areas, and the recognition and tracking accuracy of low-altitude targets is insufficient.

[0196] Implementation plan:

[0197] Equipment deployment: Multiple photoelectric monitoring devices are deployed at key locations in the port cargo area, such as cargo yards, unloading docks, etc. Through the method of the present invention, the equipment is fully calibrated before installation to ensure that it can accurately cover the required monitoring area.

[0198] Real-time detection and alarm: The system monitors the movement of low-altitude aircraft in real time and can quickly identify and locate any suspicious low-altitude targets, such as drones. When abnormal activity is detected, the system immediately issues an alarm and automatically guides the zoom device to the target location as needed to obtain a clearer image.

[0199] Historical data management: All surveillance videos and related data are automatically saved by the system. Managers can replay and view historical records at any time, or import data from other devices for comprehensive analysis.

[0200] Application effect:

[0201] Security in the port cargo area has been significantly improved, especially in terms of low-altitude surveillance, which can effectively prevent illegal activities and potential security threats.

[0202] Through real-time automated monitoring and data management, port management can better control and dispatch security resources, improving overall operational efficiency.

[0203] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. A person of ordinary skill in the art will appreciate that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVDROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0204] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A short-range low-altitude photoelectric blind spot monitoring method, characterized in that: First, the optoelectronic equipment is calibrated, including calibration of parameters, position, optical axis azimuth and pitch angle, and the attitude changes of the variable-focus equipment are sensed in real time; in the image information extraction stage, the marine target of interest is identified and located, and the three-coordinate position of the target is determined through coordinate transformation and joint solution. The variable-focus equipment is used to further improve the positioning accuracy, and multiple targets are autonomously allocated and located and tracked. At the same time, through user interaction, parameters are corrected, system status is monitored, and historical video data is managed to achieve efficient and accurate monitoring blind spots.

2. The short-range low-altitude photoelectric blind spot monitoring method according to claim 1, characterized in that: include: S1, pre-preparation, optoelectronic equipment calibration: before the equipment is deployed, the optoelectronic equipment parameters, position, height, optical axis azimuth, and optical axis pitch angle are calibrated one by one using the calibration board; the relevant parameters of each optoelectronic equipment are saved and managed through the data management module; the optoelectronic equipment calibration module uses the calibration board to calibrate the internal parameters of each optoelectronic equipment, and after selecting a reference point, the azimuth, distance, and height of each optoelectronic equipment relative to the reference point are calibrated. For fixed optoelectronic equipment, the azimuth and pitch angle of the optical axis need to be calibrated, and for variable-focus optoelectronic equipment, the optical axis reference azimuth and pitch angle need to be calibrated. During use, the attitude sensing module is used to sense the changes in the focal length, optical axis pitch angle, and optical axis azimuth of the variable-focus optoelectronic equipment in real time. If the position of the optoelectronic equipment changes relative to the reference point, the position of the optoelectronic equipment relative to the reference point is sensed in real time; S2, image information extraction, identifies the marine targets of interest in the image sequence data and locates their positions in the images; S3, coordinate conversion: according to the imaging principle, the position of the target is converted into the orientation relative to the optoelectronic device; S4, joint solution: using the orientation information of the target relative to two or more fixed optoelectronic devices and the relative positions of the optoelectronic devices, solve the three-coordinate position of the target relative to the reference point and convert it into an approximate position; S5, improving positioning accuracy: using the position of the target obtained in step 3, guiding the variable-focus optoelectronic device to find the target, and re-solving the precise position of the target to determine the target attributes; S6, autonomous target allocation: for multiple targets in the air at the same time, the variable-focus optoelectronic equipment is automatically allocated to locate them separately; S7, target tracking: Use the target tracking algorithm to track the position of the target and drive the servo of the variable-focus optoelectronic device to ensure continuous grasp of the target; S8, user interaction: manually adjust parameters, correct accuracy, control system, display tracking status, focus on high-threat targets, and feedback the working status of the blind spot monitoring system; S9, Data Management: Save surveillance video information, query, replay, delete or import historical data from other devices.

3. The short-range low-altitude photoelectric blind spot monitoring method according to claim 2, characterized in that: S1 specifically includes: in the preliminary preparation work, mainly calibrating the internal and external parameters of the optoelectronic equipment; The internal parameter matrix includes the focal length f of the optoelectronic device, the width d of the pixel unit x , the height of the pixel unit y , and the width w0 and height h0 of the CCD photosensitive element; the distortion mainly includes radial distortion (k1, k2, k3) and tangential distortion (p1, p2); the intrinsic parameter matrix and distortion matrix of the optoelectronic device are calibrated by Zhang Zhengyou calibration method; The external parameters of the optoelectronic device mainly include the position of the optoelectronic device (X0, Y0, Z0) and the angle of the optical axis of the optoelectronic device. The center of the platform is used as the origin of the horizontal coordinate system, the north direction is the Y axis, the east direction is the X axis, and the vertical horizontal plane is the Z axis; (X0, Y0) represents the position of the optoelectronic device relative to the reference point, and Z0 represents the height of the optoelectronic device above sea level. It indicates the azimuth of the optical axis of the optoelectronic device in the horizontal direction, that is, the angle between the optical axis of the optoelectronic device and the Y axis. The clockwise direction is positive and the value range is 90°~90°. θ0 indicates the elevation angle of the optical axis of the optoelectronic device, that is, the angle between the optical axis of the optoelectronic device and the horizontal plane. The upward direction is positive and the value range is 0°~90°. S2 specifically includes: using image processing technology to extract information about targets of interest such as low-altitude drones from optoelectronic equipment images; using a target detection network model to train the neural network using massive data, automatically fitting target recognition features, and realizing target detection and recognition; Collect low-altitude video images through the Internet or on-site, annotate and name the targets of interest, and convert them into data sets for neural network training; obtain network weights through multiple batches of training, and finally use them to detect targets in the image; the results detected by the target detection network are expressed as: t=T(x,y,w,h,class,conf); In the formula, (x, y) represents the coordinates of the upper left corner of the target, (w, h) represents the width and height of the target, class represents the category of the target; conf represents the confidence that the target is a certain category, which is used to adjust the sensitivity of the target; For the sky background, the edge detection method is used to identify the basic shape of the target, and the target's position reference point (x0, y0) in the image is determined by calculating the center of mass.

4. The short-range low-altitude photoelectric blind spot monitoring method according to claim 2, characterized in that: S3 specifically includes: using the position of the target relative to the center of the image and the focal length, calculating the azimuth angle and the pitch angle of the target relative to the optical axis of the optoelectronic device; Then the azimuth and elevation angles of the target relative to the optoelectronic device are 5. The short-range low-altitude optoelectronic blind spot monitoring method according to claim 2, characterized in that: S4 specifically includes: using the positions of multiple fixed optoelectronic devices and the orientation of the target relative to the fixed optoelectronic devices to jointly solve the approximate position of the target; The positions of the two photoelectric devices P1 (X1, Y1, Z1), P2 (X2, Y2, Z2) obtained in the preparatory work and the orientation of the same target relative to the two photoelectric devices From the definition of the symmetric equation of a space line, we know that a space line can be represented by the direction of the line and a point on the line; the vectors of two lines can be represented as Thus, two straight lines connecting the target and the two photoelectric devices can be obtained; l1: l2: If the two lines intersect, the intersection is the location of the target; The joint solution can obtain the position of the target (X t ,Y t ,Z t ) In the actual solution process, since the centroids of the target at different angles do not necessarily coincide, and there are errors in the coordinate conversion process, the two straight lines often cannot intersect. In this case, the midpoint of the common perpendicular line segment of the two straight lines is the position of the target. Assume that the intersection points of the common perpendicular line and the two straight lines are P3(X3, Y3, Z3) and P4(X4, Y4, Z4) respectively. According to the representation method of points on the straight line, they can be expressed as: And the perpendicular bisector is perpendicular to the two straight lines respectively, then From this, we can solve for t1 and t2, and then we can obtain P3 (X3, Y3, Z3) and P4 (X4, Y4, Z4); The midpoint coordinates can be expressed as From this, we can find the position of the target (X t ,Y t ,Z t ); According to the performance of optoelectronic equipment, fixed optoelectronic equipment has a wide field of view but its accuracy is low due to the influence of focal length. Variable-focus optoelectronic equipment can accurately identify the target and lock the target position by changing the focal length. Therefore, the error of the target position required at this time is large and it is the approximate position of the target.

6. The short-range low-altitude optoelectronic blind spot monitoring method according to claim 2, characterized in that: S5 specifically includes: according to the approximate position of the target, guiding the variable-focus optoelectronic device to track the target, calculating the precise position of the target and performing fine-grained detection of the properties of the target to improve the accuracy of target detection; Using the target approximate location (X t ,Y t ,Z t ), calculate its azimuth and elevation angle relative to each variable-focus optoelectronic device When the zoom photoelectric device is in small focal length mode, it drives the servo to adjust the optical axis of the photoelectric device to azimuth and pitch angle. Repeat S3 to S5 until the focal length of the variable-focus optoelectronic device is adjusted to a maximum value or accurate characteristic information of the target is obtained; S6 specifically includes: when multiple optoelectronic devices grasp a target at the same time, multiple positions of the same target can be obtained. By taking a weighted average of each position point, the error can be minimized and the accuracy of the solution can be improved; when there are multiple aerial targets in the fixed optoelectronic device, the zoom optoelectronic device is allocated based on the principle of prioritizing the nearest target; that is, according to the nature and position of the target identified by the fixed optoelectronic device, the importance is ranked, and important targets are tracked first. If there are multiple important targets or the number of zoom optoelectronic devices is greater than the number of targets, tracking is carried out based on the principle of proximity.

7. The short-range low-altitude optoelectronic blind spot monitoring method according to claim 2, characterized in that: S7 specifically includes: since aerial targets usually move fast and their trajectories intersect with each other, the positions of the targets at multiple moments are used to predict the position of the target at the next moment using a Kalman filter, and the actual position of the target is compared to minimize the mean square of the prediction error using Bayesian inference, and the prediction and covariance matrix are cyclically performed to finally achieve stable target tracking; S8 specifically includes: the manual input parameters of the user interaction part include the location of the optoelectronic device, the accuracy requirement threshold and the internal parameters of the optoelectronic device; the control function includes completing the selection of optoelectronic devices, information source reading, information fusion mode, key target selection, detection and tracking; the tracking status display includes the video information of each optoelectronic device, target category information, location information, motion information and threat level judgment; S9 specifically includes: a data management part, which is mainly used for recording the posture information, video image information, target detection and tracking conditions, and equipment operation conditions of each optoelectronic device and managing related data.

8. A short-range low-altitude photoelectric blind spot monitoring system using the short-range low-altitude photoelectric blind spot monitoring method according to any one of claims 1 to 7, characterized in that: include: Image information extraction module, used for object detection in images; Photoelectric equipment detection module, used for detecting and transmitting various posture parameters of photoelectric equipment; An optoelectronic device control module, used to control the rotation, pitch and focal length change of the variable-focus optoelectronic device; Data processing module, used for continuous frame fusion, target position calculation and target trajectory tracking; User interaction module, used for parameter input, monitoring control and monitoring information display; The data management module is used to save information, query historical information and import external information.

9. A computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the short-range and low-altitude optoelectronic blind spot monitoring method as described in any one of claims 1 to 6.

10. An information data processing terminal, comprising the short-range and low-altitude optoelectronic blind spot monitoring system as claimed in claim 7.

Images