Invasive target safety monitoring method, safety monitoring device and engineering machinery

Abstract

The invention discloses an intrusion target safety monitoring method, a safety monitoring device and engineering machinery. The intrusion target safety monitoring method comprises the following steps: acquiring peripheral image information of the engineering machinery; inputting the peripheral image information of the current frame into the dynamic target detection model to obtain the current actual target information of the intrusion target; inputting the preorder actual target information of the intrusion target into the target tracking prediction model to obtain the current prediction tracking information of the intrusion target; performing target matching according to the current actual target information and the current prediction tracking information; in the case of successful matching, the successfully matched intrusion target of the current frame is associated with the intrusion target of the previous frame to output a trajectory tracking chain, so that an operator can quickly identify possible dangers according to the trajectory tracking chain of the intrusion target and effectively deal with the dangers. Therefore, safety accidents caused by sight blind areas or misoperation are reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of security monitoring, and in particular relates to a security monitoring method for an intrusion target, a security monitoring device and engineering machinery. Background Art

[0002] In engineering construction, construction machinery, especially construction cranes, are important construction equipment. They have a wide range of operations, high operational complexity, and large blind spots, posing a potential threat to the surrounding environment and personnel safety. Traditional safety monitoring methods often rely on manual observation or simple physical isolation measures, which cannot quickly identify dangers and are difficult to effectively respond to emergencies, posing safety hazards. Summary of the invention

[0003] In view of the above-mentioned defects or shortcomings, the present invention provides an intrusion target security monitoring method, a security monitoring device and engineering machinery, aiming to solve the technical problem that the existing security monitoring means of engineering machinery cannot quickly identify dangers.

[0004] To achieve the above-mentioned object, the first aspect of the present invention provides a method for monitoring the safety of an intrusion target, wherein the method for monitoring the safety of an intrusion target is applied to engineering machinery and comprises:

[0005] Acquire surrounding image information of construction machinery;

[0006] Input the surrounding image information of the current frame into the dynamic target detection model to obtain the current actual target information of the intrusion target;

[0007] Inputting the previous actual target information of the intrusion target into the target tracking prediction model to obtain the current predicted tracking information of the intrusion target, wherein the previous actual target information of the intrusion target is obtained by inputting the surrounding image information of the previous frame into the dynamic target detection model;

[0008] Target matching is performed based on current actual target information and current predicted tracking information;

[0009] In the case of successful matching, the successfully matched intrusion target of the current frame is associated with the intrusion target of the previous frame to output a trajectory tracking chain.

[0010] In one embodiment of the present invention, the intrusion target security monitoring method further includes:

[0011] When the trajectory tracking chain confirms that the intrusion target is close to the warning area, the first warning prompt is issued.

[0012] In one embodiment of the present invention, the intrusion target security monitoring method further includes:

[0013] If it is confirmed that the intrusion target has entered the warning area, a second warning prompt will be issued.

[0014] In one embodiment of the present invention, a plurality of cameras are sequentially arranged at intervals along the circumference of the engineering machine, and obtaining the surrounding image information of the engineering machine includes:

[0015] Get the original images taken by multiple cameras;

[0016] Calibrate multiple cameras separately to obtain the intrinsic parameters and distortion parameters of each camera;

[0017] The corresponding original images are corrected according to the intrinsic parameters and distortion parameters of each camera to obtain the corrected images in the peripheral image information;

[0018] The corrected image is calibrated based on the equipment principal coordinates of the construction machinery to obtain the transformation matrix between each camera in the surrounding image information.

[0019] In one embodiment of the present invention, the camera is a fisheye camera, and the de-distortion model of the fisheye camera adopts the Kannala-Brandt model.

[0020] In one embodiment of the present invention, the dynamic target detection model is a transformer model based on BEV, and the surrounding image information of the current frame is input into the dynamic target detection model to obtain the current actual target information of the intrusion target, including:

[0021] Input the surrounding image information of the current frame into the BEV-based transformer model to obtain the current actual target information, which includes the category information and location information of the intrusion target;

[0022] Among them, the training method of the bev-based transformer model includes:

[0023] Input the training image into the feature network of the BEV-based transformer model to obtain multi-scale feature information of the training image;

[0024] Based on temporal self-attention and spatial cross-attention, the transformed bird's-eye view features corresponding to the training image are output;

[0025] The transformed bird's-eye view features are feature aligned with the actual bird's-eye view features to form a set of query objects.

[0026] In one embodiment of the present invention, the target tracking prediction model is a Kalman filter prediction model, and the preceding actual target information of the intrusion target is input into the target tracking prediction model to obtain the current prediction tracking information of the intrusion target, including:

[0027] Initialize the Kalman filter prediction model to obtain the initialized state vector and covariance matrix;

[0028] The Kalman filter prediction model is used to perform state covariance prediction based on the preceding actual target information and the covariance matrix of the intrusion target to predict the current prediction tracking information of the intrusion target.

[0029] In one embodiment of the present invention, after performing state covariance prediction on the Kalman filter prediction model according to the preceding actual target information and the covariance matrix of the intrusion target to predict the current predicted tracking information of the intrusion target, the following steps are also included:

[0030] The state vector and covariance matrix are modified according to the deviation between the current predicted tracking information of the intrusion target and the current actual target information.

[0031] In one embodiment of the present invention, before matching the intrusion target according to the current actual target information and the current predicted tracking information, the method further includes:

[0032] When the category confidence of the intrusion target is less than the set threshold, it is determined that the intrusion target does not participate in target matching.

[0033] In one embodiment of the present invention, performing target matching according to current actual target information and current predicted tracking information includes:

[0034] The current actual target information and the current predicted tracking information are matched by a greedy algorithm to obtain the global optimal target match for multiple targets.

[0035] To achieve the above objective, a second aspect of the present invention provides a security monitoring device, wherein the security monitoring device is configured to execute the intrusion target security monitoring method described above.

[0036] To achieve the above objective, a third aspect of the present invention provides an engineering machine, wherein the engineering machine includes the safety monitoring device described above.

[0037] Through the above technical solution, the intrusion target security monitoring method provided by the present invention has the following beneficial effects:

[0038] When the above-mentioned intrusion target security monitoring method is used, after the surrounding image information of the current frame is input into the dynamic target detection model, the current actual target information of the intrusion target can be obtained, and after the previous actual target information of the intrusion target is input into the target tracking prediction model, the current predicted tracking information of the intrusion target can be obtained, and the target matching can be performed according to the current actual target information and the current predicted tracking information. If the intrusion target of the current frame successfully matches the intrusion target of the previous frame, the targets of the previous and next frames can be related to output a complete trajectory tracking chain of the intrusion target. The operator can quickly identify the possible dangers and take effective measures according to the trajectory tracking chain of the intrusion target to reduce safety accidents caused by blind spots or operational errors. In addition, by matching and associating the intrusion targets of the previous and next frames, the tracking of intrusion targets (for example, human-machine targets) can be made more accurate and reliable.

[0039] Other features and advantages of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on the structures shown in these drawings without creative work. In the accompanying drawings:

[0041] Figure 1 is a flow chart of a method for security monitoring of intrusion targets according to an embodiment of the present invention;

[0042] Figure 2 is a flowchart of a training method of a transformer model based on BEV according to an embodiment of the present invention;

[0043] Figure 3 is a schematic diagram of viewing angles of multiple cameras arranged on an engineering machine according to an embodiment of the present invention;

[0044] Figure 4 It is a schematic diagram of a construction framework of a bev-based transformer model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0046] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0047] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0048] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection, an electrical connection, or communication with each other; it can be a direct connection, or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0049] The following describes the intrusion target security monitoring method, security monitoring device and engineering machinery of the present invention with reference to the accompanying drawings.

[0050] like Figure 1 As shown, the present invention provides a method for monitoring the security of an intrusion target, wherein the method for monitoring the security of an intrusion target comprises:

[0051] Step S100, obtaining surrounding image information of the construction machinery.

[0052] Specifically, the construction machinery can be equipped with a camera for acquiring surrounding image information. If the intruding target in front of or behind the construction machinery is to be tracked, the camera can be installed on the front or rear side of the construction machinery accordingly. If 360-degree monitoring of the construction machinery is required, multiple cameras are required to be spaced along the sides of the construction machinery to cover all sides of the construction machinery.

[0053] Step S110, inputting the surrounding image information of the current frame into the dynamic target detection model to obtain the current actual target information of the intrusion target.

[0054] It can be understood that a dynamic target detection model is pre-built, and after the surrounding image information of the current frame is input into the dynamic target detection model, the dynamic target detection model can output the current actual target information of all intrusion targets around the construction machinery. At the same time, all intrusion targets in the surrounding image information of the previous frame can also obtain the previous actual target information.

[0055] Step S120, inputting the previous actual target information of the intrusion target into the target tracking prediction model to obtain the current predicted tracking information of the intrusion target, wherein the previous actual target information of the intrusion target is obtained by inputting the surrounding image information of the previous frame into the dynamic target detection model.

[0056] Specifically, after obtaining the preceding actual target information of all intruding targets in the surrounding image information of the previous frame through the dynamic target detection model, the preceding actual target information of the intruding targets can be input into the pre-built target tracking prediction model. The target tracking prediction model can predict the current predicted tracking information of the intruding targets in the next frame (that is, the current frame) based on the preceding actual target information.

[0057] Step S130, performing target matching based on current actual target information and current predicted tracking information.

[0058] It can be understood that compared with the surrounding image information of the previous frame, the surrounding image information of the current frame has the phenomenon of newly appeared intrusion targets or lost intrusion targets. In order to be able to output a more accurate and reliable trajectory tracking chain later, it is necessary to match the intrusion targets in the surrounding image information of the current frame with the intrusion targets in the surrounding image information of the previous frame one by one. Specifically, it can be calculated by using the current actual target information and the current predicted tracking information to determine the target matching result that best conforms to reality.

[0059] Step S140, when the match is successful, the successfully matched intrusion target of the current frame is associated with the intrusion target of the previous frame to output a trajectory tracking chain.

[0060] When the above-mentioned intrusion target security monitoring method is used, after the surrounding image information of the current frame is input into the dynamic target detection model, the current actual target information of the intrusion target can be obtained, and after the previous actual target information of the intrusion target is input into the target tracking prediction model, the current predicted tracking information of the intrusion target can be obtained, and the target matching can be performed according to the current actual target information and the current predicted tracking information. If the intrusion target of the current frame successfully matches the intrusion target of the previous frame, the targets of the previous and next frames can be related to output a complete trajectory tracking chain of the intrusion target. The operator can quickly identify the possible dangers and take effective measures according to the trajectory tracking chain of the intrusion target to reduce safety accidents caused by blind spots or operational errors. In addition, by matching and associating the intrusion targets of the previous and next frames, the tracking of intrusion targets (for example, human-machine targets) can be made more accurate and reliable.

[0061] In one embodiment of the present invention, step S130, after performing target matching according to the current actual target information and the current predicted tracking information, further includes:

[0062] In case of unsuccessful matching, the intrusion target of the current frame is determined to be a newly appeared target or the intrusion target of the previous frame is determined to be a lost target.

[0063] Specifically, the remaining target after unsuccessful matching may be the intrusion target of the current frame or the intrusion target of the previous frame. If the remaining target after unsuccessful matching is the intrusion target of the current frame, it is determined to be a newly appeared target. If the remaining target after unsuccessful matching is the intrusion target of the previous frame, it is determined to be a lost target.

[0064] In one embodiment of the present invention, the intrusion target security monitoring method further includes: issuing a first warning prompt when it is confirmed according to the trajectory tracking chain that the intrusion target is close to the warning area.

[0065] It is understandable that a warning area can be set for the engineering machinery according to the operating characteristics and safety specifications of the engineering machinery. If the trajectory tracking chain of a certain intrusion target approaches the warning area, it is determined that the intrusion target has a tendency to enter the warning area. At this time, the alarm device on the engineering machinery can be controlled to issue a first warning prompt, such as a voice broadcast alarm. Specifically, the distance between the intrusion target and the warning area can be confirmed through the trajectory tracking chain. If the distance is less than the set distance, it is determined that the intrusion target is close to the warning area; or after confirming the distance between the intrusion target and the warning area through the trajectory tracking chain, the estimated time for the intrusion target to enter the warning area is calculated based on the distance and the current speed of the intrusion target. If the estimated time is less than the set time, it is determined that the intrusion target is close to the warning area. The addition of the first warning prompt allows operators to quickly and autonomously identify dangers and leave sufficient time to respond to emergencies.

[0066] In one embodiment of the present invention, the intrusion target security monitoring method further includes: issuing a second warning prompt when it is confirmed that the intrusion target has entered the warning area.

[0067] Furthermore, when the intrusion target is confirmed to enter the warning area according to the trajectory tracking chain, the alarm device can also be controlled to issue a second warning prompt, such as: voice broadcast alarm plus roarer, LED light alarm, to remind the operator again of the danger.

[0068] Furthermore, the current actual target information of the current frame can be input into the target tracking prediction model to obtain the subsequent prediction tracking information of the intrusion target. When it is predicted that the intrusion target will enter the warning area in the next frame according to the subsequent prediction tracking information, an early warning prompt can be issued.

[0069] In one embodiment of the present invention, Figure 3 As shown, a plurality of cameras are sequentially arranged along the circumference of the construction machinery. Specifically, the construction machinery may be a construction crane. A plurality of cameras are sequentially arranged along the circumference of the turntable of the construction crane. Specifically, there may be six cameras. The six cameras are sequentially arranged in the front, rear, left front, left rear, right front and right rear directions of the turntable to obtain image data of a 360-degree range around the construction crane, and the field of view of each camera may be greater than or equal to 180 degrees. Step S100, obtaining the surrounding image information of the construction machinery includes:

[0070] Get the original images taken by multiple cameras;

[0071] Calibrate multiple cameras separately to obtain the intrinsic parameters and distortion parameters of each camera;

[0072] The corresponding original images are corrected according to the intrinsic parameters and distortion parameters of each camera to obtain the corrected images in the peripheral image information;

[0073] The corrected image is calibrated based on the equipment principal coordinates of the construction machinery to obtain the transformation matrix between each camera in the surrounding image information.

[0074] Specifically, after multiple cameras capture the original image, the camera lens has different incident angles and different paths in the lens because the edge light and the center light are at different angles. The area in the original image far from the center of the lens (i.e., the edge of the image) may be greatly curved, resulting in picture deformation, so camera calibration is required to correct the distortion in the original image. By calibrating multiple cameras, the intrinsic parameters and distortion parameters of each camera can be obtained respectively, so that the original image of the camera can be distorted using the above parameters to obtain a corrected image in the peripheral image information. Each original image corresponds to a corrected image, and then the corrected image is calibrated based on the device principal coordinates of the engineering machinery to obtain the transformation matrix between each camera in the peripheral image information, that is, the peripheral image information includes the corrected image corresponding to each original image and the transformation matrix between each camera.

[0075] In one embodiment of the present invention, the camera may be a fisheye camera, and the field of view of the fisheye camera may reach 180 to 270 degrees to obtain a larger shooting field of view. The fisheye camera is designed to adopt non-similarity imaging, and distortion is introduced in the imaging process. The purpose of introducing distortion is to break through the limitation of the imaging angle of view by compressing the diameter space, thereby achieving wide-angle imaging. At the same time, the de-distortion model of the fisheye camera adopts the Kannala-Brandt model. There are many assumptions about the projection method of the fisheye camera, such as equidistant projection, equi-stereoscopic projection, orthogonal projection, stereoscopic projection and linear projection, but the real fisheye camera does not completely follow the above model assumptions, and the Kannala-Brandt model follows a general form of estimation, which can be applied to different types of fisheye cameras. The formula of the Kannala-Brandt model is:

[0076] δ=γ 2 *tan(θ)(1+k1*tan 2 θ+k2*tan 4 θ+k3*tan 6 θ)

[0077] Where δ is the distortion distance, γ is the incident angle of the light, k1, k2, k3 are distortion parameters, and θ is the arc length of the distorted pixel point (x, y, z) in polar coordinates.

[0078] In one embodiment of the present invention, the dynamic target detection model may be a transformer model based on BEV. Step S120, inputting the surrounding image information of the current frame into the dynamic target detection model to obtain the current actual target information of the intrusion target includes:

[0079] The surrounding image information of the current frame is input into the BEV-based transformer model to obtain the current actual target information, which includes the category information and location information of the intrusion target.

[0080] Specifically, the BEV-based transformer model is a deep learning model based on the Transformer architecture, which aims to convert multi-view camera images into a unified bird's-eye view (BEV) representation. The model can generate BEV features with strong characterization capabilities by fusing spatial features and temporal features. The BEV features are the current actual target information including the category information and position information of the intrusion target, which not only improves the perception accuracy but also meets the real-time requirements. In addition, the current actual target information can also include the size information of the intrusion target, and the speed information and acceleration information of the intrusion target can be calculated through the position information. Of course, the present invention is not limited to this. The dynamic target detection model can also be a comprehensive use of multiple sensors (such as laser radar, millimeter wave radar, visual sensor, IMU, etc.) by the image to obtain the full range of information of the target. Through the data fusion algorithm, the image is fused with the data of multiple sensors to improve the accuracy and robustness of target detection, and the fused data is used for target tracking and trajectory prediction. Multiple sensors can complement each other's shortcomings and improve the overall performance of the system. The data fusion algorithm can make full use of the information of various sensors to improve the accuracy and speed of target detection.

[0081] Specifically, see Figure 2 and Figure 4 , the training methods of the bev-based transformer model include:

[0082] Step S200: input the training image into the feature network of the BEV-based transformer model to obtain multi-scale feature information of the training image.

[0083] Specifically, the surrounding image information of the construction machinery is collected as training images, and the training images are input into the feature network of the BEV-based transformer model. The feature network is a 2D image feature network and can extract multi-scale feature information in the training images.

[0084] Step S210, outputting the converted bird's-eye view features corresponding to the training image based on the temporal self-attention and the spatial cross-attention.

[0085] Specifically, the encoding module of the temporal self-attention and spatial cross-attention mechanism in the BEV-based transformer model is used to extract and transform the bird's-eye view features, completing the transformation from the surround training image to the bird's-eye view features. After conversion to the bird's-eye view features, different task heads can be connected, such as: 3D detection, BEV segmentation, trajectory prediction, etc.

[0086] Step S220: aligning the converted bird's-eye view features with the actual bird's-eye view features to form a set of query objects.

[0087] It can be understood that the actual bird's-eye view corresponding to the training image is obtained in advance. After obtaining the converted bird's-eye view features after model conversion, the converted bird's-eye view features are aligned with the actual bird's-eye view features to form a set of object queries. The object query is the query object, which is represented by a vector describing the target. After decoding, the category information and location information can be obtained.

[0088] In one embodiment of the present invention, the target tracking prediction model may be a Kalman filter prediction model. Step S120, inputting the previous actual target information of the intrusion target into the target tracking prediction model to obtain the current prediction tracking information of the intrusion target includes:

[0089] Initialize the Kalman filter prediction model to obtain the initialized state vector and covariance matrix;

[0090] The Kalman filter prediction model is used to perform state covariance prediction based on the preceding actual target information and the covariance matrix of the intrusion target to predict the current prediction tracking information of the intrusion target.

[0091] Specifically, the role of Kalman filter prediction is to make the best estimate of the state of the system through the dynamic model and observation data of the system, and update the uncertainty of the state (i.e., the error covariance matrix). The core idea of ​​Kalman filter is to use the dynamic model and measurement data of the system in a recursive way to estimate the state of the system and update the uncertainty of the system state.

[0092] Furthermore, the Kalman filter prediction process can be:

[0093] (1) Initialization parameters: Initialize the state transfer matrix F in the Kalman filter parameters. Specifically, the state transfer matrix F can be constructed based on the constant acceleration. According to the state transfer matrix F and the initial current actual target information detected from the bev perspective, the initialized state vector S and covariance matrix C are set.

[0094] (2) State covariance prediction: Based on the previous state vector of the previous frame (which can be determined based on the previous actual target information) and the covariance matrix C, the state vector (F*S 上一帧 ) and the corresponding covariance (F*C 上一帧 *F T +Q 噪声 ), F T is the transposed matrix of the state transfer matrix F, Q 噪声 is the noise value, according to the state vector of the current frame (F*S 上一帧 ) to determine the current forecast tracking information.

[0095] Furthermore, the state transfer matrix F can be:

[0096]

[0097] In the formula, Δt is set to 1 / frequency, frequency is the frequency of continuous frame acquisition of the image; I 3*3 for 0 3*3 for

[0098] In one embodiment of the present invention, after performing state covariance prediction on the Kalman filter prediction model according to the preceding actual target information and the covariance matrix of the intrusion target to predict the current predicted tracking information of the intrusion target, the following steps are also included:

[0099] The state vector and covariance matrix are modified according to the deviation between the current predicted tracking information of the intrusion target and the current actual target information.

[0100] Specifically, in the correction step, the Kalman filter uses the deviation between the current observation value and the predicted value for correction to obtain a more accurate estimate, thereby improving the stability and accuracy of the system.

[0101] In one embodiment of the present invention, step S130, before matching the intrusion target according to the current actual target information and the current predicted tracking information, further includes:

[0102] When the category confidence of the intrusion target is less than the set threshold, it is determined that the intrusion target does not participate in target matching.

[0103] Specifically, in target detection, the category confidence is used to filter prediction results. By setting a threshold (such as 0.5), the model can exclude predictions with lower confidence, thereby improving the accuracy and efficiency of detection. When the category confidence of the intrusion target is greater than the set threshold, it is determined to track the intrusion target and participate in target matching.

[0104] In one embodiment of the present invention, step S130, performing target matching according to current actual target information and current predicted tracking information includes:

[0105] The current actual target information and the current predicted tracking information are matched by a greedy algorithm to obtain the global optimal target match for multiple targets.

[0106] Specifically, the greedy algorithm refers to taking the best choice in each step of the selection when solving the problem, that is, the local optimum, so as to hope that the final result can also reach the optimum. The result obtained by the greedy algorithm is not necessarily the optimal solution, but it is relatively close to the optimal solution. More specifically, the current actual target information contains the measured position information, and the current predicted tracking information contains the predicted position information. When matching all the intrusion targets of the current frame with the intrusion targets of the previous frame, the position gap between all the intrusion targets of the current frame and all the intrusion targets of the previous frame is calculated through the measured position information and the predicted position information. The position gap of the successfully matched intrusion targets is relatively close to the minimum, so as to achieve the global optimal target matching of multiple targets.

[0107] In addition, the present invention also provides a security monitoring device, wherein the security monitoring device is configured to execute the intrusion target security monitoring method described above. Since the security monitoring device adopts all the technical solutions of the above embodiments, it at least has all the beneficial effects brought by the technical solutions of the above embodiments, which will not be described one by one here.

[0108] Specifically, the intrusion target security monitoring method program completes the onnx conversion and accelerates quantization, and is deployed to the JetsonOrin nx embedded board. The GPU interface is called to load the model and run it, and other module algorithms are deployed to the CPU to run. After the deployment of the entire algorithm is completed, the surrounding security monitoring of construction machinery can be realized.

[0109] In addition, the present invention provides an engineering machine, wherein the engineering machine includes the safety monitoring device described above. Since the engineering machine adopts all the technical solutions of the above embodiments, it at least has all the beneficial effects brought by the technical solutions of the above embodiments, which will not be described one by one here. It should be particularly noted that the engineering machine includes but is not limited to an engineering crane.

[0110] Therefore, the intrusion target security monitoring method, security monitoring device and engineering machinery provided by the present invention have the following advantages:

[0111] 1. Cameras are installed in the six directions of the construction machinery, which can conduct 360-degree all-round monitoring of the surrounding area of ​​the construction machinery. Through real-time monitoring and early warning, safety accidents caused by blind spots or operational errors can be effectively reduced.

[0112] 2. Using multi-camera visual detection, the target perception fusion in the multi-camera is converted to a bird's-eye view, the position information of the detected target under the bird's-eye view is obtained, and then the trajectory of the target under the bird's-eye view is predicted through the acceleration-invariant Kalman filter. The targets detected in the previous and next frames are correlated and matched through the greedy algorithm to obtain the tracking trajectory of the target, which helps to improve the stability and accuracy of the human-machine intrusion warning system.

[0113] 3. Converting the human-machine targets in the surround image to a single bird's-eye view can better obtain the corresponding position relationship and trajectory relationship between the current engineering machinery and the human-machine dynamic targets, making the tracking of human-machine targets more accurate and reliable. Through the BEV conversion network based on transformer (neural network), the image features collected by the surround camera are converted into bird's-eye view image features, making image fusion more accurate, avoiding fusion errors caused by inaccurate parameters, reducing the complexity of fusion, and making processing more efficient.

[0114] 4. This algorithm is deployed in an embedded system, making it easy to build the device on various construction machinery. It can follow the construction machinery to perform all-round human-machine intrusion detection in various scenarios. It is easy to install and maintain, and has good versatility and scalability.

[0115] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0116] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0117] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0119] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0120] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0121] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0122] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0123] The above are only embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for monitoring the security of an intrusion target, characterized in that: The intrusion target security monitoring method is applied to engineering machinery and includes: Acquire surrounding image information of construction machinery; Input the surrounding image information of the current frame into the dynamic target detection model to obtain the current actual target information of the intrusion target; Inputting the previous actual target information of the intrusion target into the target tracking prediction model to obtain the current predicted tracking information of the intrusion target, wherein the previous actual target information of the intrusion target is obtained by inputting the surrounding image information of the previous frame into the dynamic target detection model; Target matching is performed based on current actual target information and current predicted tracking information; In the case of successful matching, the successfully matched intrusion target of the current frame is associated with the intrusion target of the previous frame to output a trajectory tracking chain.

2. The intrusion target security monitoring method according to claim 1, characterized in that: The intrusion target security monitoring method also includes: When it is confirmed according to the trajectory tracking chain that the intrusion target is close to the warning area, a first warning prompt is issued; And / or, when it is confirmed that the intrusion target has entered the warning area, a second warning prompt is issued.

3. The intrusion target security monitoring method according to claim 1, characterized in that: The engineering machine is provided with a plurality of cameras spaced in sequence along the periphery, and the acquisition of the peripheral image information of the engineering machine includes: Get the original images taken by multiple cameras; Calibrate multiple cameras separately to obtain the intrinsic parameters and distortion parameters of each camera; The corresponding original images are corrected according to the intrinsic parameters and distortion parameters of each camera to obtain the corrected images in the peripheral image information; The corrected image is calibrated based on the equipment principal coordinates of the construction machinery to obtain the transformation matrix between each camera in the surrounding image information.

4. The intrusion target security monitoring method according to claim 3, characterized in that: The camera is a fisheye camera, and the dedistortion model of the fisheye camera adopts the Kannala-Brandt model.

5. The intrusion target security monitoring method according to claim 1, characterized in that: The dynamic target detection model is a transformer model based on BEV. The inputting of the surrounding image information of the current frame into the dynamic target detection model to obtain the current actual target information of the intrusion target includes: Input the surrounding image information of the current frame into the BEV-based transformer model to obtain the current actual target information, which includes the category information and location information of the intrusion target; The training method of the bev-based transformer model includes: Input the training image into the feature network of the BEV-based transformer model to obtain multi-scale feature information of the training image; Based on temporal self-attention and spatial cross-attention, the transformed bird's-eye view features corresponding to the training image are output; The transformed bird's-eye view features are feature aligned with the actual bird's-eye view features to form a set of query objects.

6. The intrusion target security monitoring method according to claim 1, characterized in that: The target tracking prediction model is a Kalman filter prediction model, and the inputting of the previous actual target information of the intrusion target into the target tracking prediction model to obtain the current prediction tracking information of the intrusion target includes: Initialize the Kalman filter prediction model to obtain the initialized state vector and covariance matrix; The Kalman filter prediction model is used to perform state covariance prediction based on the preceding actual target information and the covariance matrix of the intrusion target to predict the current prediction tracking information of the intrusion target.

7. The intrusion target security monitoring method according to claim 6, characterized in that: The method further includes: performing state covariance prediction on the Kalman filter prediction model according to the preceding actual target information of the intrusion target and the covariance matrix obtained by initialization to predict the current predicted tracking information of the intrusion target: The state vector and covariance matrix are modified according to the deviation between the current predicted tracking information of the intrusion target and the current actual target information.

8. The intrusion target security monitoring method according to any one of claims 1 to 7, characterized in that: Before matching the intrusion target according to the current actual target information and the current predicted tracking information, the following steps are also included: When the category confidence of the intrusion target is less than the set threshold, it is determined that the intrusion target does not participate in the target matching; And / or, the target matching according to the current actual target information and the current predicted tracking information includes: The current actual target information and the current predicted tracking information are matched by a greedy algorithm to obtain the global optimal target match for multiple targets.

9. A security monitoring device, characterized in that: The security monitoring device is configured to execute the intrusion target security monitoring method according to any one of claims 1 to 8.

10. An engineering machine, characterized in that: The construction machine comprises the safety monitoring device according to claim 9.

Images