Equipment dynamic obstacle avoidance method and system based on behavior trend prediction, equipment and medium

Abstract

The invention relates to the technical field of machine vision recognition, in particular to a dynamic obstacle avoidance method and system for equipment based on behavior trend prediction, equipment and a medium, and the projection range of the equipment on the ground can be effectively adjusted by introducing an image segmentation expansion scale regression mechanism driven by equipment PLC features and an image segmentation expansion prediction model. The method not only improves the fitting precision of the real boundary, but also ensures the consistency of the landmark covered by the expansion boundary, and provides stronger data support for collision modeling of image-physical fusion. A human body positioning and equipment lower edge convex edge construction method based on image bottom edge center point back projection is introduced to serve as a key link of collision modeling, so that the two-dimensional space relation between equipment and personnel is restored. Through the above method, not only is the diversity of different equipment forms paid attention to, but also the directivity and orientation characteristics of the equipment can be accurately modeled, and the stability and interpretation of collision pre-judgment are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine vision recognition, in particular to a device dynamic obstacle avoidance method and system based on behavior trend prediction, device and medium. BACKGROUND

[0002] With the rapid development of intelligent perception and industrial computing technology, image segmentation and physical projection technology has shown great potential in intelligent manufacturing, safety monitoring, and human-machine collaboration. As a multi-modal fusion method combining machine vision and device state modeling, the device-person collision prediction system introduces image back-projection modeling and multi-scale inflation weight regression mechanism, which not only significantly improves the fitting accuracy of the prediction area, but also ensures the ground projection consistency and collision judgment accuracy, thereby promoting the development of drilling platform safety automation technology. In the drilling operation environment, ensuring the accuracy of device spatial distribution and personnel behavior prediction is crucial for executing high-precision safety avoidance. Operation safety often requires temporal and spatial accuracy, boundary judgment stability, and prediction foresight to meet the needs of real-time obstacle avoidance and behavior prompting. However, the limitations of existing technology make collision detection based on camera images have boundary uncertainty and insufficient spatial restoration, resulting in large final judgment error and poor warning effect. SUMMARY

[0003] The purpose of the present application is to provide a device dynamic obstacle avoidance method and system based on behavior trend prediction, device and medium to solve the above problems of the prior art.

[0004] The present application is realized by the following technical solutions: A device dynamic obstacle avoidance method based on behavior trend prediction, comprising: Obtaining a current image input frame, obtaining a target detection frame of a personnel target detection and a device semantic segmentation mask based on the current image input frame: Establishing a ground two-dimensional coordinate system, calculating the coordinates of the bottom center pixel point of the target detection frame, and back-projecting the bottom center pixel point to the ground two-dimensional coordinate system; Obtaining all pixel points of the device semantic segmentation mask and camera parameters, back-projecting the pixel points to the ground two-dimensional coordinate system based on the camera parameters to obtain a projection area contour polygon, obtaining a device effective projection area based on the projection area contour polygon, and performing morphological dilation correction on the device effective projection area to obtain a final device effective projection area; Obtaining a plurality of continuous frame images after the current image input frame, obtaining state vectors of personnel and devices based on the continuous frame images, respectively, performing k-step recursion using the state vectors and state transition matrix to obtain predicted positions of the personnel and devices; An early warning model is established. Based on the predicted location of personnel and equipment, the current early warning score is output through the evaluation model. An early warning threshold is set. When the early warning score is greater than the early warning threshold, an alarm control signal is issued.

[0005] Preferably, the calculation of the coordinates of the center pixel of the bottom edge of the target detection box includes: ; ; ; In the formula, For the target detection bounding box, These are the pixel coordinates of the target detection bounding box. , These are the coordinates of the center pixel of the bottom edge. , These represent the coordinates of the personnel on the ground after back projection. The inverse function of the camera. The height is the 3D world coordinate of the center pixel of the bottom edge.

[0006] Preferably, obtaining the projection region outline polygon and obtaining the effective projection region of the device based on the projection region outline polygon includes: ; ; In the formula, and The first The coordinates of the pixels of the semantic segmentation mask of a device projected onto the ground in a two-dimensional coordinate system. and These are the first three semantic segmentation masks for the device. The coordinates of each pixel The set of pixels for the semantic segmentation mask. The projection region is a polygonal outline, specifically a set of projection points; Calculate the polygon outline of the projection area Geometric centroid coordinates Select Greater than or equal to Using the target point as the objective point, calculate the minimum circumscribed convex polygon based on the objective point to construct the effective projection area of ​​the device.

[0007] Preferably, the morphological dilation correction of the effective projection area of ​​the device includes: Set the expansion kernel to a square structuring element, and finally define the final expansion mask: ; Prediction with machine learning model: ; Machine learning model input PLC state , output an integer value , finally according to the number of times learned by the machine learning model, get the new boundary of the inflation ; ; ； In the formula, is the final inflation mask, is the inflation kernel, is the effective projection area of the device, is the output of the machine learning model, is the PLC state, is the machine learning model, is the maximum scale, is the convex hull algorithm, is the XOR operation.

[0008] Preferably, the state vector includes: ; ; ; ; ; In the formula, is the state vector, , is the speed at time , , is the acceleration at time , , is the jerk at time , is the interframe time, is the disturbance term.

[0009] Preferably, the k-step recursion with the state vector and the state transition matrix to obtain the predicted position of the personnel and the device includes: ; In the formula, predicted state, is the state transition matrix, is the prediction step.

[0010] Preferably, the establishment of the early warning model includes: ; ; ; ; wherein, is an intersection over union of the bounding box of the person and the inflated bounding box of the device, is a center distance, , is the corresponding geometric center coordinate of the person, , is the corresponding geometric center coordinate, is a relative kinetic energy, is a planar velocity vector of the person in the ground coordinate system, is a planar velocity vector of the device in the ground coordinate system, is a body equivalent mass, is a warning score, , , is a risk weight.

[0011] In a second aspect, the present application also provides a device dynamic obstacle avoidance system based on behavior trend prediction, used for executing the device dynamic obstacle avoidance method based on behavior trend prediction described above, comprising: a projection processing module configured to obtain a current image input frame, obtain a target detection box of a person target detection and a device semantic segmentation mask based on the current image input frame, establish a ground two-dimensional coordinate system, calculate the coordinate of the bottom edge center pixel point of the target detection box, and project the bottom edge center pixel point to the ground two-dimensional coordinate system; obtain all pixel points of the device semantic segmentation mask and camera parameters, project the pixel points to the ground two-dimensional coordinate system based on the camera parameters, obtain a projection area contour polygon, obtain a device effective projection area based on the projection area contour polygon, perform morphological inflation correction on the device effective projection area, and obtain a final device effective projection area; a warning module configured to obtain a plurality of continuous frame images after the current image input frame, obtain state vectors of the person and the device based on the plurality of continuous frame images, respectively, perform k-step recursion on the state vectors and a state transition matrix to obtain predicted positions of the person and the device; establish a warning model, output a current warning score based on the predicted positions of the person and the device through an evaluation model, set a warning threshold, and when the warning score is greater than the warning threshold, issue an alarm control signal.

[0012] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the device dynamic obstacle avoidance method based on behavior trend prediction when executing the computer program.

[0013] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the device dynamic obstacle avoidance method based on behavior trend prediction.

[0014] The technical scheme of the present application has at least the following advantages and beneficial effects: The present application introduces a device PLC feature driven image segmentation inflation scale regression mechanism, which can effectively adjust the projection range of the device on the ground. Not only improves the fitting accuracy of the real boundary, but also ensures the consistency of the inflation boundary covering the landmark, provides stronger data support for image-physical fusion collision modeling. The introduction of human body positioning based on image bottom edge center point back projection and device lower edge convex edge construction method as the key link of collision modeling makes the two-dimensional spatial relationship between the device and the personnel restored. Through the above method, not only the diversity of different device forms is concerned, but also the device directionality and orientation characteristics can be accurately modeled, which significantly improves the stability and interpretability of collision prediction. Especially under the condition of high angle oblique shooting camera, this technology significantly improves the projection stability and readability.

[0015] Combined with the upgraded behavior state prediction method, the state quantity update mechanism including speed, acceleration and momentum is adopted to realize the continuous prediction of the behavior trend of the device or personnel. Compared with the traditional Kalman filter, this method has stronger dynamic response, and has high robustness under multi-source disturbance and non-stationary behavior, which significantly improves the prediction effect and warning range. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical scheme of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0017] Fig. 1 The flowchart of the present application; Fig. 2 The system structure diagram of the present application. DETAILED DESCRIPTION

[0018] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0019] The division of the modules appearing in the present application is a logical division, and in actual application, another division mode can be used, for example, a plurality of modules can be combined or integrated in another system, or some features can be ignored or not executed. In addition, the connection, coupling or communication in the present application can be a direct connection, coupling or communication between associated objects, or indirect connection, coupling or communication through other devices, and in addition, the connection, coupling or communication between objects can be electrical or other similar forms, which are not limited in the present application.

[0020] The independently described modules or sub-modules can be physically separated or not physically separated, can be software implemented or hardware implemented, and part of the modules or sub-modules can be implemented by software, and the functions of the part of the modules or sub-modules are called by a processor, and the other part of the modules or sub-modules are implemented by hardware, for example, by hardware circuit. In addition, part or all of the modules can be selected to achieve the purposes of the present application according to actual needs.

[0021] Please refer to Figs. 1-2 The present application provides a device dynamic obstacle avoidance method based on behavior trend prediction, comprising: S101: acquiring a current image input frame, obtaining a target detection frame of personnel target detection and a device semantic segmentation mask based on the current image input frame: S102: establishing a ground two-dimensional coordinate system, calculating the coordinates of the bottom center pixel point of the target detection frame, and back-projecting the bottom center pixel point to the ground two-dimensional coordinate system; Since the drilling platform camera generally shoots at a downward angle, the real positions of personnel and equipment will be distorted in the image, and direct collision judgment will cause misjudgment. Therefore, the image pixels need to be back-projected back to the ground coordinate system to construct a unified two-dimensional space. The position of the object on the ground is obtained by back-projection of the camera imaging model. Homogeneous coordinate conversion is performed using camera internal and external parameters.

[0022] ; The two-dimensional pixel coordinates after projection are represented by u and v, which are usually in the form of homogeneous coordinates; The camera internal parameter matrix is represented by K, which contains the focal length and the principal point, and the size is ; represents camera extrinsic parameters, represents a spatial transformation; is a point in the world coordinate system, expressed in homogeneous form; both the device and the person are projected to the two-dimensional projection plane in this way.

[0023] Under the condition of shooting from the overhead angle of the drilling platform, the image target frame of the person does not represent the actual position, and the center point of the bottom edge can best express the ground position where the person is actually located. Therefore, the point needs to be back-projected from the image pixel to the ground plane as the coordinate basis for collision judgment.

[0024] S103: Obtain all pixel points of the device semantic segmentation mask and camera parameters, back-project the pixel points to the ground two-dimensional coordinate system based on the camera parameters, obtain a projection area contour polygon, obtain a device effective projection area based on the projection area contour polygon, and perform morphological dilation correction on the device effective projection area to obtain a final device effective projection area; S104: Obtain a plurality of continuous frame images after a current image input frame, obtain state vectors of the personnel and the device based on the continuous frame images respectively, perform k-step recursion using the state vectors and a state transition matrix to obtain predicted positions of the personnel and the device; S105: Establish an early warning model, output a current early warning score based on the predicted positions of the personnel and the device through an evaluation model, set an early warning threshold, and when the early warning score is greater than the early warning threshold, issue an alarm control signal.

[0025] The image segmentation and inflation prediction model can effectively adjust the projection range of the device on the ground by introducing the image segmentation and inflation scale regression mechanism driven by the PLC feature of the device. Not only is the fitting accuracy of the real boundary improved, but also the consistency of the inflation boundary covering the landmark is ensured, providing stronger data support for the collision modeling of image-physical fusion. The human body positioning based on the back-projection of the image bottom edge center point and the device lower edge convex edge construction method are introduced as key links of the collision modeling, so that the two-dimensional spatial relationship between the device and the personnel is restored. Through the above method, not only the diversity under different device forms is concerned, but also the device directionality and orientation features can be accurately modeled, significantly improving the stability and interpretability of collision prediction. Especially under the condition of high-angle oblique shooting camera, this technology significantly improves the projection stability and readability.

[0026] Combined with the upgraded behavior state prediction method, a state quantity update mechanism including velocity, acceleration and momentum is adopted to realize continuous prediction of the behavior trend of the device or the personnel. Compared with the traditional Kalman filter, this method has stronger dynamic responsiveness and high robustness under multi-source disturbance and non-stationary behavior, significantly improving the prediction effect and warning range.

[0027] In an example embodiment of the present application, the calculation of the coordinates of the bottom center pixel point of the target detection frame includes: ; ; ; In the formula, is a target detection frame, are pixel coordinates of the target detection frame, , are coordinates of the bottom center pixel point, , are coordinate positions of the personnel on the ground after back projection, is a camera inverse function, is the height of the three-dimensional world coordinate of the bottom center pixel point, represents the bottom center pixel point on the ground.

[0028] In an example embodiment of the present application, under the overhead camera condition, the segmentation mask of the device in the image has a perspective compression effect. To determine the collision on the ground, the “most likely contact ground area” of the device needs to be determined. The geometric center of the polygon formed after back projection of the mask area in the image is closest to the projection area generated by the camera view angle imaging. Therefore, the ground projection area is constructed.

[0029] Specifically, all pixel points in the device segmentation mask are obtained, and the camera parameters are used to back project them to the ground coordinate plane. The mask points are back projected to the ground to construct the device projection area contour polygon.

[0030] The obtained projection area contour polygon, the device effective projection area based on the projection area contour polygon includes: ; ; In the formula, and are the coordinates of the pixel points of the first device semantic segmentation mask back projected to the ground two-dimensional coordinate system, and are the coordinates of the first pixel point in the device semantic segmentation mask, is a pixel point set of the semantic segmentation mask, is a projection area contour polygon, specifically a projection point set; The geometric center coordinates of the projection area contour polygon are calculated, is greater than or equal to The points are taken as target points, and a minimum circumscribed convex polygon is calculated based on the target points to construct the device effective projection area.

[0031] ; ; The minimum circumscribed convex polygon is calculated for these points, which is defined as the device effective projection area, and the device effective projection area is constructed. ； The ground point set is located below the center of gravity (including the center of gravity); The device effective projection area (minimum circumscribed convex polygon) is used for subsequent collision judgment and inflation modeling.

[0032] In an example embodiment of the application, due to the change of camera angle and device height, the device outline segmented in the image is usually smaller than the actual footprint projection and cannot cover the actual area. Morphological inflation is used. Based on the image segmentation area, a fixed structure element is used for multiple inflation operations to expand its ground projection boundary. In the multi-scale inflation scheme, the structure element acts on the segmented device effective projection area, and the weight represents the contribution of different scale inflation. However, the device in the image cannot be infinite. When the inflation area has covered the entire image, it is meaningless to increase the size of the inflation kernel. Therefore, the weight vector is set according to the image size (H, W) and the inflation kernel size, and the maximum dimension upper limit of

[0033] The morphological inflation correction of the device effective projection area includes; The inflation kernel is set as a square structure element with a size of (2r+1)* (2r+1), where r is the circular kernel radius, and the final inflation mask is defined as ; Prediction is made using a machine learning model: ; The machine learning model inputs the PLC state , and outputs an integer value , and finally the new boundary of the inflation is obtained according to the number of times learned by the machine learning model , wherein the PLC state The state vector represents the PLC, which is a set of continuous signals acquired and output by the controller. Examples include features such as multiple digital inputs, multiple digital outputs, multiple analog inputs, multiple program segment numbers, and multiple alarm flags.

[0034] ； In the formula, For the final expansion mask, For an expanding core, The effective projection area of ​​the device. The output of the machine learning model, PLC status. For machine learning models (Naive Bayes is selected). For the largest scale, For convex hull algorithm, This is an XOR operation.

[0035] In one exemplary embodiment of the present invention, to predict the future position of equipment or personnel, it is necessary to consider not only velocity but also acceleration, and even jerk, to improve the accuracy of short-term dynamic prediction. An eight-dimensional state vector is established, and the future position state is recursively derived using third-order Newtonian kinematics formulas.

[0036] The state vector includes: ; ; ; ; ; In the formula, For state vectors, , for The speed of time , for acceleration at any moment , for The acceleration of time, Inter-frame time, This is a disturbance term.

[0037] By using image segmentation models or key point detection, the pixel position of the center of a person's foot or the bottom edge of equipment protrusion in the image can be identified. , the imaging-anti-projection matrix is obtained by camera calibration, the pixel position is converted into the ground projection position, the point is taken as the starting point of all dynamic states, belongs to the observation value, and the subsequent speed and acceleration are all differential / fitted estimated values. When the frame interval is , the position difference between the front and rear frames is estimated: ; and is the coordinate of t time, and is the coordinate of t-1 time.

[0038] Using the past frame position information, a linear trend is fitted by least squares: ; In the formula, is the frame number, is the speed to be solved.

[0039] The position sequence obtained from the image requires at least 3 frames for each target to stably estimate the trend direction and speed of the device or personnel moving on the ground, which is the primary factor for predicting future position.

[0040] ; The speed change is extracted through continuous frame images. People suddenly accelerate (such as running), devices suddenly start, decelerate, and turn, all of which correspond to obvious acceleration changes; acceleration mutation is often an important leading information of behavior intention.

[0041] Jerk, the change rate of acceleration is defined as: ; It is obtained by differentiating the acceleration estimation sequence; it can be removed by low-pass filtering to remove high-frequency jitter and retain trend changes; it represents the "mutability of motion behavior"; personnel suddenly turn around, jump, devices start violently, or stop the process all have large jerk values; the larger the jerk value, the more likely it is a "impact behavior" with high danger. The derivation of the three variables follows the time derivative chain in physics: Position Speed Acceleration Jerk.

[0042] In a discrete-time system, it is approximately: ; In the formula, , and velocity, acceleration and jerk in discrete time system, respectively.

[0043] In one example embodiment of the present application, the predicting the position of the person and the equipment using the state vector and the state transition matrix for k steps of recursion comprises: ; wherein, the predicted state, is the state transition matrix, is the prediction step.

[0044] In this embodiment, the state transition matrix is: ; Note that the first two rows update the coordinates : the position is determined by the velocity and the acceleration; the 3rd-4th rows update the velocity: the velocity is affected by the acceleration; the 5th-6th rows keep the acceleration constant; the 7th-8th rows are identity, indicating that the angular velocity, the height remain unchanged.

[0045] In one example embodiment of the present application, the establishing the early warning model comprises: ; ; ; ; wherein, is the intersection over union of the predicted bounding box of the person and the inflated bounding box of the equipment, is the center distance, , is the corresponding geometric center coordinate (that is, the coordinate of the person) , , , is the corresponding geometric center coordinate, is the relative kinetic energy, is the planar velocity vector of the person in the ground coordinate system, is the planar velocity vector of the equipment in the ground coordinate system, is the equivalent mass of the human body (here, it is set to be the common 70 kg), is the early warning score, , , , is the risk weight.

[0046] The weight coefficient is The center distance, the kinetic energy term, and the weight of the super parameter to be debugged on the labeled "safe / dangerous" sample can be set according to specific conditions.

[0047] It should be noted that the bottom midpoint of the target detection frame selected in the embodiment can be understood as a small circle with a certain area, and the intersection ratio calculation is performed with the device effective projection area which is also a two-dimensional polygon.

[0048] wherein, , The judgment threshold can be flexibly set according to the on-site safety strategy. The alarm mode includes: real-time voice prompt "personnel close to dangerous area"; record alarm log in the control system; control the device to enter the automatic stop state, etc.

[0049] A device dynamic obstacle avoidance system based on behavior trend prediction is used to execute the device dynamic obstacle avoidance method based on behavior trend prediction, and comprises: The projection processing module is configured to obtain a current image input frame, obtain a target detection frame of a personnel target detection and a device semantic segmentation mask based on the current image input frame: establish a ground two-dimensional coordinate system, calculate the coordinates of the bottom center pixel point of the target detection frame, and project the bottom center pixel point to the ground two-dimensional coordinate system; obtain all pixel points of the device semantic segmentation mask and camera parameters, project the pixel points to the ground two-dimensional coordinate system based on the camera parameters, obtain a projection area contour polygon, obtain a device effective projection area based on the projection area contour polygon, and perform morphological dilation correction on the device effective projection area to obtain a final device effective projection area; The early warning module is configured to obtain a plurality of continuous frame images after the current image input frame, obtain state vectors of personnel and devices in the plurality of continuous frame images based on the continuous frame images, perform k-step recursion on the state vectors and the state transition matrix to obtain predicted positions of the personnel and the devices, establish an early warning model, output a current early warning score based on the predicted positions of the personnel and the devices through the evaluation model, set an early warning threshold, and output an alarm control signal when the early warning score is greater than the early warning threshold.

[0050] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0051] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. The computer software product stored in a storage medium includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the embodiments of the method of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0052] The above merely describes the preferred embodiments of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A device dynamic obstacle avoidance method based on behavioral trend prediction, characterized in that, include: Obtain the current image input frame, and based on the current image input frame, obtain the object detection bounding box for person object detection and the device semantic segmentation mask: Establish a ground two-dimensional coordinate system, calculate the coordinates of the center pixel of the bottom edge of the target detection box, and back-project the center pixel of the bottom edge onto the ground two-dimensional coordinate system; Obtain all pixels of the device semantic segmentation mask and camera parameters, back-project the pixels to the ground two-dimensional coordinate system based on the camera parameters to obtain the outline polygon of the projection area, obtain the effective projection area of ​​the device based on the outline polygon of the projection area, and perform morphological dilation correction on the effective projection area of ​​the device to obtain the final effective projection area of ​​the device. Acquire several consecutive frames of images after the current image input frame. Based on the consecutive frames of images, obtain the state vectors of personnel and equipment for the personnel and equipment in the several consecutive frames of images. Use the state vectors and state transition matrix to perform k-step recursion to obtain the predicted positions of personnel and equipment. An early warning model is established. Based on the predicted location of personnel and equipment, the current early warning score is output through the evaluation model. An early warning threshold is set. When the early warning score is greater than the early warning threshold, an alarm control signal is issued.

2. The device dynamic obstacle avoidance method based on behavioral trend prediction according to claim 1, characterized in that, The calculation of the coordinates of the center pixel of the bottom edge of the target detection box includes: ； ； ； In the formula, For the target detection bounding box, These are the pixel coordinates of the target detection bounding box. , These are the coordinates of the center pixel of the bottom edge. , These represent the coordinates of the personnel on the ground after back projection. The inverse function of the camera. The height is the 3D world coordinate of the center pixel of the bottom edge.

3. The device dynamic obstacle avoidance method based on behavioral trend prediction according to claim 2, characterized in that, The obtained projection region contour polygon, and the effective projection region of the device based on the projection region contour polygon, include: ； ； In the formula, and The first The coordinates of the pixels of the semantic segmentation mask of a device projected onto the ground in a two-dimensional coordinate system. and These are the first three semantic segmentation masks for the device. The coordinates of each pixel The set of pixels for the semantic segmentation mask. The projection region is a polygonal outline, specifically a set of projection points; Calculate the polygon outline of the projection area Geometric centroid coordinates Select Greater than or equal to Using the target point as the objective point, calculate the minimum circumscribed convex polygon based on the objective point to construct the effective projection area of ​​the device.

4. The device dynamic obstacle avoidance method based on behavioral trend prediction according to claim 3, characterized in that, The morphological dilation correction of the effective projection area of ​​the device includes: Set the expansion kernel to a square structuring element, and finally define the final expansion mask: ； Using machine learning models for prediction: ； Machine learning model input PLC status Output an integer value. Finally, based on the number of times learned by the machine learning model, the result of the inflation is obtained. Constructing new boundaries ; ； In the formula, For the final expansion mask, For an expanding core, The effective projection area of ​​the device. The output of the machine learning model, PLC status. For machine learning models, For the largest scale, For convex hull algorithm, This is an XOR operation.

5. The device dynamic obstacle avoidance method based on behavioral trend prediction according to claim 4, characterized in that, The state vector includes: ； ； ； ； ； In the formula, For state vectors, , for The speed of time, , for acceleration at any moment , for The acceleration of time, Inter-frame time, This is a disturbance term.

6. The device dynamic obstacle avoidance method based on behavioral trend prediction according to claim 5, characterized in that, The process of using state vectors and state transition matrices to perform k-step recursion to obtain the predicted locations of personnel and equipment includes: In the formula, Predicted state Here is the state transition matrix. To predict the step size.

7. The device dynamic obstacle avoidance method based on behavioral trend prediction according to claim 6, characterized in that, The establishment of the early warning model includes: ； ； ； ； In the formula, To predict the intersection-union ratio (IU / UK) of the bounding boxes for personnel and the expanded bounding boxes for equipment, Distance from the center , The geometric center coordinates of the personnel. , for Corresponding geometric center coordinates As relative kinetic energy, Let V be the planar velocity vector of the person in the ground coordinate system. Let V be the planar velocity vector of the device in the ground coordinate system. For human equivalent mass For warning scores, , , Risk weights.

8. A device dynamic obstacle avoidance system based on behavioral trend prediction, characterized in that, A device dynamic obstacle avoidance method based on behavioral trend prediction as described in any one of claims 1-7, comprising: The projection processing module is configured to acquire the current image input frame, and based on the current image input frame, acquire the target detection box for personnel target detection and the device semantic segmentation mask: establish a ground two-dimensional coordinate system, calculate the coordinates of the center pixel of the bottom edge of the target detection box, and back-project the center pixel of the bottom edge onto the ground two-dimensional coordinate system; acquire all pixels of the device semantic segmentation mask and camera parameters, back-project the pixels onto the ground two-dimensional coordinate system based on the camera parameters to obtain the projection area outline polygon, obtain the device effective projection area based on the projection area outline polygon, and perform morphological dilation correction on the device effective projection area to obtain the final device effective projection area; The early warning module is configured to acquire several consecutive frames of images after the current image input frame, acquire the state vectors of personnel and equipment in the several consecutive frames of images, and perform k-step recursion using the state vectors and state transition matrix to obtain the predicted positions of personnel and equipment; establish an early warning model, output the current early warning score based on the predicted positions of personnel and equipment through the evaluation model, set an early warning threshold, and issue an alarm control signal when the early warning score is greater than the early warning threshold.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a device dynamic obstacle avoidance method based on behavioral trend prediction as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements a device dynamic obstacle avoidance method based on behavioral trend prediction as described in any one of claims 1-7.

Images