Target behavior detection method and device, equipment, storage medium and program product

Abstract

The invention discloses a target behavior detection method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring a monitoring video of a target job task in a target place, wherein the monitoring video comprises continuous multi-frame monitoring images; inputting the continuous multiple frames of monitoring images into a preset rotating target detection model to obtain rotating target frame information of each monitoring object in each monitoring image, the rotating target detection model being obtained by training according to a plurality of training samples and annotation data of each training sample; according to the rotating target frame information of the crane and the rotating target frame information of the suspension arm in each monitoring image, the operation state of the target operation task in each monitoring image is determined; under the condition that the operation state represents that the suspension arm is in operation, the intersection ratio of the rotating target frame information of each operator to a target area is calculated, and the target area is determined and obtained by the rotating target frame information of the crane and the suspension arm; and when the intersection ratio is greater than a preset threshold value, determining that the target behavior exists in the target place.

Description

Technical Field

[0001] The present application belongs to the field of computer vision technology, and in particular, relates to a method, device, equipment, storage medium and program product for detecting target behavior. Background Art

[0002] In the process of industrial construction, various types of lifting equipment are increasingly used in the fields of construction, production and logistics. Especially under complex working conditions, cranes, truck cranes and other equipment are widely used in the installation of large equipment, emergency rescue, cargo transportation and other fields due to their efficient, safe and flexible operation characteristics. However, this is accompanied by various safety hazards caused by the lifting equipment. Among them, people standing under the boom has become one of the main causes of lifting safety accidents. The existing technology is that each production unit supervises by arranging supervisors or video surveillance at the lifting site. However, this kind of manual detection method, on the one hand, is prone to the situation that the supervisor may lose concentration during long-term work and fail to discover safety hazards in time. On the other hand, the supervisor's judgment often relies on personal experience, resulting in the inability to guarantee the accuracy of supervision. Summary of the invention

[0003] The embodiments of the present application provide a method, apparatus, device, storage medium and program product for detecting a target behavior, which can improve the accuracy of target behavior detection.

[0004] In a first aspect, an embodiment of the present application provides a method for detecting a target behavior, the method comprising: Acquire a surveillance video of a target operation task in a target location, wherein the surveillance video includes a plurality of continuous surveillance images; Inputting the continuous multiple frames of monitoring images into a preset rotating target detection model to obtain rotating target frame information of each monitoring object in each monitoring image, wherein the monitoring object includes one of a crane, a boom and at least one operator, and the rotating target detection model is trained based on multiple training samples and labeled data of each training sample; Determine the operation status of the target operation task in each monitoring image according to the rotation target frame information of the crane and the rotation target frame information of the boom in each monitoring image, wherein the operation status is used to indicate whether the boom is operating or not operating; When the operation state represents that the boom is operating, calculating the intersection ratio of the rotating target frame information of each operator and the target area, the target area being determined by the rotating target frame information of the crane and the rotating target frame information of the boom; When the intersection ratio is greater than a preset threshold, it is determined that a target behavior exists in the target location, and the target behavior indicates that the operator is in the target area.

[0005] In a second aspect, an embodiment of the present application provides a device for detecting a target behavior, the device comprising: A first acquisition module is used to acquire a monitoring video of a target operation task in a target location, wherein the monitoring video includes a continuous plurality of monitoring frames; A detection module, used for inputting the continuous multiple frames of monitoring images into a preset rotating target detection model to obtain rotating target frame information of each monitoring object in each monitoring image, wherein the monitoring object includes one of a crane, a boom and at least one operator, and the rotating target detection model is trained based on multiple training samples and labeled data of each training sample; A first determination module is used to determine the operation status of the target operation task in each monitoring image according to the rotation target frame information of the crane and the rotation target frame information of the boom in each monitoring image, wherein the operation status is used to indicate whether the boom is operating or not operating; a calculation module, for calculating, when the operation state represents that the boom is operating, an intersection ratio of the rotation target frame information of each operator and a target area, wherein the target area is determined by the rotation target frame information of the crane and the rotation target frame information of the boom; The second determination module is used to determine that there is a target behavior in the target place when the intersection ratio is greater than a preset threshold, and the target behavior indicates that the operator is in the target area.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method for detecting the target behavior as described in any one of the above items is implemented.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, a method for detecting a target behavior as described in any one of the above is implemented.

[0008] In a fifth aspect, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes a method for detecting a target behavior as described in any one of the above.

[0009] The target behavior detection method, device, equipment, storage medium and program product of the embodiment of the present application can determine the rotation target frame information of each monitored object in each monitoring image through the rotation target detection model, and determine the operation status of the target operation task in each monitoring image according to the rotation target frame information of the crane and the rotation target frame information of the boom in each monitoring image, and then calculate the intersection ratio of the rotation target frame information of each operator and the target area when the operation status represents that the boom is in operation, and if the intersection ratio is greater than the preset threshold, it means that the target behavior exists in the target place, that is, the operator is in the target area. In this way, in the embodiment of the present application, the rotation target frame information of each monitored object in each monitoring image is determined by the rotation target detection model, and the operation status of the target operation task can be accurately identified, and then the intersection ratio of the rotation target frame information of each operator and the target area is calculated when the operation status represents that the boom is in operation, and whether there is a target behavior in the target place can be accurately identified. Compared with the manual detection method of the prior art, it can not only reduce the errors of manual detection, but also make a unified judgment by comparing the intersection ratio, thereby improving the accuracy of target behavior detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0011] Figure 1 It is a flowchart of a method for detecting a target behavior provided in an embodiment of the present application; Figure 2 It is a flowchart of an application scenario embodiment provided by the embodiment of the present application; Figure 3 is a detection schematic diagram of a rotating target frame provided in an embodiment of the present application; Figure 4 is a schematic diagram of corner point distribution provided in an embodiment of the present application; Figure 5 is a schematic diagram of the structure of a target behavior detection device provided in an embodiment of the present application; Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0012] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0013] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0014] In the process of industrial construction, various types of lifting equipment are increasingly used in the fields of construction, production and logistics. Especially under complex working conditions, cranes, truck cranes and other equipment are widely used in the installation of large equipment, emergency rescue, cargo transportation and other fields due to their efficient, safe and flexible operation characteristics. However, this is accompanied by various safety hazards caused by the lifting equipment. Among them, people standing under the boom has become one of the main causes of lifting safety accidents. The existing technology is that each production unit supervises by arranging supervisors or video surveillance at the lifting site. However, this kind of manual detection method, on the one hand, is prone to the situation that the supervisor may lose concentration during long-term work and fail to discover safety hazards in time. On the other hand, the supervisor's judgment often relies on personal experience, resulting in the inability to guarantee the accuracy of supervision.

[0015] In order to solve the problems in the prior art, the embodiments of the present application provide a method, device, equipment, storage medium and program product for detecting a target behavior. The following first introduces the method for detecting a target behavior provided by the embodiments of the present application.

[0016] Figure 1 FIG. 1 is a flow chart showing a method for detecting target behavior provided by an embodiment of the present application. Figure 1As shown, a method for detecting a target behavior may include the following steps S101 to S105: S101, obtaining a surveillance video of a target operation task in a target location, where the surveillance video includes a plurality of continuous surveillance images; S102, inputting a plurality of continuous frames of monitoring images into a preset rotating target detection model, obtaining rotating target frame information of each monitoring object in each monitoring image, wherein the monitoring object includes one of a crane, a boom, and at least one operator, and the rotating target detection model is trained based on a plurality of training samples and labeled data of each training sample; S103, determining the operation status of the target operation task in each monitoring image according to the rotating target frame information of the crane and the rotating target frame information of the boom in each monitoring image, where the operation status is used to indicate whether the boom is operating or not operating; S104, when the operation state indicates that the boom is in operation, calculating the intersection ratio of the rotating target frame information of each operator and the target area, where the target area is determined by the rotating target frame information of the crane and the rotating target frame information of the boom; S105. When the intersection ratio is greater than a preset threshold, it is determined that a target behavior exists in the target location, and the target behavior indicates that the operator is in the target area.

[0017] The target behavior detection method of the embodiment of the present application can determine the rotation target frame information of each monitored object in each monitoring image through the rotation target detection model, and determine the operating state of the target operation task in each monitoring image according to the rotation target frame information of the crane and the rotation target frame information of the boom in each monitoring image, and then calculate the intersection ratio of the rotation target frame information of each operator and the target area when the operating state represents that the boom is operating, and if the intersection ratio is greater than the preset threshold, it means that the target behavior exists in the target place, that is, the operator is in the target area. In this way, in the embodiment of the present application, the rotation target frame information of each monitored object in each monitoring image is determined by the rotation target detection model, and the operating state of the target operation task can be accurately identified, and then the intersection ratio of the rotation target frame information of each operator and the target area is calculated when the operating state represents that the boom is operating, and whether there is a target behavior in the target place can be accurately identified. Compared with the manual detection method of the prior art, it can not only reduce the errors of manual detection, but also make a unified judgment by comparing the intersection ratio, thereby improving the accuracy of target behavior detection.

[0018] In S101, the target operation task may be a lifting operation task. The target location may be a lifting operation site. The monitoring video may include multiple continuous monitoring images.

[0019] The above-mentioned acquisition of the monitoring video of the target operation task in the target place may, for example, be acquisition of the monitoring video of the lifting operation task in the lifting operation site by means of a monitoring device, wherein the monitoring device may be a camera.

[0020] In S102, the monitoring object may include one of a crane, a boom and at least one operator.

[0021] The above-mentioned rotated target frame information may include at least one of the coordinates of the center point of the target frame, the width and height of the target frame, multiple category confidences, and the rotation angle of the target frame.

[0022] The above-mentioned rotating object detection model can be trained based on a plurality of training samples and the labeled data of each training sample, wherein the labeled data can include a labeled category and a labeled rotating rectangular box.

[0023] In S103, the above-mentioned operation status can be used to indicate whether the boom is operating or not operating.

[0024] The above-mentioned method determines the working status of the target working task in each monitoring image based on the rotating target frame information of the crane and the rotating target frame information of the boom in each monitoring image. Exemplarily, when the rotating target frame information of the crane and the rotating target frame information of the boom in the monitoring image indicate that the rotation angle difference between the crane and the boom is greater than a preset angle threshold, the working status of the target working task in the monitoring image is determined to be working; when the rotating target frame information of the crane and the rotating target frame information of the boom in the monitoring image indicate that the rotation angle difference between the crane and the boom is less than or equal to the angle threshold, the working status of the target working task in the monitoring image is determined to be not working.

[0025] In S104, the above-mentioned calculation of the intersection ratio of the rotating target frame information of each operator and the target area can be, for example, determined based on the rotating target frame information of the crane and the rotating target frame information of the boom, the target area being the area extending from the minimum circumscribed rectangular frame of the boom to the target midpoint in the rotating target frame of the crane, the target midpoint being the midpoint of the longer side with the larger vertical axis value of the two long sides of the rotating target frame of the crane; the intersection ratio of the rotating target frame information of each operator and the target area is calculated.

[0026] The intersection ratio is the ratio of the intersection area of ​​two sets to the area of ​​the smaller set. In the embodiment of the present application, the intersection ratio can be used to characterize the ratio of the target frame area of ​​each operator to the area of ​​the target region.

[0027] The target area can be determined by the rotating target frame information of the crane and the rotating target frame information of the boom. For example, it can be the area extending from the minimum circumscribed rectangular frame of the boom to the midpoint of the target in the rotating target frame of the crane.

[0028] In S105, the preset threshold, illustratively, may be 60%. Of course, the present application is not limited thereto and may be set according to actual needs of the user, which is not specifically limited here.

[0029] The above target behavior can indicate that the operator is in the target area.

[0030] When the intersection ratio is greater than the preset threshold, it is determined that the target behavior exists in the target location. For example, the target area is compared with the area of ​​the operator's target frame. If the area of ​​the operator's target frame is larger than the target area or smaller than 10% of the target area, it means that the operator and the crane are not in the same depth of field, and the risk of the operator's dangerous operation is ruled out. Similarly, the intersection ratio of each operator and the target area is calculated for the remaining operators.

[0031] In some embodiments, the above S102 may specifically include: Extracting at least one frame of target monitoring image from a plurality of consecutive monitoring image frames at a preset frame number interval; At least one frame of target monitoring image is input into a preset rotating target detection model to obtain the rotating target frame information of each monitoring object in each target monitoring image.

[0032] The above-mentioned step of extracting at least one frame of target monitoring image at a preset frame number interval from the continuous multiple frames of monitoring images may be to extract one frame as the target monitoring image every N frames from the continuous multiple frames of monitoring images, wherein the value of N is determined according to the requirements of the target operation task, and generally the default value is N=1.

[0033] In an embodiment of the present application, at least one frame of target monitoring image is extracted from multiple consecutive frames of monitoring images at a preset frame interval, and then the at least one frame of target monitoring image is input into a preset rotation target detection model to obtain the rotation target frame information of each monitored object in each target monitoring image. This can not only avoid repeated calculations of similar frames and reduce the consumption of computing resources, but also reduce the error accumulation caused by subtle changes between consecutive frames and improve the stability of detection.

[0034] As an implementation of the present application, in order to further improve the accuracy of target behavior detection, before the above S102, the above method may further include: Obtaining a training sample set, the training sample set includes multiple training samples and annotation data of each training sample, the annotation data includes an annotation category and an annotation rotation rectangle; According to multiple training samples and the labeled data of each training sample, a preset neural network model is trained to obtain a rotating target detection model.

[0035] The training sample set may include multiple training samples and annotation data of each training sample, wherein the annotation data may include an annotation category and an annotation rotation rectangle.

[0036] The above-mentioned acquisition of the training sample set, for example, can be to use monitoring equipment to obtain monitoring video of the lifting operation site, filter out images of cranes and operators from the monitoring video as multiple training samples, and then manually annotate the acquired images of cranes and operators with rotating rectangular frames and categories, with the annotated categories being crane, boom, and operator, to obtain the labeled data of each training sample.

[0037] The preset neural network model is trained based on multiple training samples and the labeled data of each training sample to obtain a rotated target detection model. For example, the training sample set can be divided into training samples and verification samples in a ratio of 8:2, and then the preset neural network model is trained by using the training samples and verification samples in the labeled data after horizontally flipping the data for data enhancement to obtain a trained rotated target detection model.

[0038] In an embodiment of the present application, the training sample set includes multiple training samples and annotation data of each training sample, and the annotation data includes annotation categories and annotation rotation rectangular boxes. By training the preset neural network model, a rotating target detection model with accurate detection capabilities can be obtained, so that the monitored object can be described more accurately, and the accuracy of target behavior detection can be further improved.

[0039] In some embodiments, the above S103 may specifically include: When the rotating target frame information of the crane and the rotating target frame information of the boom in the monitoring image indicate that the difference in the rotation angles of the crane and the boom is greater than a preset angle threshold, determining that the operation status of the target operation task in the monitoring image is in operation; When the rotation target frame information of the crane and the rotation target frame information of the boom in the monitoring image indicate that the rotation angle difference between the crane and the boom is less than or equal to the angle threshold, the operation status of the target operation task in the monitoring image is determined to be not in operation.

[0040] The above-mentioned rotation angle difference between the crane and the boom may be a difference between a target frame rotation angle in the rotation target frame information of the crane and a target frame rotation angle in the rotation target frame information of the boom.

[0041] The above preset angle threshold, for example, may be 10 degrees. Of course, the present application is not limited thereto, and may also be set according to the actual situation of the target task, and is not specifically limited here.

[0042] In the embodiment of the present application, since the direction of the boom is consistent with the direction of the crane when the boom is folded and not working, the rotation angle of the target frame is not much different. Therefore, by calculating the difference in the rotation angle of the boom and the crane target frame, it can be determined whether the boom is raised for operation or folded for non-operation, thereby accurately determining the operation status of the target operation task in the monitoring image, thereby improving the accuracy of target behavior detection.

[0043] In some embodiments, the above S104 may specifically include: When the operation state is used to characterize the boom in operation, the target area is determined according to the rotating target frame information of the crane and the rotating target frame information of the boom. The target area is an area extending from the minimum circumscribed rectangular frame of the boom to the target midpoint in the rotating target frame of the crane. The target midpoint is the midpoint of the longer side with the larger longitudinal axis value among the two longer sides of the rotating target frame of the crane. The intersection ratio between the rotation target frame information and the target area of ​​each operator is calculated.

[0044] The target area can be the area extending from the minimum circumscribed rectangular frame of the boom to the target midpoint in the rotating target frame of the crane, that is, the dangerous area under the boom where the heavy lifting task is located. The target midpoint is the midpoint of the longer side with the larger longitudinal axis value of the two long sides of the rotating target frame of the crane.

[0045] The above calculation of the intersection ratio of the rotation target frame information of each operator and the target area may be performed, for example, according to the following calculation formula to calculate the intersection ratio of the rotation target frame information of each operator and the target area. The formula is as follows:

[0046] Among them, A represents the target frame area in the rotation target frame information of each operator, and B represents the area of ​​the target area.

[0047] In an embodiment of the present application, when the operating status is used to characterize the operation of the boom, the target area with safety hazards can be accurately identified based on the rotating target frame information of the crane and the rotating target frame information of the boom, and the intersection ratio of the rotating target frame information of each operator and the target area can be calculated to further improve the accuracy of target behavior detection.

[0048] In some embodiments, determining the target area according to the rotating target frame information of the crane and the rotating target frame information of the boom may specifically include: The rotating target frame information of the crane is converted to obtain the coordinate information of each vertex in the rotating target frame of the crane; The rotating target frame information of the crane arm is converted to obtain the coordinate information of each vertex in the rotating target frame of the crane arm; According to the coordinate information of each vertex in the rotating target frame of the crane, the coordinate information of the target midpoint in the rotating target frame of the crane is calculated; According to the coordinate information of each vertex in the rotating target frame of the crane arm, the coordinate information of the minimum circumscribed rectangular frame of the crane arm is determined; The target area is determined according to the coordinate information of the target midpoint in the rotating target frame of the crane and the coordinate information of the minimum circumscribed rectangular frame of the boom.

[0049] The above-mentioned rotation target frame information may include the coordinates of the target frame center point, the width and height of the target frame, multiple category confidences, and the target frame rotation angle. The format of the rotation target frame information is ,in, They represent the target frame center point coordinates, target frame width and height respectively. Represents the confidence of each category, Corresponding target frame rotation angle.

[0050] The above conversion of the rotating target frame information of the crane obtains the coordinate information of each vertex in the rotating target frame of the crane. For example, Format conversion to ( ) form, where ( ) represents the coordinates of a vertex of the rotation target box. Taking the point in the upper left corner as an example, other vertices are converted in this way. The following are the specific steps: First, calculate the coordinates of the upper left vertex of the target box without rotation:

[0051] right Rotate and get the rotated coordinates The rotation transformation formula is:

[0052]

[0053] Angle needs to be converted to radians. .

[0054] The above conversion of the rotating target frame information of the boom obtains the coordinate information of each vertex in the rotating target frame of the boom. The specific conversion process is the same as the conversion process of the rotating target frame information of the crane. Please refer to the above content and will not be repeated here.

[0055] According to the coordinate information of each vertex in the rotating target frame of the crane, the coordinate information of the target midpoint in the rotating target frame of the crane is calculated. For example, according to the coordinate information of each vertex in the rotating target frame of the crane, the coordinates of the two end points of the longer side with the larger longitudinal axis value among the two longer sides of the rotating target frame of the crane are found, for example , and then calculate the coordinate information of the target midpoint according to the following formula ,formula: .

[0056] According to the coordinate information of each vertex in the rotating target frame of the boom, the coordinate information of the minimum circumscribed rectangular frame of the boom is determined. For example, according to the coordinate information of each vertex in the rotating target frame of the boom ( ), the coordinate information of the minimum circumscribed rectangular frame of the boom can be the coordinate of the upper left corner , the coordinates of the lower right corner are

[0057] In an embodiment of the present application, the rotating target frame information of the crane and the rotating target frame information of the boom are converted to determine the coordinate information of each vertex in the rotating target frame of the crane and the coordinate information of each vertex in the rotating target frame of the boom, and the coordinate information of the midpoint of the target in the rotating target frame of the crane is calculated based on the coordinate information of each vertex in the rotating target frame of the crane, and the coordinate information of the minimum circumscribed rectangular frame of the boom is determined based on the coordinate information of each vertex in the rotating target frame of the boom, and finally, the target area is determined based on the coordinate information of the midpoint of the target in the rotating target frame of the crane and the coordinate information of the minimum circumscribed rectangular frame of the boom, which can significantly improve the accuracy of the target area.

[0058] In order to facilitate the understanding of the target behavior detection method in the embodiment of the present application, the actual application process of the target behavior detection method is described as follows: Step S1: Use monitoring equipment to obtain monitoring video of the lifting operation site ; Step S2: Filter out images containing cranes and operators from the surveillance video ; Step S3: Manually label the acquired images of the crane and the operator with a rotating rectangular frame, and label the categories as crane, boom, and operator to obtain the label data , and divided into training samples according to the ratio of 8:2 and validation samples ; Step S4: The present invention uses the YOLOv8-OBB rotating target detection algorithm model to perform algorithm model training and crane, boom, and personnel detection tasks. and validation samples The neural network model is trained after horizontal flipping data enhancement to obtain a trained neural network model (equivalent to the above-mentioned rotation target detection model); Step S5: Access the video stream (equivalent to the above-mentioned surveillance video) of the surveillance camera at the work site (equivalent to the above-mentioned target location) and use FFmpeg to extract frames, and extract 1 frame as the detection image (equivalent to the above-mentioned target monitoring image); Step S6: Use the trained rotating target detection model to detect the positions of the crane, boom, and operator in the image (equivalent to the rotating target frame information described above).

[0059] Step S7: Obtain the rotation angle of the rotating rectangular frame of the crane and the boom, and determine whether a lifting operation (equivalent to the above-mentioned target operation task) is being performed based on whether the angle difference (equivalent to the above-mentioned rotation angle difference) is greater than a certain threshold.

[0060] Step S8: Calculate the midpoints of the two long sides of the crane rotating rectangular frame, and obtain the midpoint of the long side with the larger Y-axis value (equivalent to the above target midpoint). Obtain the minimum circumscribed rectangular frame of the boom rotating rectangular frame, extend downward from the bottom of the minimum circumscribed rectangular frame to the midpoint, and use the extended area as the danger detection area (equivalent to the above target area).

[0061] Step S9: Obtain a rotated rectangular frame of the person in the image to be detected, and compare the intersection ratio between the danger detection area and each person. If the intersection ratio is greater than a set threshold, it means that the person is in the danger area below the boom.

[0062] like Figure 2 As shown, the embodiment of the present application also provides a specific process of the above steps S5 to S9, the steps are as follows: Step S5: Using monitoring equipment to obtain monitoring video of the lifting operation site ; The video stream of the surveillance camera connected to the work site is extracted using FFmpeg, and one frame is extracted as the detection image every N frames (the value of N is determined according to the inference speed and task requirements of the site, and the default value is N=1) ;because The data transmission is encoded in base64 format, so the PIL library needs to be used to decode the image before passing it into the model.

[0063] Step S6: decode the image The data is input into the trained rotating target detection model to obtain the rotating target frame information of the crane, boom, and personnel. The rotating target frame detection diagram is shown in the figure. Figure 3 The format of the rotation target frame information is , They represent the coordinates of the center point of the bounding box, the width and height of the bounding box respectively. represents the confidence of each category, Corresponding to the rotation angle of the target box.

[0064] Will Format conversion to ( ) form, where ( ) represents the coordinates of a vertex of the rotation target box. The distribution order of the vertices is as follows: Figure 4 Here we take the point in the upper left corner as an example, and the other vertices are converted in this way. The following are the specific steps: First calculate the coordinates of the upper left vertex of the rectangle without rotation:

[0065] right Rotate and get the rotated coordinates The rotation transformation formula is:

[0066]

[0067] The angle here needs to be converted to radians. , the converted target frame result is filtered through the rotated-NMS rotated target frame non-maximum suppression to filter out redundant rotated target frames of the same target, and the rotated target frame with the highest confidence is retained.

[0068] Step S7: When the boom is folded and not working, the direction of the boom is consistent with the direction of the crane, so the rotation angle of the rotation target frame is not much different. Calculate the difference between the rotation angle of the boom and the crane target frame. If it is greater than 10 degrees, it means that the boom is raised and working, and if it is less than 10 degrees, it means that the boom is folded and not working.

[0069] Step S8: Calculate the midpoint of the long side with the larger Y-axis value among the two long sides of the crane rotating rectangular frame, that is, calculate the corner point The midpoint of .

[0070]

[0071] Get the minimum circumscribed rectangle of the boom rotation rectangle. The boundary of the minimum circumscribed rectangle can be determined by finding the maximum and minimum x-coordinates and y-coordinates of the four vertices of the rotation rectangle, for example: Figure 4 The rotated rectangle in the corner point distribution diagram, then the coordinates of the upper left corner of its minimum circumscribed rectangle are , the coordinates of the lower right corner are . Extends from the bottom edge of the minimum bounding rectangle downward to The extended area is used as a danger detection area.

[0072] Step S9: The intersection ratio is the ratio of the intersection area of ​​the two sets to the area of ​​the smaller set. First, compare the area of ​​the danger detection area with the area of ​​the personnel target frame. If the area of ​​the personnel is larger than the danger detection area or smaller than 10% of the danger detection area, it means that the personnel and the crane are not in the same depth of field, and the risk of dangerous operations for the personnel is eliminated. At this time, compare the intersection ratio of each person and the danger detection area among the remaining personnel. After the filtering in the previous step, the area of ​​the personnel is now all smaller than the area of ​​the danger detection area, so what is calculated now is actually how much of the area of ​​the personnel in the danger detection area accounts for.

[0073]

[0074] If the intersection ratio is greater than 60%, it means that most of the person's body is in the danger detection area and the person is in the danger area below the boom; if the intersection ratio is less than 60%, it means that most of the person's body is not in the danger detection area and the person is not in the danger area below the boom.

[0075] Based on the target behavior detection method provided in the above embodiment, the present application also provides a specific implementation of the target behavior detection device. Please refer to the following embodiment.

[0076] like Figure 5 As shown, the target behavior detection device 500 provided in the embodiment of the present application may include the following modules: a first acquisition module 501 , a detection module 502 , a first determination module 503 , a calculation module 504 and a second determination module 505 .

[0077] The first acquisition module 501 is used to acquire a monitoring video of a target operation task in a target location, wherein the monitoring video includes a plurality of continuous monitoring images; A detection module 502 is used to input a plurality of continuous surveillance images into a preset rotating target detection model to obtain rotating target frame information of each surveillance object in each surveillance image, wherein the surveillance object includes one of a crane, a boom and at least one operator, and the rotating target detection model is trained based on a plurality of training samples and labeled data of each training sample; The first determination module 503 is used to determine the operation status of the target operation task in each monitoring image according to the rotation target frame information of the crane and the rotation target frame information of the boom in each monitoring image, where the operation status is used to indicate whether the boom is operating or not operating; A calculation module 504 is used to calculate the intersection ratio of the rotating target frame information of each operator and the target area when the operation state represents that the boom is in operation, and the target area is determined by the rotating target frame information of the crane and the rotating target frame information of the boom; The second determination module 505 is used to determine that there is a target behavior in the target location when the intersection ratio is greater than a preset threshold, and the target behavior indicates that the operator is in the target area.

[0078] The target behavior detection device of the embodiment of the present application can determine the rotation target frame information of each monitored object in each monitoring image through the rotation target detection model, and determine the operation status of the target operation task in each monitoring image according to the rotation target frame information of the crane and the rotation target frame information of the boom in each monitoring image, and then calculate the intersection ratio of the rotation target frame information of each operator and the target area when the operation status represents that the boom is in operation, and if the intersection ratio is greater than the preset threshold, it means that the target behavior exists in the target place, that is, the operator is in the target area. In this way, in the embodiment of the present application, the rotation target frame information of each monitored object in each monitoring image is determined by the rotation target detection model, and the operation status of the target operation task can be accurately identified, and then the intersection ratio of the rotation target frame information of each operator and the target area is calculated when the operation status represents that the boom is in operation, and whether there is a target behavior in the target place can be accurately identified. Compared with the manual detection method of the prior art, it can not only reduce the errors of manual detection, but also make a unified judgment by comparing the intersection ratio, thereby improving the accuracy of target behavior detection.

[0079] In some embodiments, the detection module 502 may specifically include: An extraction unit, used for extracting at least one frame of target monitoring image from a plurality of consecutive frames of monitoring images according to a preset frame number interval; The detection unit is used to input at least one frame of target monitoring image into a preset rotating target detection model to obtain the rotating target frame information of each monitoring object in each target monitoring image.

[0080] As an implementation of the present application, in order to further improve the accuracy of target behavior detection, the above-mentioned device 500 may also include: A second acquisition module is used to acquire a training sample set, where the training sample set includes multiple training samples and annotation data of each training sample, where the annotation data includes an annotation category and an annotation rotation rectangle; The training module is used to train a preset neural network model based on multiple training samples and the labeled data of each training sample to obtain a rotating target detection model.

[0081] In some embodiments, the first determining module 503 may specifically include: A first determining unit is used to determine that the operation status of the target operation task in the monitoring image is in operation when the rotation target frame information of the crane and the rotation target frame information of the boom in the monitoring image indicate that the difference in the rotation angle of the crane and the boom is greater than a preset angle threshold; The second determination unit is used to determine that the operation status of the target operation task in the monitoring image is not in operation when the rotation target frame information of the crane and the rotation target frame information of the boom in the monitoring image indicate that the difference in rotation angles of the crane and the boom is less than or equal to an angle threshold.

[0082] In some embodiments, the calculation module 504 may specifically include: A third determination unit is used to determine a target area according to the rotating target frame information of the crane and the rotating target frame information of the crane when the operation state is used to characterize that the boom is in operation, wherein the target area is an area extending from the minimum circumscribed rectangular frame of the boom to the midpoint of the target in the rotating target frame of the crane, and the midpoint of the target is the midpoint of the long side with the larger longitudinal axis value among the two long sides of the rotating target frame of the crane; The calculation unit is used to calculate the intersection ratio between the rotation target frame information of each operator and the target area.

[0083] In some embodiments, the third determining unit may specifically include: A conversion subunit, used to convert the rotating target frame information of the crane to obtain the coordinate information of each vertex in the rotating target frame of the crane; The conversion subunit is also used to convert the rotation target frame information of the boom to obtain the coordinate information of each vertex in the rotation target frame of the boom; A calculation subunit, used for calculating the coordinate information of the target midpoint in the rotating target frame of the crane according to the coordinate information of each vertex in the rotating target frame of the crane; A first determining subunit is used to determine the coordinate information of the minimum circumscribed rectangular frame of the boom according to the coordinate information of each vertex in the rotating target frame of the boom; The second determination subunit is used to determine the target area according to the coordinate information of the target midpoint in the rotating target frame of the crane and the coordinate information of the minimum circumscribed rectangular frame of the boom.

[0084] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0085] The electronic device may include a processor 601 and a memory 602 storing computer program instructions.

[0086] Specifically, the processor 601 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0087] The memory 602 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 602 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 602 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 602 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 602 is a non-volatile solid-state memory.

[0088] In certain embodiments, the memory 602 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage media device, an optical storage media device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0089] The processor 601 implements any target behavior detection method in the above embodiments by reading and executing computer program instructions stored in the memory 602 .

[0090] In one example, the electronic device may further include a communication interface 603 and a bus 610. Figure 6 As shown, the processor 601, the memory 602, and the communication interface 603 are connected via a bus 610 and communicate with each other.

[0091] The communication interface 603 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0092] Bus 610 includes hardware, software or both, and couples the components of the electronic device to each other. For example, but not limitation, the bus may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industrial standard architecture (EISA) bus, a front-side bus (FSB), a hypertransport (HT) interconnect, an industrial standard architecture (ISA) bus, an infinite bandwidth interconnect, a low pin count (LPC) bus, a memory bus, a microchannel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus or other suitable bus or a combination of two or more of these. Where appropriate, bus 610 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnect.

[0093] The electronic device can execute the target behavior detection method in the embodiment of the present application, thereby realizing the combination Figure 1 and Figure 2 A method and apparatus for detecting target behavior is described.

[0094] In addition, in combination with the target behavior detection method in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any target behavior detection method in the above embodiment is implemented.

[0095] In combination with the target behavior detection method in the above embodiments, an embodiment of the present application may provide a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes any one of the target behavior detection methods above.

[0096] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0097] The functional blocks shown in the structural block diagram described above can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier. "Machine-readable medium" may include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0098] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0099] Aspects of the present disclosure are described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the functions / actions specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0100] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application.

Claims

1. A method for detecting a target behavior, characterized in that: include: Acquire a surveillance video of a target operation task in a target location, wherein the surveillance video includes a plurality of continuous surveillance images; Inputting the continuous multiple frames of monitoring images into a preset rotating target detection model to obtain rotating target frame information of each monitoring object in each monitoring image, wherein the monitoring object includes one of a crane, a boom and at least one operator, and the rotating target detection model is trained based on multiple training samples and labeled data of each training sample; Determine the operation status of the target operation task in each monitoring image according to the rotation target frame information of the crane and the rotation target frame information of the boom in each monitoring image, wherein the operation status is used to indicate whether the boom is operating or not operating; When the operation state represents that the boom is operating, calculating the intersection ratio of the rotating target frame information of each operator and the target area, the target area being determined by the rotating target frame information of the crane and the rotating target frame information of the boom; When the intersection ratio is greater than a preset threshold, it is determined that a target behavior exists in the target location, and the target behavior indicates that the operator is in the target area.

2. The method according to claim 1, characterized in that The step of inputting the continuous multiple frames of monitoring images into a preset rotating target detection model to obtain the rotating target frame information of each monitoring object in each monitoring image includes: Extracting at least one frame of target monitoring image from the continuous multiple frames of monitoring images according to a preset frame number interval; The at least one frame of target monitoring image is input into a preset rotating target detection model to obtain the rotating target frame information of each monitoring object in each of the target monitoring images.

3. The method according to claim 1, characterized in that Before inputting the continuous multiple frames of monitoring images into a preset rotating target detection model to obtain the rotating target frame information of each monitoring object in each monitoring image, the method further includes: Acquire a training sample set, wherein the training sample set includes the plurality of training samples and annotation data of each of the training samples, wherein the annotation data includes an annotation category and an annotation rotation rectangle; According to the multiple training samples and the labeled data of each training sample, a preset neural network model is trained to obtain the rotating target detection model.

4. The method according to claim 1, characterized in that: The step of determining the operation status of the target operation task in each monitoring image according to the rotation target frame information of the crane and the rotation target frame information of the boom in each monitoring image includes: When the rotation target frame information of the crane and the rotation target frame information of the boom in the monitoring image indicate that the rotation angle difference between the crane and the boom is greater than a preset angle threshold, determining that the operation status of the target operation task in the monitoring image is in operation; When the rotation target frame information of the crane and the rotation target frame information of the boom in the monitoring image indicate that the rotation angle difference between the crane and the boom is less than or equal to the angle threshold, it is determined that the operating status of the target operating task in the monitoring image is not in operation.

5. The method according to claim 1, characterized in that When the operation state is used to represent that the boom is operating, calculating the intersection ratio of the rotating target frame information of each operator and the target area includes: In the case where the operation state is used to characterize that the boom is in operation, a target area is determined according to the rotating target frame information of the crane and the rotating target frame information of the boom, wherein the target area is an area extending from the minimum circumscribed rectangular frame of the boom to the target midpoint in the rotating target frame of the crane, and the target midpoint is the midpoint of the longer side with the larger longitudinal axis value among the two longer sides of the rotating target frame of the crane; The intersection ratio between the rotation target frame information of each worker and the target area is calculated.

6. The method according to claim 5, characterized in that The determining of the target area according to the rotating target frame information of the crane and the rotating target frame information of the boom includes: Converting the rotating target frame information of the crane to obtain coordinate information of each vertex in the rotating target frame of the crane; Converting the rotation target frame information of the boom to obtain coordinate information of each vertex in the rotation target frame of the boom; Calculate the coordinate information of the target midpoint in the rotating target frame of the crane according to the coordinate information of each vertex in the rotating target frame of the crane; Determine the coordinate information of the minimum circumscribed rectangular frame of the boom according to the coordinate information of each vertex in the rotating target frame of the boom; The target area is determined according to the coordinate information of the target midpoint in the rotating target frame of the crane and the coordinate information of the minimum circumscribed rectangular frame of the boom.

7. A target behavior detection device, characterized in that: The device comprises: A first acquisition module is used to acquire a monitoring video of a target operation task in a target location, wherein the monitoring video includes a continuous plurality of monitoring frames; A detection module, used for inputting the continuous multiple frames of monitoring images into a preset rotating target detection model to obtain rotating target frame information of each monitoring object in each monitoring image, wherein the monitoring object includes one of a crane, a boom and at least one operator, and the rotating target detection model is trained based on multiple training samples and labeled data of each training sample; A first determination module is used to determine the operation status of the target operation task in each monitoring image according to the rotation target frame information of the crane and the rotation target frame information of the boom in each monitoring image, wherein the operation status is used to indicate whether the boom is operating or not operating; a calculation module, for calculating, when the operation state represents that the boom is operating, an intersection ratio of the rotation target frame information of each operator and a target area, wherein the target area is determined by the rotation target frame information of the crane and the rotation target frame information of the boom; The second determination module is used to determine that there is a target behavior in the target place when the intersection ratio is greater than a preset threshold, and the target behavior indicates that the operator is in the target area.

8. An electronic device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method for detecting a target behavior as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the method for detecting a target behavior as described in any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the target behavior detection method as described in any one of claims 1-6.

Images