An image detection method, device, and electronic device

Abstract

The application relates to the computer technical field, in particular to an image detection method and device and electronic equipment. In the method, an image to be detected is acquired. Target detection is performed on the image to be detected, target regions and target region information of the targets in the image to be detected are obtained, the target region information comprises a category label and target region position information, and the category label represents the category of the target. In a set of set rule sets, a first rule set related to the target is determined according to the category label. According to the target region position information, association information between different target regions is determined. In the case that the association information meets a preset condition of a target rule, the target rule is determined. The above scheme detects whether the target in the image to be detected violates the rule through the association information between different target regions, compared with the behavior analysis of only the behavior result of a single target in the prior art, the detection efficiency and accuracy of the image are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to an image detection method and device and electronic equipment. BACKGROUND

[0002] At present, after obtaining a to-be-detected image, a traditional manual auditing method is usually used to audit whether the to-be-detected image is qualified. However, the above method has the problems of low auditing efficiency and low accuracy rate due to large auditing workload.

[0003] How to improve the detection efficiency and accuracy rate of the detected image is a problem worth discussing. SUMMARY

[0004] Embodiments of the present application provide an image detection method and device, electronic equipment and storage medium, which are used to improve the detection efficiency and accuracy rate of the detected image.

[0005] In a first aspect, an object detection method is provided, including: obtaining a to-be-detected image; performing object detection on the to-be-detected image to obtain a target region and target region information of a target in the to-be-detected image. The target region information includes a category label and target region position information, and the category label represents a category of the target. A first violation rule set related to the target is determined from a set of violation rules according to the category label, and the set of violation rules includes a corresponding relationship between a violation rule and a category label. Association information between different target regions is determined according to the target region position information. The association information includes an intersection-over-union and a center point distance of different targets in different target regions, and the intersection-over-union is used to represent a ratio of an intersection to a union between bounding boxes corresponding to different target regions. In a case where the association information meets a preset condition of a target violation rule, a target violation is determined. The target violation rule is any violation rule in the first violation rule set.

[0006] In the above method, by analyzing the association relationship between different target regions in the to-be-detected image, it is determined whether the target violates the rule using the association relationship. Compared with the behavior analysis in the prior art which is only directed to the behavior result of a single target, the accuracy rate of image detection can be improved. Compared with manual detection of images, the efficiency and accuracy rate of image detection can be improved.

[0007] Optionally, the target detection is performed on the to-be-detected image to obtain a target region and target region information of each element in the to-be-detected image, and the target detection specifically includes:

[0008] The target detection is performed on the to-be-detected image based on a pre-established target detection model to obtain a target region and target region information of a target in the to-be-detected image.

[0009] In the above method, the target detection model can be pre-established to more accurately extract the target region of the target and obtain target region information, facilitating subsequent determination of whether the target violates the rules based on the target region and the target region information.

[0010] Optionally, the target detection model is pre-established in the following manner:

[0011] Based on historical detection images, training samples are determined, including target violation historical detection images and target non-violation historical detection images;

[0012] Using the training samples, the yolov5 network model is trained based on the Pytorch deep learning framework to obtain the target detection model.

[0013] In the above method, the data volume of the training samples required for training the yolov5 network model based on the Pytorch deep learning framework is low, reducing the difficulty of target extraction. At the same time, after continuously adjusting the learning rate (LR) and batch size, the MIOU (Mean Intersection over Union, the evaluation value of the current model) can be approximately 99%. That is, the above target detection model can more accurately extract the target region of the target and obtain target region information, facilitating subsequent determination of whether the target violates the rules based on the target region and the target region information.

[0014] Optionally, after performing target detection on the to-be-detected image to obtain the target region and target region information of each element in the to-be-detected image, the method further includes:

[0015] Comparing the target region information with the set target standard;

[0016] In the case where the target region information does not meet the target standard, deleting the to-be-detected image;

[0017] In the case where the target region information meets the target standard, determining a first violation rule set related to the target according to the category label in the set violation rule set.

[0018] In the above method, by comparing the target region information with the set target standard and deleting the to-be-detected image that does not meet the target standard, the efficiency of image detection can be improved. At the same time, since the to-be-detected image that meets the standard is detected in the subsequent image detection process, the efficiency and accuracy of image detection can be improved.

[0019] Optionally, in the case where the association information meets the preset condition of the target violation rule, it is determined that the target violates the rules, specifically including:

[0020] determining a preset IoU threshold and a distance threshold in the target violation rule as the target;

[0021] comparing the IoU with the IoU threshold and comparing the center point distance with the distance threshold;

[0022] In the case that the IoU meets the preset condition and / or the center point distance meets the preset condition, the target violation is determined.

[0023] In the above method, compared with the behavior analysis of the behavior result of a single target in the prior art, the behavior correlation analysis between multiple targets in the to-be-detected image is performed to improve the accuracy of image detection. In subsequent optimization of the image detection method, compared with the need to update the algorithm and logic in the prior art, the present application only needs to update the set of violation rules.

[0024] Optionally, the above method further comprises:

[0025] In the case that the correlation information does not meet the preset condition of the target violation rule, the next violation rule in the first set of violation rules is taken as a new target violation rule, and the comparison of the IoU with the new IoU threshold and the comparison of the center point distance with the new distance threshold are returned to be performed until the correlation information meets the preset condition of the target violation rule, or all violation rules in the first set of violation rules are determined as target violation rules.

[0026] In the above method, the violation rule related to the target in the first set of violation rules is taken as the target violation rule, so as to determine whether the target violates the rule, which facilitates more rapid detection of whether the target in the to-be-detected image violates the rule.

[0027] Optionally, in the case that the correlation information meets the preset condition of the target violation rule, after the target violation is determined, the method further comprises:

[0028] generating an alarm information according to the target violation rule.

[0029] In the above method, the alarm information is generated according to the target violation rule, which facilitates subsequent analysis of the violation behavior appearing in the to-be-detected image and timely prevention of the occurrence of the violation behavior.

[0030] In a second aspect, an embodiment of the present application provides an image detection device, comprising:

[0031] an acquisition module configured to acquire a to-be-detected image, the to-be-detected image comprising a target object and a background object;

[0032] The processing module is configured to perform image segmentation on the image to be detected to extract a target contour of a target object.

[0033] The processing module is further configured to determine a minimum enclosing rectangle of the target contour.

[0034] The processing module is further configured to divide the target contour according to a preset condition to obtain a plurality of sub-target contours.

[0035] The detection module is configured to determine that the target object passes the detection when a first contour line of the plurality of sub-target contours and a length of the minimum enclosing rectangle meet a first preset rule and a second contour line of the plurality of sub-target contours meets a second preset rule, the first contour line being a length of a line segment between two sub-target contours, and the second contour line being a length of a line segment in a sub-target contour that is in the same direction as a width of the minimum enclosing rectangle.

[0036] In a third aspect, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable by the processor, when the computer program is executed by the processor, the processor implements any of the image detection methods in the first aspect.

[0037] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, when the computer program is executed by a processor, the image detection method in the first aspect is implemented.

[0038] In a fifth aspect, a computer program product is provided, including a computer program, when the computer program is executed by a processor, the image detection method in any of the first aspect is implemented.

[0039] The technical effects brought by any of the implementation manners of the second aspect to the fifth aspect can be referred to the technical effects brought by the corresponding implementation manners of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 An application scenario of an image detection method provided by the embodiments of the present application is shown in the figure.

[0041] Figure 2 A flowchart of an image detection method provided by the embodiments of the present application is shown in the figure.

[0042] Figure 3 An illustration of an intersection over union provided by the embodiments of the present application is shown in the figure.

[0043] Figure 4 An exemplary flowchart of an image detection method provided by the embodiments of the present application is shown in the figure.

[0044] Figure 5A device schematic diagram for image detection provided by an embodiment of the present application is shown in FIG. 1.

[0045] Figure 6 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0046] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0047] The application scenarios described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those of ordinary skill in the art can know that, with the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems. In the description of the present application, unless otherwise specified, the meaning of "multiple" is two or more.

[0048] At present, after obtaining the to-be-detected image, a traditional manual auditing method is usually used to audit whether the to-be-detected image is qualified. However, the above method has the problems of large auditing workload, low auditing efficiency and low accuracy. How to improve the detection efficiency and accuracy of the detected image is a problem worth discussing.

[0049] For example, in the prior art, in the detection and monitoring scene of dangerous operation of a compressed natural gas (CNG) filling station, the current method uses separate target detection analysis processing. For example, a target detection is performed on the behavior of whether a safety helmet is worn, and a behavior analysis is directly performed according to the result after algorithm processing. This kind of method needs to perform a check model and post-processing on multiple behaviors in the CNG area, greatly increasing the detection resource occupation, and there is no behavior correlation and rule analysis between multiple targets, so the accuracy is low.

[0050] To solve the above problems, the embodiment of the present application provides an image detection method, device and electronic equipment. For example, a to-be-detected image is acquired. Target detection is performed on the to-be-detected image to obtain a target region of a target in the to-be-detected image and target region information. The target region information includes a category label and target region position information, and the category label represents the category of the target. In a set of set rule sets, a first rule set related to the target is determined according to the category label, and the rule set includes a corresponding relationship between a rule and a category label. According to the target region position information, association information between different target regions is determined, and the association information includes an intersection-over-union and a center point distance of different targets in different target regions. The intersection-over-union is used to represent the ratio of the intersection to the union between the bounding boxes corresponding to different target regions. In the case where the association information meets a preset condition of a target rule, the target rule is determined. The target rule is any rule in the first rule set.

[0051] As shown in Figure 1 An application scenario diagram of an optional image detection method of the embodiment of the present application includes a server 103 and an image acquisition device 101. The server 103 and the image acquisition device 101 can be communicatively connected through a network 102 to implement the image detection method of the present application.

[0052] In a possible case, the image acquisition device 101 can perform real-time monitoring and send the real-time monitoring to the server 103. The server 103 performs frame extraction processing on the real-time monitoring video to obtain a plurality of to-be-detected images. In another possible case, the image acquisition device 101 can also shoot to-be-detected images. The image acquisition device 101 sends the to-be-detected images to the server 103. The server 103 acquires a plurality of to-be-detected images.

[0053] After the server 103 acquires the to-be-detected images, target detection is performed on the to-be-detected images to obtain a target region of a target in the to-be-detected image and target region information. The target region information includes a category label and target region position information, and the category label represents the category of the target. In a set of set rule sets, a first rule set related to the target is determined according to the category label, and the rule set includes a corresponding relationship between a rule and a category label. According to the target region position information, association information between different target regions is determined, and the association information includes an intersection-over-union and a center point distance of different targets in different target regions. The intersection-over-union is used to represent the ratio of the intersection to the union between the bounding boxes corresponding to different target regions. In the case where the association information meets a preset condition of a target rule, the target rule is determined. The target rule is any rule in the first rule set.

[0054] As shown in Figure 2As shown, the image detection method provided in the embodiment of the present application can specifically include the following steps.

[0055] S201, obtaining a to-be-detected image.

[0056] In a possible case, the server can perform frame extraction processing on the real-time monitoring video to obtain a plurality of to-be-detected images. In another possible case, the server can also obtain the to-be-detected image from the database. In another possible case, the server can also receive the to-be-detected image from the terminal.

[0057] S202, performing target detection on the to-be-detected image to obtain a target region of the target in the to-be-detected image and target region information.

[0058] The target region information includes a category label and target region position information. The category label represents the category of the target.

[0059] For example, the category label can include categories such as gas filling line, safety helmet, person, vehicle, work clothes, etc. The target region position information can include the coordinates of the target in the to-be-detected image.

[0060] Optionally, the target region information can also include the size ratio of the target in the target region in the to-be-detected image and the confidence score. The confidence score is used to represent the probability of whether the target region is the target.

[0061] In a possible embodiment, the server can perform target detection on the to-be-detected image based on a pre-trained target detection model to obtain the target region of the target in the to-be-detected image and the target region information. For example, after performing target detection on the to-be-detected image, the obtained target region can be a target rectangular frame.

[0062] The target detection model in the embodiment of the present application can be trained in the following way:

[0063] Determine training samples based on historical detection images. The training samples include historical detection images of target violations and historical detection images of targets not violating rules. Train the yolov5 network model based on the Pytorch deep learning framework using the above training samples to obtain the target detection model.

[0064] Optionally, the server can also perform frame extraction processing on the historical monitoring video to obtain a plurality of historical detection images. For example, the server can perform frame extraction processing on the historical monitoring video to obtain a plurality of historical detection images. Train the yolov5 network model based on the Pytorch deep learning framework using 1500 historical detection images not containing violating targets and 1500 historical detection images containing violating targets to obtain the target detection model.

[0065] It can be understood that in the embodiments of the present application, the number of samples and training samples when training the target detection model can be pre-set by those skilled in the art, and the number of samples and training samples can also be changed according to the specific application scenario.

[0066] In the above method, the data amount of the training sample quantity required for training the yolov5 network model based on the Pytorch deep learning framework is low, which reduces the difficulty of target extraction. At the same time, after continuously adjusting the learning rate (learning rate, LR) and batch size, the MIOU (Mean Intersection over Union, evaluation value of the current model) obtained can be about 99%. That is, the above target detection model can more accurately extract the target region of the target, obtain the target region information, and facilitate subsequent determination of whether the target is in violation according to the target region and the target region information.

[0067] In an optional embodiment, in order to improve the accuracy and efficiency of image detection, the server can determine whether the to-be-detected image meets the set target standard according to the target region information after obtaining the target region information. In the case that the target region information does not meet the target standard, the to-be-detected image that does not meet the target standard is deleted. In the case that the target region information meets the target standard, the first violation rule set related to the target is determined according to the category label in the set violation rule set.

[0068] It can be understood that there can be one or more targets in the to-be-detected image, and in the case that the target region information of any target in the to-be-detected image does not meet the target standard, the server can delete the to-be-detected image. The target standard in the embodiments of the present application can be pre-set by those skilled in the art, and the target standard can also be changed according to the specific application scenario. For example, the target standard includes whether the confidence score of the target in the target region should be greater than or equal to 90. For another example, the target standard includes that the target size of the label category of a person in the to-be-detected image should be less than 0.01%.

[0069] The following illustrates how to determine whether the target in the to-be-detected image meets the target standard according to the target region information:

[0070] It is assumed that the to-be-detected image 1 only contains the target 1. The confidence score of the target 1 is 95. The target standard includes: 1, whether the confidence score in the target rectangular frame is greater than or equal to 90. 2, the size of the target of the label category in the to-be-detected image should be less than 0.01%. Since the confidence score of the target 1 is 95, which is greater than 90, the confidence score of the target 1 meets the target standard. The label category of the target 1 is a person, and the size of the target 1 in the to-be-detected image is 0.2%. Since the size of the target 1 in the to-be-detected image is 0.2%, which is greater than 0.01%, the target 1 does not meet the size of the target 1 in the to-be-detected image. The server can delete the to-be-detected image 1. For another example, it is assumed that the to-be-detected image 2 contains the target 2 and the target 3. Among them, the target 2 meets the confidence score and the size of the target of the label category in the target standard. The label category of the target 3 is a person, and the size of the target 3 in the to-be-detected image is 0.15%. The label category in the target standard includes: the size of the target of the label category in the to-be-detected image should be less than 0.01%. Since 0.15% is greater than 0.01%, the target 3 does not meet the size of the target 3 in the to-be-detected image. The server can delete the to-be-detected image 2.

[0071] Optionally, in order to improve the accuracy of the subsequent detection image, the server can adjust the to-be-detected image. For example, the server can adjust the parameters, positions and the like of the to-be-detected image. For example, the server can perform steps of enhancing the clarity of the to-be-detected image, adjusting the angle of the to-be-detected image, increasing the contrast of the to-be-detected image, and the like. It can be understood that the above steps of adjusting the to-be-detected image in the embodiments of the present application can be pre-set by those skilled in the art, and the above steps of adjusting the to-be-detected image can be reasonably set according to the specific application scene. The present application does not make specific limitation.

[0072] S203, determining a first set of violation rules related to the target according to the category label in the set of set violation rules.

[0073] Among them, the set of violation rules contains the corresponding relationship between the violation rules and the category label.

[0074] In an optional embodiment, the corresponding relationship between the category label and the violation rule can be one-to-one. The corresponding relationship between the category label and the violation rule can also be that one category label corresponds to multiple violation rules. It can be understood that the set of violation rules in the embodiments of the present application can be pre-set by those skilled in the art, and the above set of violation rules can be reasonably set according to the specific application scene. The present application does not make specific limitation.

[0075] For example, assuming that the category labels of the target in the image to be detected include a safety helmet, a person, and a gas line. The server can determine, according to the category label “safety helmet”, that the first set of violation rules related to the category label “safety helmet” includes: 1, the worker does not wear a safety helmet. The server can also determine, according to the category label “person”, that the first set of violation rules related to the category label “person” includes: 1, the worker does not wear a work uniform correctly. 2, a non-worker crosses the gas line. The server can also determine, according to the category label “gas line”, that the first set of violation rules related to the category label “gas line” includes: 1, a non-worker crosses the gas line.

[0076] S204, determining association information between different target regions according to the target region position information.

[0077] The association information includes an intersection-over-union and a center point distance of different targets in different target regions, and the intersection-over-union is used to represent a ratio of an intersection to a union between the bounding boxes corresponding to the different target regions.

[0078] In a possible case, the server can determine the center point distance of different targets in different target regions according to the target regions of different targets in the image to be detected. It can be understood that the present application does not limit the manner of determining the distance between different target regions. For example, the center point distance matching algorithm is used to determine the distance between the target region and the center point of other target regions in the image to be detected.

[0079] In another possible case, the server can also determine the intersection-over-union (IoU) between two target regions. The IoU is the overlap rate of the generated target region 1 and the target region 2, that is, the ratio of their intersection to union. When the target region 1 and the target region 2 completely overlap, the IoU is 1. As shown in FIG. 2, the intersection-over-union is IOU=(A∩B) / (A∪B)=A and B intersection / A and B union. Figure 3

[0080] In the above method, the association relationship between different targets is determined through the intersection-over-union and the center point distance between different target regions. In subsequent detection of whether the target violates the rules, the above association relationship can be directly compared with the target violation rule to determine whether the target violates the rules.

[0081] S205, determining that the target violates the rules in a case where the association information meets a preset condition of the target violation rule.

[0082] The target violation rule is any violation rule in the first set of violation rules.

[0083] ​In a possible embodiment, the server can take any rule in the first rule set as a target rule, and determine a preset IoU threshold and distance threshold in the target rule. The IoU is compared with the IoU threshold, and the center point distance is compared with the distance threshold. In the case that the IoU meets a preset condition and / or the center point distance meets a preset condition, the target rule is determined.

[0084] It can be understood that the rule in the application can include a plurality of sub-rules. For example, the rule "non-staff crossing gas line": sub-rule 1 needs to be met, the target with a category label of "person" is non-staff. 2, make a crossing action in the gas line area.

[0085] In the above method, compared with the behavior analysis of the behavior result of a single target in the prior art, the application improves the accuracy of image detection by performing behavior correlation analysis between a plurality of targets in the to-be-detected image. In subsequent optimization of the image detection method, compared with the need to update the algorithm and logic in the prior art, the application only needs to update the rule set.

[0086] For example, assuming that the server detects the to-be-detected image and determines that the categories of the targets in the to-be-detected image include people, gas lines, cars, safety helmets, and work clothes. The server can determine the associated information, including: the IoU between the target rectangular frame 1 where the "person" is located and the target rectangular frame 2 where the "work clothes" is located is 0%, the IoU between the target rectangular frame 1 where the "person" is located and the target rectangular frame 3 where the "gas line" is located is 40%, and the center point distance between the target rectangular frame 1 where the "person" is located and the target rectangular frame 3 where the "gas line" is located is 0.08 cm.

[0087] The server can determine that, in the rule set, the first rule set related to the category label "person" includes: 1, the worker does not wear safety clothes correctly; 2, the non-worker makes a crossing action in the gas line area. Specifically, the server determines that the preset condition for the target with the category label "person" being a worker includes: the IoU of the target area where the target with the category label "person" is located and the target area where the target with the category label "work clothes" is located is greater than or equal to 20%. The server determines that the preset condition for the worker not wearing safety clothes correctly includes: the IoU of the target area where the target with the category label "person" is located and the target area where the target with the category label "work clothes" is located is greater than or equal to 85%. The server determines that the preset condition for the worker not wearing safety clothes correctly includes: the IoU of the target area where the target with the category label "person" is located and the target area where the target with the category label "work clothes" is located is greater than or equal to 85%. The server determines that the preset condition for the non-worker making a crossing action in the gas line includes: the IoU of the target area where the non-worker is located and the target area where the gas line is located is greater than or equal to 30%, and the center point distance between the target area where the non-worker is located and the target area where the gas line is located is less than or equal to 0.1 cm.

[0088] Then the server can take "the worker does not wear safety clothes correctly" as a target rule, and determine whether the target meets the preset condition of the target rule. Through the association information, it is determined that the target rectangle frame 1 where the above-mentioned "person" is located does not meet the preset condition of determining that the target with the category label "person" is a worker, and the target is a non-worker. The server can modify the above-mentioned target with the category label "person" to a target with the category label "non-worker". The server determines a sub-rule set related to the category label "non-worker" in the first rule set. The server can take "the non-worker makes a crossing action in the gas line area" as a target rule. Because the IoU and the center point distance between the target rectangle frame 1 and the target rectangle frame 3 meet the above-mentioned preset condition of determining that the non-worker makes a crossing action in the gas line, the server can determine that the target rule is violated. The rule violation in the to-be-detected image includes the non-worker making a crossing action in the gas line area.

[0089] In a possible embodiment, in the case that the association information does not meet the preset condition of the target rule, the next rule in the first rule set is taken as a new target rule, and the IoU is compared with the new IoU threshold, and the center point distance is compared with the new distance threshold, until the association information meets the preset condition of the target rule, or all rules in the first rule set are determined as target rules.

[0090] For example, assuming that the target class label in the image to be detected includes "person". The server can determine, according to "person", that the first rule set of violations related to the target "person" includes: 1, the staff does not wear the uniform correctly. 2, the non-staff crosses the gas line. After determining that the target meets "the staff does not wear the uniform correctly", the server can stop image detection and determine that the target violates the rule. For another example, assuming that all violations included in all targets in the image to be detected need to be detected. Then, after detecting that the target meets "the staff does not wear the uniform correctly", the server can continue to detect whether the target with the class label "person" meets "the non-staff crosses the gas line". And detect whether the target with other class labels violates the rule until all violation rules in the first rule set of violations are determined as target violation rules.

[0091] In the above method, by taking the violation rules in the first rule set of violations related to the target as the target violation rules, it is determined whether the target violates the rule, which facilitates rapid detection of the violation behavior included in the target in the image to be detected.

[0092] Optionally, the server can also determine the intersection-over-union threshold and the distance threshold included in each violation rule in the first rule set of violations. The server can compare the intersection-over-union between the target regions where different targets are located with the multiple intersection-over-union thresholds in the first rule set of violations in parallel. The multiple intersection-over-union thresholds include the intersection-over-union threshold included in the violation rule related to the target in the first rule set of violations. The server can also compare the center point distance between the target regions where different targets are located with the multiple distance thresholds in the first rule set of violations in parallel. The multiple distance thresholds include the distance threshold included in the violation rule related to the target in the first rule set of violations. In the case where the server can determine that the intersection-over-union meets the preset condition with the intersection-over-union threshold, and / or the center point distance meets the preset condition with the distance threshold, one or more violation rules, the target is determined to violate the rule.

[0093] For example, assuming that the server detects the image to be detected and determines that the class of the target in the image to be detected includes person, safety helmet, and uniform. The server can determine the associated information to include: the IoU of the target rectangular frame 1 where the "person" is located and the target rectangular frame 2 where the "uniform" is located is 0%, and the IoU of the target rectangular frame 1 where the "person" is located and the target rectangular frame 4 where the "safety helmet" is located is 95%.

[0094] The server can determine that, in the rule violation set, the first rule violation set related to the category label "person" includes: 1, the worker does not wear safety clothes correctly; 2, the worker does not wear a safety helmet correctly. Specifically, the server determines that the preset condition for the target with the category label "person" being a worker includes: the IoU of the target area where the target with the category label "person" is located and the target area where the category label "safety clothes" is located is greater than or equal to 20%. The server determines that the preset condition for the worker not wearing safety clothes correctly includes: the IoU of the target area where the target with the category label "person" is located and the target area where the category label "safety clothes" is located is less than or equal to 85%. The server determines that the preset condition for the worker not wearing a safety helmet correctly includes: the IoU of the target area where the target with the category label "person" is located and the target area where the category label "safety helmet" is located is less than or equal to 90%.

[0095] Then the server can compare the IoU of the target rectangular frame 1 where the "person" is located and the target rectangular frame 2 where the "safety clothes" is located, which is 0%, with the IoU threshold value 20%. The IoU of the target rectangular frame 1 where the "person" is located and the target rectangular frame 2 where the "safety clothes" is located is 0%, which is compared with the IoU threshold value 85%. The IoU of the target rectangular frame 1 where the "person" is located and the target rectangular frame 4 where the "safety helmet" is located is 95%, which is compared with another IoU threshold value 90%. The server can determine that the preset condition met by the target rectangular frame 1 where the "person" is located is that the IoU of the target area where the target with the category label "person" is located and the target area where the category label "safety clothes" is located is less than or equal to 85% and greater than or equal to 20%. The target rule violation meets the preset condition that the target with the category label "person" is a worker and the preset condition that the worker does not wear safety clothes correctly.

[0096] Optionally, after determining the to-be-detected image, the server can also generate an alarm information according to the rule violation behavior detected in the to-be-detected image. The server sends the alarm information to the user. For example, assuming that the rule violation behavior detected in the to-be-detected image includes: a non-worker making a crossing action in the gas line area, the server can generate an alarm information including the time point where the to-be-detected image is located and the rule violation behavior included in the to-be-detected image.

[0097] It can be understood that the application does not make specific limitations on how to send the alarm information. For example, the alarm information can be sent in the form of a short message. For another example, the alarm information can be sent in the form of an email.

[0098] In the above method, the alarm information is generated according to the target rule violation. This facilitates subsequent analysis of the rule violation behavior appearing in the to-be-detected image and timely prevention of the occurrence of the rule violation behavior.

[0099] Optionally, the server can arrange the alarm information in the preset time period to obtain an alarm information set. After obtaining the alarm information set, the server can send the alarm information set to the user.

[0100] For example, the server can arrange the alarm information in the preset time period according to different violation rules to obtain an alarm information set. After obtaining the alarm information set, the server can arrange the alarm information in the preset time period according to the category label of the violation target to obtain an alarm information set.

[0101] In the above method, the alarm information in the preset time period is arranged to generate an alarm information set. The violation behavior appearing in the to-be-detected image is analyzed to prevent the appearance of the violation behavior in a timely manner.

[0102] It can be understood that the preset time period in the embodiments of the present application can be set in advance by a person skilled in the art, and the preset time period can be reasonably set according to a specific application scenario. The form of the alarm information set in the embodiments of the present application is not limited. For example, the alarm information set can be displayed in the form of a table. For another example, the alarm information set can be displayed in the form of a document.

[0103] For example, as shown in Table 1, the embodiments of the present application provide a schematic diagram of an alarm information set. It is assumed that the preset time period is 2023 / 3 / 9. Among them, the number of times of the violation behavior: non-staff making a crossing action in the gas filling line area is 3 times. The appearing times are 2023 / 03 / 09 09:52:35, 2023 / 03 / 09 13:02:35 and 2023 / 03 / 09 14:42:03. The number of times of the violation behavior: abnormal wearing of work clothes is 1 time. The appearing times are 2023 / 03 / 09 07:02:50.

[0104]

[0105] Table 1

[0106] In the above method, by arranging the alarm information in the preset time period, the user can analyze the alarm information in a subsequent manner, and the appearance of the violation behavior can be prevented in a timely manner.

[0107] The embodiments of the present application are described below: Figure 2

[0108] ​Assuming that the server can frame the real-time monitoring video, get multiple images to be detected. The server can detect multiple images to be detected based on the pre-trained target detection model, and get the target area and target area information of each target in the image to be detected. Assume that the image to be detected 1 only contains target 1. The confidence score of target 1 is 95. The target standard includes: 1, whether the confidence score of the target rectangular box is greater than or equal to 90. 2, the label category is a person, and the size of the target in the image to be detected should be less than 0.01%. Since the confidence score of target 1 is 95, which is greater than 90. The confidence score of target 1 meets the target standard. The label category of target 1 is a person, and the size of target 1 in the image to be detected is 0.2%. Since the size of target 1 in the image to be detected is 0.2%, which is greater than 0.01%. Therefore, target 1 does not meet the size of the person. The server can delete the image to be detected 1.

[0109] Assuming that all targets in the image to be detected 2 meet the set target standard. The server can determine a first set of violation rules related to the target according to the category label of each target in the image to be detected 2 in the set of violation rules. The first set of violation rules related to the category label "person" includes: 1, the worker does not wear safety clothes correctly. 2, non-workers make crossing actions in the gas line area. Specifically, the server determines the preset condition that the target with the category label "person" is a worker includes: the IoU of the target area where the target with the category label "person" is located and the target area where the category label "work clothes" is located is greater than or equal to 20%. The server determines the preset condition that the worker does not wear safety clothes correctly includes: the IoU of the target area where the target with the category label "person" is located and the target area where the category label "work clothes" is located is greater than or equal to 85%. The server determines the preset condition that non-workers make crossing actions in the gas line includes: the IoU of the target area where the non-worker is located and the target area where the gas line is located is greater than or equal to 30%, and the center point distance between the target area where the non-worker is located and the target area where the gas line is located is less than or equal to 0.1 centimeters, then it can be determined that the non-worker crosses the gas line.

[0110] The server can determine the associated information, including: the IoU of the target rectangular box 1 where the "person" is located and the target rectangular box 2 where the "work clothes" is located is 0%, the IoU of the target rectangular box 1 where the "person" is located and the target rectangular box 3 where the "gas line" is located is 40%, and the center point distance between the target rectangular box 1 where the "person" is located and the target rectangular box 3 where the "gas line" is located is 0.08 centimeters.

[0111] The server can take "worker not wearing safety clothes correctly" as a target violation rule, and determine whether the target meets the preset condition of the target violation rule. Through the association information, it is determined that the target rectangular frame 1 where the "person" is located does not meet the preset condition of determining the target as a worker with the category label "person", and the target is a non-worker. The server can modify the target with the category label "person" to the target with the category label "non-worker". The server determines a sub-violation rule set related to the category label "non-worker" in the first violation rule set. The server can take "non-worker making crossing action in the gas line area" as a target violation rule. Since the IoU and the center point distance between the target rectangular frame 1 and the target rectangular frame 3 meet the preset condition of determining that the non-worker makes the crossing action in the gas line, the server can determine the target violation. The violation behavior in the to-be-detected image includes the non-worker making the crossing action in the gas line area.

[0112] As shown in Figure 4 , the present application provides an exemplary image detection flowchart. The steps include:

[0113] S401, acquiring a to-be-detected image;

[0114] S402, performing target detection on the to-be-detected image based on a pre-established target detection model to obtain a target region and target region information of the target in the to-be-detected image;

[0115] S403, comparing the target region information with a set target standard to determine whether the to-be-detected image meets the target standard, if yes, performing S405, and if no, performing S404;

[0116] S404, deleting the to-be-detected image;

[0117] S405, determining a first violation rule set related to the target in a set violation rule set according to the category label;

[0118] S406, determining association information between different target regions according to the target region position information;

[0119] S407, taking any violation rule in the first violation rule set as a target violation rule, and determining a preset intersection over union threshold and distance threshold in the target violation rule;

[0120] S408, comparing the intersection over union with the intersection over union threshold, and comparing the center point distance with the distance threshold;

[0121] S409, determining the target violation in the case that the intersection over union is less than or equal to the intersection over union threshold, and the center point distance is less than or equal to the distance threshold;

[0122] S410, generate an alarm information according to the target violation rule;

[0123] S411, send the alarm information to the user.

[0124] Figure 5 A structural schematic diagram of an image detection device provided by an embodiment of the present application is shown in the figure, which includes a transceiver module 501, a detection module 502, and a determination module 503. Figure 5

[0125] The transceiver module 501 is configured to acquire a to-be-detected image.

[0126] The detection module 502 is configured to perform target detection on the to-be-detected image to obtain a target region of a target in the to-be-detected image and target region information, the target region information including a category label and target region position information, the category label representing a category of the target.

[0127] The determination module 503 is configured to determine, according to the category label, a first violation rule set related to the target from a set of violation rules set, the set of violation rules set including a corresponding relationship between a violation rule and a category label.

[0128] The determination module 503 is further configured to determine, according to the target region position information, association information between different target regions, the association information including an intersection-over-union and a center point distance of different targets in different target regions, the intersection-over-union being used to represent a ratio of an intersection to a union between the bounding boxes corresponding to the different target regions.

[0129] The determination module 503 is further configured to determine a target violation in a case where the association information meets a preset condition of a target violation rule, the target violation rule being any violation rule in the first violation rule set.

[0130] Optionally, the target detection is performed on the to-be-detected image to obtain a target region and target region information of each element in the to-be-detected image, and the detection module 502 is specifically configured to:

[0131] The target detection is performed on the to-be-detected image based on a pre-established target detection model to obtain the target region and the target region information of the target in the to-be-detected image.

[0132] Optionally, the target detection model is pre-established by the following method, and the detection module 502 is further configured to:

[0133] A training sample is determined based on historical detection images, the training sample including a historical detection image of a target violation and a historical detection image of a target non-violation;

[0134] The training sample is used to train a yolov5 network model based on a Pytorch deep learning framework to obtain the target detection model.​

[0135] Optionally, after target detection is performed on the to-be-detected image to obtain the target region and the target region information of each element in the to-be-detected image, the determining module 503 is further configured to:

[0136] compare the target region information with a set target standard;

[0137] delete the to-be-detected image in a case where the target region information does not satisfy the target standard;

[0138] in a case where the target region information satisfies the target standard, determine a first rule set related to the target from the rule set according to the category label.

[0139] Optionally, in a case where the association information meets a preset condition of the target rule, the determining module 503 is specifically configured to:

[0140] take any rule in the first rule set as the target rule, and determine a preset intersection-over-union threshold and distance threshold in the target rule;

[0141] compare the intersection-over-union with the intersection-over-union threshold, and compare the center point distance with the distance threshold;

[0142] in a case where the intersection-over-union meets a preset condition with the intersection-over-union threshold, and / or, the center point distance meets a preset condition with the distance threshold, determine the target rule.

[0143] Optionally, the determining module 503 is further configured to:

[0144] in a case where the association information does not meet the preset condition of the target rule, take a next rule in the first rule set as a new target rule, and return to perform comparison of the intersection-over-union with a new intersection-over-union threshold and comparison of the center point distance with a new distance threshold, until the association information meets the preset condition of the target rule, or all rules in the first rule set are determined as target rules.

[0145] Optionally, after the association information meets the preset condition of the target rule, the determining module 503 is further configured to:

[0146] generate an alarm information according to the target rule.

[0147] Figure 6 A structural schematic diagram of an electronic device provided by an embodiment of the present application.

[0148] The at least one processor 601 and the memory 602 connected with the at least one processor 601 are not limited to the specific connection medium between the processor 601 and the memory 602 in the embodiments of the present application, Figure 6 The connection between the processor 601 and the memory 602 is taken as an example of connection through the bus 600 in the foregoing. Figure 6 The connection mode between other components is only schematically illustrated, and is not limited to this. The bus 600 can be divided into an address bus, a data bus, a control bus, and the like, for the convenience of representation, Figure 6 In the foregoing, only one thick line is used to represent the bus, but this does not mean that there is only one bus or only one type of bus. Alternatively, the processor 601 can also be referred to as a controller, and the name is not limited.

[0149] In the embodiments of the present application, the memory 602 stores instructions executable by the at least one processor 601, and the at least one processor 601 can execute the image detection method discussed in the foregoing by executing the instructions stored in the memory 602. The processor 601 can realize the functions of various modules in the apparatus shown in Figure 6 Or Figure 6 The functions of various modules in the apparatus shown in the foregoing.

[0150] The processor 601 is the control center of the apparatus, can utilize various interfaces and lines to connect various parts of the entire control device, and through running or executing instructions stored in the memory 602 and calling data stored in the memory 602, various functions and processing data of the apparatus are realized, so as to perform overall monitoring on the apparatus.

[0151] In a possible design, the processor 601 can include one or more processing units, and the processor 601 can integrate an application processor and a modem processor, where the application processor mainly processes an operating system, a driver interface and an application program, and the modem processor mainly processes wireless communication. It can be understood that the foregoing modem processor can also not be integrated into the processor 601. In some embodiments, the processor 601 and the memory 602 can be implemented on the same chip, and in some embodiments, they can also be respectively implemented on independent chips.

[0152] The processor 601 can be a general-purpose processor, for example, a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can realize or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the image detection method disclosed in the embodiments of the present application can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0153] The memory 602, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 602 can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. The memory 602 is any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but is not limited to this. The memory 602 in the embodiment of the present application can also be a circuit or any other device capable of realizing a storage function, used for storing program instructions and / or data.

[0154] By designing and programming the processor 601, the code corresponding to the image detection method introduced in the foregoing embodiments can be fixed in the chip, so that the chip can execute the image detection method of the embodiments shown in the running time. Figure 2 How to design and program the processor 601 is a technology known to those skilled in the art, which will not be described here.

[0155] It should be noted that the above electronic device provided by the embodiment of the present application can realize all method steps realized by the method embodiment and achieve the same technical effects, and the same parts and beneficial effects of the method embodiment in the embodiment will not be described in detail here.

[0156] The embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium stores computer executable instructions, and the computer executable instructions are used to make the computer execute the image detection method in the above embodiment.

[0157] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, and the like) embodying computer readable program code.

[0158] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart Figure 1 flow or flows and / or blocks in the flowchart Figure 1 apparatus for performing the functions specified in the flowchart

[0159] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart Figure 1 flow or flows and / or blocks in the flowchart Figure 1 apparatus for performing the functions specified in the flowchart

[0160] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart Figure 1 flow or flows and / or blocks in the flowchart Figure 1 apparatus for performing the functions specified in the flowchart

[0161] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the apparent to those skilled in the art that various modifications and changes can be made thereto without departing from the scope and spirit of the present application. It is intended that the scope of the present application be limited only by the appended claims, alongside with the full scope of equivalents to which such claims are entitled.

Claims

1. An image detection method, characterized in that, The method includes: Acquire the image to be detected; Target detection is performed on the image to be detected to obtain the target region and target region information of the target in the image to be detected. The target region information includes a category label and target region location information. The category label indicates the category of the target. When both a first type of target and a second type of target exist in the image to be detected, the target state of the first type of target is determined based on the first intersection-union ratio between the target regions of the first type of target and the second type of target; wherein, the category label of the first type of target and the category label of the second type of target are associated. In the set of violation rules, a first set of violation rules related to the target is determined based on the category label and the target state. The set of violation rules contains the correspondence between violation rules and category labels. Based on the target area location information, the association information between different target areas is determined. The association information includes the intersection-union ratio and the distance between the center points of different targets in the different target areas. The intersection-union ratio is used to represent the ratio of the intersection to the union of the borders corresponding to the different target areas. If the associated information meets the preset conditions of the target violation rule, the target violation is determined, and the target violation rule is any violation rule in the first set of violation rules.

2. The method according to claim 1, characterized in that, The step of performing target detection on the image to be detected, to obtain the target region and target region information for each element in the image to be detected, specifically includes: Based on a pre-established target detection model, target detection is performed on the image to be detected to obtain the target region and target region information of the target in the image to be detected.

3. The method according to claim 2, characterized in that, The target detection model is pre-established in the following manner: Training samples are determined based on historical detection images, and the training samples include historical detection images of targets that violate regulations and historical detection images of targets that do not violate regulations. Using the training samples, the YOLOv5 network model is trained based on the PyTorch deep learning framework to obtain the object detection model.

4. The method according to claim 1, characterized in that, After performing target detection on the image to be detected to obtain the target region and target region information for each element in the image to be detected, the method further includes: The target area information is compared with the set target standard; If the target region information does not meet the target criteria, the image to be detected is deleted. If the target area information meets the target criteria, a first set of violation rules related to the target is determined from the set of violation rules based on the category label.

5. The method according to claim 1, characterized in that, The step of determining the target violation when the associated information meets the preset conditions of the target violation rule specifically includes: Take any one of the violation rules in the first set of violation rules as the target violation rule, and determine the preset intersection-union ratio threshold and distance threshold in the target violation rule; The intersection-union ratio is compared with the intersection-union ratio threshold, and the center point distance is compared with the distance threshold; If the intersection-to-union ratio (CUI) meets a preset condition and / or the center point distance meets a preset condition, the target violation is determined.

6. The method according to claim 5, characterized in that, The method further includes: If the associated information does not meet the preset conditions of the target violation rule, any next violation rule in the first set of violation rules is taken as the new target violation rule, and the process is returned to compare the intersection-union ratio with the new intersection-union ratio threshold and the center point distance with the new distance threshold, until the associated information meets the preset conditions of the target violation rule, or all violation rules in the first set of violation rules are determined as the target violation rule.

7. The method according to claim 1, characterized in that, After determining the target violation when the associated information meets the preset conditions of the target violation rule, the method further includes: Based on the target violation rules, generate alarm information.

8. An image detection device, characterized in that, include: The transceiver module is used to acquire the image to be detected; The detection module is used to perform target detection on the image to be detected, and obtain the target region and target region information of the target in the image to be detected. The target region information includes a category label and target region location information, and the category label indicates the category of the target. The determination module is used to determine the target state of the first type of target based on a first intersection-union ratio (IUU) between the target regions of the first type of target and the second type of target when both a first type of target and a second type of target exist simultaneously in the image to be detected; wherein the category label of the first type of target and the category label of the second type of target are associated. The determining module is further configured to determine a first set of violation rules related to the target based on the category label and the target state in the set set of violation rules, wherein the set of violation rules contains the correspondence between violation rules and category labels; The determining module is further configured to determine the association information between different target regions based on the target region location information. The association information includes the intersection-union ratio and the distance between the center points of different targets in the different target regions. The intersection-union ratio is used to represent the ratio of the intersection to the union of the borders corresponding to the different target regions. The determining module is further configured to determine the target violation when the associated information meets the preset conditions of the target violation rule, wherein the target violation rule is any violation rule in the first set of violation rules.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, When the computer program product is invoked by a computer, it causes the computer to perform the steps of the method as described in any one of claims 1 to 7.

Images