Safety early warning method and system and inspection robot

Abstract

The invention discloses a safety early warning method and system and an inspection robot, and the method comprises the steps: carrying out the target detection of an input image through a target detection model, obtaining a target detection result, and enabling the target detection result to comprise the position, category label and confidence of a target; constructing a triple according to the category labels; querying a knowledge graph according to the constructed triple to obtain semantic information related to the entity; and obtaining a risk judgment result according to the target detection result and the semantic information. According to the safety early warning method and system and the inspection robot, the image recognition and the knowledge graph are combined, the position, the category label, the confidence coefficient and the semantic information of the target are comprehensively considered, accurate judgment of the risk is achieved, an accurate risk judgment result is obtained, timely and effective early warning is facilitated, and the safety of the inspection robot is improved. The technical problem of poor risk judgment accuracy in the prior art is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of security, and more specifically, relates to a security warning method, system, and inspection robot. Background Art

[0002] With the booming development of e-commerce, the importance of the logistics industry has become increasingly prominent. As the core area for goods distribution, the safety status of logistics parks is directly related to the stability and efficiency of the entire supply chain.

[0003] The environment of logistics parks is complex, accidents occur frequently, and the casualty rate is high. Therefore, it is necessary to conduct safety monitoring on staff during high-risk operations such as loading and unloading, inventory checking, maintenance, and emergency rescue.

[0004] However, the existing safety monitoring and prevention means are single, the support and response capabilities for safety accidents are poor, it is difficult to analyze a common and prevalent safety problem or accident in the logistics park, unable to accurately judge unsafe behaviors in the logistics park, unable to give timely warnings, and fail to achieve the purpose of effectively preventing and reducing the occurrence of safety accidents. Summary of the Invention

[0005] The present invention provides a security warning method, which solves the technical problem of poor accuracy in risk judgment in the prior art.

[0006] To solve the above technical problems, the present invention adopts the following technical solutions to achieve:

[0007] A security warning method, comprising:

[0008] Performing object detection on an input image using a trained object detection model to obtain an object detection result, where the object detection result includes the position, class label, and confidence of the object;

[0009] Constructing a triple according to the class label;

[0010] Querying a knowledge graph according to the constructed triple to obtain semantic information related to the entity;

[0011] Obtaining a risk judgment result according to the semantic information and the object detection result.

[0012] In some embodiments of the present application, before constructing the triple according to the class label, it further includes:

[0013] Judging whether the confidence of the object reaches a set confidence level;

[0014] If the confidence of the object reaches the set confidence level, then constructing a triple according to the class label.

[0015] In some embodiments of the present application, the object detection model is a YOLOv8 model.

[0016] In some embodiments of the present application, the security warning method further includes a model training process, specifically including:

[0017] Perform label annotation on the images in the dataset, and divide the labeled dataset into a training set, a validation set, and a test set;

[0018] Train a neural network model based on the training set and the validation set;

[0019] Test the trained neural network model based on the test set to obtain the target detection model.

[0020] In some embodiments of the present application, obtaining a risk judgment result according to the semantic information and the target detection result specifically includes:

[0021] Obtain the risk scores corresponding to the semantic information, confidence level, position, and class label respectively;

[0022] Perform weighted summation on each risk score according to its respective weight to obtain a total risk score;

[0023] Obtain the corresponding risk judgment result according to the total risk score.

[0024] In some embodiments of the present application, obtaining the corresponding risk judgment result according to the total risk score specifically includes:

[0025] If the total risk score is within the first score interval, the risk judgment result is the first risk level;

[0026] If the total risk score is within the second score interval, the risk judgment result is the second risk level, and a reminder message is sent;

[0027] If the total risk score is within the third score interval, the risk judgment result is the third risk level, and an alarm message is sent;

[0028] Wherein, any value within the third score interval is greater than any value within the second score interval,

[0029] Any value within the second score interval is greater than any value within the first score interval.

[0030] In some embodiments of the present application, the target detection result and the risk judgment result are visually displayed on the input image.

[0031] In some embodiments of the present application, the position of the target is the coordinate of the bounding box of the target;

[0032] Display the bounding box of the target on the input image, and display the confidence level, class label, and total risk score on the bounding box.

[0033] Based on the design of the above-mentioned security warning method, the present invention proposes a security warning system, including a data processing module, and the data processing module executes the above-mentioned security warning method.

[0034] Based on the design of the above-mentioned security warning system, the present invention proposes an inspection robot, including the above-mentioned security warning system.

[0035] Compared with the prior art, the advantages and positive effects of the present invention are as follows: The security warning method, system and inspection robot of the present invention use an object detection model to perform object detection on the input image to obtain an object detection result, and the object detection result includes the position, category label and confidence of the object; construct a triple according to the category label; query the knowledge graph according to the constructed triple to obtain semantic information related to the entity; obtain a risk judgment result according to the object detection result and semantic information. The security warning method, system and inspection robot of the present invention combine image recognition with a knowledge graph, comprehensively consider the position, category label, confidence and semantic information of the object, realize accurate judgment of risks, obtain relatively accurate risk judgment results, so as to facilitate timely and effective warning, and solve the technical problem of poor accuracy of risk judgment in the prior art. The inspection robot of the present invention can replace manual inspection to go to dangerous areas and reduce the inspection risk.

[0036] After reading the specific embodiments of the present invention in conjunction with the accompanying drawings, other features and advantages of the present invention will become clearer. Description of the Drawings

[0037] Figure 1 is a flowchart of an embodiment of the security warning method proposed by the present invention;

[0038] Figure 2 is a schematic diagram of a triple; Figure 3 is a sub-graph;

[0039] Figure 4 is a flowchart of another embodiment of the security warning method proposed by the present invention;

[0040] Figure 5 is a network architecture diagram of the YOLOv8 model;

[0041] Figure 6 is a flowchart of an embodiment of the model training process;

[0042] Figure 7 is Figure 1 a flowchart of some steps in

[0043] Figure 8 is a schematic diagram of setting labels; Figure 9 is a schematic diagram of image annotation;

[0044] Figure 10 It is a flowchart of the text extraction step;

[0045] Figure 11 It is a schematic structural diagram of the knowledge pattern layer;

[0046] Figure 12 It is a flowchart of SRL optimization processing;

[0047] Figure 13 It is a flowchart of Python-Py2neo knowledge graph construction;

[0048] Figure 14 It is a schematic diagram of the knowledge graph;

[0049] Figure 15 It is a schematic diagram of visual display; Detailed implementation manners

[0050] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0051] It should be noted that in the description of the present invention, the terms indicating directions or positional relationships such as "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance.

[0052] It should also be noted that in the description of the present invention, unless otherwise clearly defined and limited, the terms "installation", "connection", and "connection" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0053] Embodiment 1

[0054] The security warning method of this embodiment mainly includes the following steps, as shown in Figure 1 shown.

[0055] Step S11: Use the trained target detection model to perform target detection on the input image to obtain the target detection result, and the target detection result includes the position, category label, and confidence of the target.

[0056] The object detection model adopts a deep learning-based object detection algorithm, which can directly evaluate the entire input image through a single full convolutional network, simultaneously predict the position coordinates and category of the object, and output the confidence level.

[0057] For example, the position of the object is the coordinate of the bounding box of the object. The object detection model detects an object with a bounding box coordinate of (100, 200, 50, 80), a class label of "person sitting on a forklift", and a confidence level of 0.9.

[0058] Step S12: Construct triples according to the class label.

[0059] The triples contain the basic information of the entities and their relationships in the input image. For example, according to the class label "person sitting on a forklift", construct the triple (forklift, HAS_PERSON, person), as shown in Figure 2 shown.

[0060] Step S13: Query the knowledge graph according to the constructed triples to obtain semantic information related to the entities.

[0061] The knowledge graph contains various entities and relationships related to the target category.

[0062] Query and return the sub-graph related to the entities in the already constructed knowledge graph. When generating the sub-graph, a CSV file containing the semantic information related to the sub-graph will also be generated, which helps to make risk judgments.

[0063] For example, for the class label "person sitting on a forklift" and the triple (forklift, HAS_PERSON, person), query and return the sub-graph related to the entities "forklift" and "person" in the knowledge graph, as shown in Figure 3 shown, and the CSV file containing the semantic information related to the sub-graph. The CSV file contains the semantic information "person falling from a height" related to the entities "forklift" and "person".

[0064] Step S14: Obtain the risk judgment result according to the semantic information and the object detection result.

[0065] Based on the position, class label, confidence level, and semantic information of the object, comprehensively make a risk judgment to obtain a relatively accurate risk judgment result.

[0066] The security warning method of this embodiment uses an object detection model to perform object detection on the input image to obtain object detection results, where the object detection results include the position, class label, and confidence of the object; constructs triples according to the class label; queries the knowledge graph according to the constructed triples to obtain semantic information related to the entity; and obtains a risk judgment result according to the object detection result and the semantic information. The security warning method of this embodiment combines image recognition and the knowledge graph, comprehensively considers the position, class label, confidence, and semantic information of the object, realizes accurate judgment of risks, obtains relatively accurate risk judgment results, facilitates timely and effective warning, and solves the technical problem of poor risk judgment accuracy in the prior art.

[0067] In some embodiments of the present application, before constructing triples according to the class label, the following steps are further included, as shown in Figure 4 shown.

[0068] Step S12-1: Determine whether the confidence of the object reaches the set confidence.

[0069] If so, that is, if the confidence of the object reaches the set confidence, then execute step S12: Construct triples according to the class label.

[0070] The confidence reflects the credibility of the object detection result. A higher confidence means a more accurate detection of the object. When the confidence is lower than the set confidence, it means that the accuracy of object detection is very low, and there is no need to construct triples to avoid misjudgment of risks.

[0071] In some embodiments of the present application, the object detection model is the YOLOv8 model.

[0072] The YOLOv8 model, a single-stage object detection network, directly performs a single full convolutional network evaluation on the entire input image and can simultaneously predict the position coordinates and class of the object.

[0073] The YOLOv8 model, through optimizing the backbone network and introducing operations such as depthwise separable convolution, performs excellently in multiple fields such as image classification and object detection. YOLOv8 performs object detection by processing the entire image at once, greatly improving the detection speed and maintaining high accuracy.

[0074] The network architecture of the YOLOv8 model is as shown in Figure 5 shown, and can be mainly divided into three parts: the core network (Backbone), the neck network (Neck), and the detection head network (Head).

[0075] The core network consists of five convolutional units (Conv), four C2f units, and one SPPF unit. The convolutional unit contains 2D convolution, BatchNorm2d, and SiLU activation function, which are used to downsample the input feature map, increase the number of channels, and enhance the non-linear representation ability. Compared with the C3 module of YOLOv5, the C2f unit has fewer parameters and better feature extraction ability, enabling better feature fusion and improving the detection performance. The SPPF unit is located at the end of the backbone network, providing multi-scale representations of the input feature map. By pooling at different scales, the model can capture features at different abstraction levels, improving the performance and efficiency of the model when processing inputs of different sizes. The neck adopts a structure that combines FPN and PAN. FPN transmits strong semantic features output by the high-level network in a top-down manner, while PAN transmits strong localization features extracted by the low-level network in a bottom-up manner. These two networks perform feature aggregation in the neck part, thereby realizing parameter aggregation of features from different backbone layers, and finally generating a feature map that is rich in both semantic information and spatial information, providing effective feature information for subsequent object detection tasks. Finally, the Head comprehensively uses the feature information obtained previously to generate the predicted bounding boxes and categories of the objects in the original image, obtaining the final detection results. This design can help the network better understand the image content and improve the accuracy and robustness of object detection.

[0076] The improvements of the YOLOv8 algorithm include the optimization of the model architecture, the improvement of the training strategy, and the inference optimization, etc. The main features of YOLOv8 are as follows:

[0077] (1) Advanced backbone and neck architectures: YOLOv8 adopts state-of-the-art backbone and neck architectures, thus improving the feature extraction and object detection performance.

[0078] (2) Anchor-free decoupled Ultralytics head: YOLOv8 adopts an anchor-free decoupled Ultralytics head, which helps to improve the accuracy and a more efficient detection process compared with anchor-based methods.

[0079] (3) Optimized precision and speed trade-off: YOLOv8 focuses on maintaining the best balance between precision and speed, suitable for real-time object detection tasks in different application fields.

[0080] (4) Various pre-trained models: YOLOv8 provides a series of pre-trained models to meet various task and performance requirements, making it easier to find the right model for specific use cases.

[0081] As an advanced tool in the field of object detection and instance segmentation, the advantages of the YOLOv8 algorithm can play an important role in complex environments such as logistics parks.

[0082] In some embodiments of the present application, the security warning method further includes a model training process, specifically including the following steps. See Figure 6 as shown.

[0083] Step S21: Perform label annotation on the images in the dataset, and divide the annotated dataset into a training set, a validation set, and a test set.

[0084] The automatic annotation function of VOTT (Visual Object Tagging Tool) can significantly reduce the workload of preliminary annotation. Therefore, VOTT is used for label annotation to optimize the annotation process.

[0085] Store the original video obtained at the warehouse site in the source folder, and then import the original video through Open local project of VOTT. Considering the characteristic that on-site personnel do not move frequently over a large range, set the video frame rate to one frame per second.

[0086] Then set the tags by creating new TAGS. For example, set eight tags: "person", "smoking", "not wearing a reflective vest", "walking in the restricted area", "stacking too high", "restricted area", "parking in violation of regulations", "driving a forklift in violation of regulations". See Figure 8 as shown.

[0087] Next, use the annotation function of VOTT to annotate the relevant images in each frame of the video. See Figure 9 as shown.

[0088] Then randomly divide the annotated dataset with the help of Python to obtain a training set, a test set, and a validation set.

[0089] Step S22: Train the neural network model based on the training set and the validation set.

[0090] First, build an environment that can run the YOLOv8 model. After the construction is completed, training can start.

[0091] Step S23: Test the trained neural network model based on the test set to obtain the target detection model.

[0092] Given the complexity of the light conditions in the warehouse environment, such as the variation in the intensity of light, in order to improve the adaptability of the model under different light conditions, the brightness channel (V channel) is adjusted in the HSV color space to simulate bright and dark change scenarios. At the same time, considering the variability of the workers' postures during actual operations, a variety of data augmentation strategies are introduced before model training, including augmentation means such as simulating bright and dark light changes and angle transformation. These enhancement measures not only improve the robustness of the model to complex environments but also effectively enhance the generalization ability of the model to targets under different postures and working conditions.

[0093] Since a variety of data augmentation means are added during the training process, in order to ensure the stability and convergence of the model under high-intensity data augmentation, the hyperparameters are set as follows.

[0094] The number of training epochs determines the number of times the entire training set will be input into the model. Increasing the number of training epochs allows the model to learn the training set multiple times on different batches, helping the model to better fit the data and improve the final accuracy. In this application, the epoch is set to 500, aiming to ensure that the model has enough iteration times for sufficient learning under data augmentation, thereby promoting the stable convergence of the loss function.

[0095] The batch size determines the number of samples input into the model in each iteration. The batch size is set to 32 to balance between computational resource limitations and model training speed. A too-large batch size will cause the model training to occupy more memory, while a too-small batch size may lead to unstable training results. By setting the batch size to 32, it can ensure that there are enough diverse and balanced samples in each training iteration, reduce gradient fluctuations, and improve the generalization ability of the model.

[0096] The initial learning rate (lr0) is one of the extremely important hyperparameters when training a deep learning model. It determines the step size of model parameter updates during each gradient descent. In this application, the initial learning rate is set to 0.01 because this value can balance the model training speed and stability. If the learning rate is too large, the model may oscillate, resulting in an unstable loss function; while if the learning rate is too small, the model training speed will be too slow to converge within a reasonable time. Choosing 0.01 as the initial value can avoid unstable gradient updates while ensuring the training speed.

[0097] In this application, SGD is selected as the optimizer for the gradient descent algorithm, and the momentum parameter is set to 0.937. It is used to accumulate the directions of previous gradients during gradient descent, thereby accelerating the convergence of the model and suppressing oscillations. Simple SGD updates are vulnerable to fluctuations in the gradient direction, while by introducing momentum, the model can update the weights more smoothly, especially when facing a highly complex and unstable loss surface. A momentum of 0.937 means that approximately 93.7% of the momentum in each update comes from the gradient direction of the previous update, thus reducing the phenomenon of gradient oscillations during model training and accelerating convergence.

[0098] The weight decay coefficient is a regularization technique used to prevent the model from overfitting. It adds a penalty term related to the model weights to the loss function, restricts the unlimited growth of the weights, thereby suppressing the complexity of the model and reducing the risk of overfitting. The setting of this coefficient can control the intensity of weight updates. A smaller weight decay value helps to retain the flexibility of the model, while a larger value will limit the capacity of the model. In this application, the weight decay coefficient is set to 0.0005 to enhance the generalization ability while maintaining the accuracy and convergence of the model.

[0099] The entire hyperparameter tuning process repeatedly tries different combinations of hyperparameters based on the training curve of the model, the loss function, the training time, and the performance of the validation set. In the initial stage, larger learning rates and smaller batch sizes were tried to quickly test the convergence of the model; in the later stage, parameters such as the learning rate and momentum were gradually refined to obtain the best performance of the model. Finally, with the cooperation of the data augmentation strategy, after multiple rounds of adjustment, a combination of epoch = 500, batch size = 32, lr0 = 0.01, momentum = 0.937, and weight decay = 0.0005 was selected, enabling the model to be stably trained in a complex data environment and achieving a good generalization effect.

[0100] After multiple rounds of training and optimization, we obtained the training effect and performance of the best model, and the recognition rate of behaviors such as not wearing a reflective vest (NWRC) reached over 90%.

[0101] The trained model is tested using the test set. When the recognition accuracy reaches the set accuracy, the model is qualified, and the target detection model is obtained.

[0102] By designing the above steps S21 - S23, a qualified target detection model can be obtained to achieve accurate detection of the target.

[0103] In some embodiments of this application, in step S14, according to the target detection result and semantic information, a risk judgment result is obtained, which specifically includes the following steps. SeeFigure 7 as shown

[0104] Step S14-1: Obtain the risk scores corresponding to semantic information, confidence level, location, and category label respectively.

[0105] Step S14-2: Weighted sum the respective risk scores according to their respective weights to obtain the total risk score.

[0106] Step S14-3: Obtain the corresponding risk judgment result according to the total risk score.

[0107] For example, R t = w1*R1 + w2*R2 + w3*R3 + w4*R4;

[0108] where, R t is the total risk score of the target;

[0109] R1 is the risk score corresponding to semantic information; R2 is the risk score corresponding to confidence level;

[0110] R3 is the risk score corresponding to location; R4 is the risk score corresponding to category label;

[0111] w1, w2, w3, w4 are preset weights, and w1 + w2 + w3 + w4 = 1.

[0112] By designing S14-1 to S14-3, comprehensively considering the risk scores and weights corresponding to semantic information, confidence level, location, and category label respectively, an accurate total risk score can be obtained, and then the corresponding risk judgment result can be obtained according to the total risk score, realizing the accurate judgment of risks.

[0113] The weights w1, w2, w3, w4 of each dimension need to be set according to the actual application scenario. For example, if there are many dangerous areas where entry is prohibited in this scenario, a larger value can be set for w3.

[0114] (1) Scoring analysis of the semantic relationship between the target and the risk entity. Score based on the triple information and the semantic relationship in the knowledge graph, such as there is a person on the forklift, the person is not wearing safety equipment, etc.

[0115] Each target category (such as "forklift", "person", "pallet", etc.) will establish a relationship with predefined risk entities (such as "people should not sit on the forklift", "the pallet stack is too high and collapses") in the knowledge graph. The closer the relationship between the target category and the relevant risk entity, the higher the risk score.

[0116] Basis for scoring:

[0117] (a) If the target category is directly associated with a risk entity (e.g., the target is "forklift" and has a "sitting" relationship with "person"), the risk score increases.

[0118] (b) If the target category has potential associations with multiple known risk entities (such as ignition sources, hazardous areas, etc.), the risk score is even higher.

[0119] For example, when semantic information indicates that there is no association between the label category and the risk entity, the risk score R1 is very low, e.g., R1 is 0. When semantic information indicates that there is an association between the label category and the risk entity, the risk score R1 is higher, e.g., R1 is the set score r11.

[0120] (2) Confidence score analysis. The confidence provided by the YOLOv8 model reflects the credibility of the object detection result. A higher confidence means that the detection of the target is more accurate, which can reduce the risk of misidentification. For targets with low confidence, further manual review or high-risk warnings may be required.

[0121] Scoring basis: Score according to the confidence of object detection, and set a scoring threshold (e.g., 0.8). If the confidence is greater than this threshold, it is regarded as a reliable target and the risk score is higher; if it is lower than this threshold, the risk score is lower.

[0122] For example, when the confidence < the set threshold, the risk score R2 is lower, e.g., R2 is the set score r21. When the confidence ≥ the set threshold, the risk score R2 is higher, e.g., R2 is the set score r22.

[0123] Among them, r22 > r21. The set threshold is greater than the set confidence.

[0124] (3) Location score analysis. When the YOLOv8 model performs object detection, it can output coordinate information such as (x, y, w, h) to represent the location of the target. Among them, x and y are the horizontal and vertical coordinates of the center point of the bounding box, and w and h are the width and height of the bounding box. Certain areas can be set in advance as high-risk areas, such as chemical storage areas, high-risk operation areas, hazardous areas, restricted areas, etc.

[0125] Scoring basis: If u i ∈V, the risk score is higher, where u i is the coordinate information of the i-th target, and V is the set of high-risk areas set in advance.

[0126] Score based on the coordinate information of the target, set high-risk areas in advance, and score according to whether the target is located in a high-risk area.

[0127] For example, when the location coordinates When the risk score R3 is low, for example, R3 is the set score r31. When the position coordinates ∈ V, the risk score R3 is high, for example, R3 is the set score r32. Among them, r32 > r31.

[0128] (4) Label category scoring analysis. In the historical accident data and violation data, query the label category. If found, the risk score is high.

[0129] (41) Historical data scoring analysis

[0130] If the target behavior is related to historical accident records, increase the risk score. Combining the past safety accident data in the park, if the target detection result is similar to the historical accident pattern (such as a certain type of dangerous behavior often leads to accidents), then increase the risk score.

[0131] Scoring basis: Traverse the historical accident data. If the label category or behavior of the target matches the historical accident records, the risk score can be automatically increased.

[0132] If the label category of the target is not found in the historical accident data, then R 41 = 0.

[0133] If the label category of the target is found in the historical accident data, then R 41 is the set score r41.

[0134] (42) Violation behavior scoring analysis

[0135] If the target has a violation behavior (such as smoking), increase the risk score. Combining the safety operation specification articles in the park, set violation behaviors such as "smoking". When the target is detected to have a violation behavior such as "smoking", increase the risk score.

[0136] Scoring basis: Traverse the label categories of the target detection. If a label such as "smoking" is detected, increase the risk score.

[0137] If the label category of the target is not found in the violation data, then R 42 = 0.

[0138] If the label category of the target is found in the violation data, then R 42 is the set score r42.

[0139] The risk score R4 corresponding to the category label = w 41 *R 41 + w 42 *R 42 . w 41 + w 42 = 1.

[0140] Among them, R41 is the risk score corresponding to the historical accident, R 42 is the risk score corresponding to the violation behavior, w 41 , w 42 is the preset weight.

[0141] In some embodiments of the present application, according to the semantic information and the target detection result, a risk judgment result is obtained, which specifically includes: respectively obtaining the risk scores corresponding to the semantic information, confidence level, position, category label, and park environment; performing weighted summation on each risk score according to its respective weight to obtain a total risk score; and obtaining the corresponding risk judgment result according to the total risk score. Among them, for the risk scores corresponding to the semantic information, confidence level, position, and category label, refer to the previous description and will not be elaborated here.

[0142] Environmental factor scoring analysis: Scoring is based on the environment of the monitoring screen. Environmental factors in the park (such as light changes, personnel density, equipment status, etc.) will also affect risk assessment. Under specific environmental conditions, the risk score R5 corresponding to the park environment can be weighted and adjusted based on these environmental factors. When the light brightness < the set brightness, the risk score R5 is relatively high, for example, R5 is the set score r51. When the light brightness ≥ the set brightness, the risk score R5 is relatively low, for example, R5 is the set score r52. r52 < r51.

[0143] Comprehensively considering the semantic relationship between the target and the risk entity, the confidence level of target detection, target position, historical accidents, violation behaviors, environmental factors, etc., a multi-dimensional weighted risk score calculation formula is designed as follows: R t = w1*R1 + w2*R2 + w3*R3 + w4*R4 + w5*R5. Wherein, w1 + w2 + w3 + w4 + w5 = 1.

[0144] In some embodiments of the present application, according to the total risk score, the corresponding risk judgment result is obtained, which specifically includes:

[0145] (1) If the total risk score is within the first score interval, the risk judgment result is the first risk level.

[0146] The first risk level is low risk, no reminder is issued, and no alarm is issued.

[0147] (2) If the total risk score is within the second score interval, the risk judgment result is the second risk level, and a reminder message is issued.

[0148] The second risk level is medium risk, a reminder message is issued, and no alarm message is issued.

[0149] (3) If the total risk score is within the third score interval, the risk judgment result is the third risk level, and an alarm message is issued.

[0150] The third risk level is high risk, and an alarm message is issued.

[0151] Any value within the third score range is greater than any value within the second score range.

[0152] Any value within the second score range is greater than any value within the first score range.

[0153] Therefore, after calculating the total risk score of the target, it is possible to determine whether to trigger a security warning based on the score range it belongs to. The judgment rules are as follows:

[0154] Low risk: If the total risk score is within the first score range, for example, if the first score range is (-∞, 50%), it is considered that the risk of the target is low and no warning needs to be triggered.

[0155] Medium risk: If the total risk score is within the second score range, for example, if the second score range is [50%, 75%), a reminder or preliminary warning is triggered.

[0156] High risk: If the total risk score is within the third score range, and the second score range is [75%, +∞), a high-risk warning is triggered and relevant measures are taken (such as notifying the management personnel, starting an emergency response, etc.).

[0157] According to the score range where the total risk score is located, the risk level is determined, and it is determined whether to issue a reminder or an alarm, thus achieving a definite risk judgment result based on the total risk score, realizing accurate risk judgment, and the judgment method is simple and accurate.

[0158] In some embodiments of the present application, in order to facilitate the monitoring personnel to obtain the risk judgment result in a timely and accurate manner, the target detection result and the risk judgment result are visually displayed on the input image.

[0159] On the input image, the position, class label, and confidence level of the detected target are visually displayed, and the risk judgment result is superimposed and displayed.

[0160] In some other embodiments of the present application, the position of the target is the coordinate of the bounding box of the target. The bounding box of the target is displayed on the input image, and the confidence level, class label, and total risk score are displayed on the bounding box.

[0161] By displaying the confidence level, class label, and total risk score on the bounding box, it is convenient for the monitoring personnel to observe timely, effectively, and intuitively.

[0162] Therefore, in the present application, for the convenience of the monitoring personnel to quickly observe and identify, the risk score and the target detection result are visually displayed. The display can be carried out in the following ways:

[0163] (1) On the input image, use color coding or markings to identify the risk level of the target. For example, use green to represent low risk, yellow to represent medium risk, and red to represent high risk.

[0164] (2) Display the total risk score on the bounding box of the target to help managers quickly identify high-risk targets.

[0165] Through risk judgment and visual display, this application combines visual object detection with a knowledge graph to achieve intelligent risk assessment based on semantics. This can not only improve the accuracy and reliability of target recognition, but also provide valuable risk information for decision-makers, thereby improving the overall security protection ability.

[0166] Next, through a specific embodiment, the specific steps of the security warning method will be specifically described.

[0167] The word frequency analysis method was used to analyze the non-standard behaviors in the logistics park, and word cloud diagrams related to "not wearing a reflective vest", "smoking", "walking in the restricted area", "stacking too high", etc. were generated. Finally, these four behaviors were selected as the key non-standard operation behaviors to be monitored. Once these behaviors occur, they may directly threaten the lives of operating personnel. Therefore, the security warning method is used to focus on monitoring these behaviors in the logistics park work to effectively prevent and reduce the occurrence of safety accidents.

[0168] I. Establishment of the knowledge graph. A knowledge graph (Knowledge Graph) is a semantic network that reveals the associations between things through entity nodes and relationship edges. A knowledge graph is a knowledge structure represented by nodes and edges. Nodes represent entities or concepts, and edges represent the semantic relationships between them.

[0169] This application uses a graph database to manage and store the knowledge graph. Graph databases are particularly suitable for storing knowledge graphs because they can naturally represent the complex relationships between data and have obvious advantages in processing the association relationships between entities. Graph databases have high performance, flexibility, and elasticity. Therefore, the knowledge graph constructed using a graph database has an efficient storage and query mechanism and rich visual representation.

[0170] This application designs the following process for extracting regulatory clause texts: The four steps of text preprocessing, article classification, NLP-SRL processing, and knowledge pattern matching are used to automatically extract the knowledge of the regulatory text. See Figure 10 as shown.

[0171] Text extraction presents the standardized clauses with different structures and semantic organization patterns in the structure of knowledge elements. This application uses the establishment of regular matching rules to classify clause texts into four categories, namely reference category (mutual reference between standardized clauses), context category (clauses with specific context constraints), attribute category (clauses stipulating the thresholds of logistics operations), and process category (clauses containing the sequential issues of logistics operations). The four categories of clauses cover the key regulations of logistics operation standard clauses. This application analyzes the semantic role components of each category as the schema layer of the knowledge graph according to the expression characteristics of the four categories of clauses.

[0172] For example, the attribute class clauses usually contain four main parts: subject, negative description, comparative description, and quantity description, which serve as the subject, adverbial, predicate, and object in the sentence respectively. In semantic role annotation, these four parts correspond to: agent (ARG0), adverbial (ARGM-ADV), predicate (PRED), and patient (ARG1). Taking the clause "People shall not stand beside the forklift" as an example, after semantic role annotation analysis, the following results are obtained: "forklift" corresponds to ARG0, that is, the agent; "not" corresponds to ARGM-ADV, that is, the adverbial; "stand" is the central predicate PRED; "people" corresponds to ARG1, that is, the patient.

[0173] Based on the semantic role relationship, the schema layer of the knowledge graph is established, and the semantic roles are structurally associated through entities and relationships to form a standardized knowledge network. The knowledge schema layer is as Figure 11 shown.

[0174] Text preprocessing: mainly for normative texts, conducting research on knowledge extraction. First, it is necessary to convert the PDF format of the normative document into a pure text txt format.

[0175] NLP-SRL: To achieve standardized knowledge extraction and determine the processing granularity, this application adopts semantic role annotation (Semantic Role Labeling, SRL) technology. NLP-SRL processing is the key to knowledge extraction, including two stages: NLP processing and SRL post-optimization processing. The former aims to obtain the word segmentation and semantic role annotation results of the sentence.

[0176] The optimization process after SRL mainly focuses on the original SRL results and utilizes the sentence tokenization results. According to different predicates, SRL generates multiple Predicate-Argument (PA) structures. Each PA structure consists of multiple Semantic Rule Elements (SREs) in the form of quadruples (entity, type, begin, end), where begin and end represent the starting and ending indices of the words in the token list. All PA structures in the SRL results of a single sentence are called multiple sets of SRL. The SREs within each set are called semantic role components, and the [begin, end) of the SRE is called the token index interval. For articles with different expression forms and semantic structures, there are mainly two points that need to be optimized in the SRL results:

[0177] (1) There are word nestings in the SREs. From the perspective of the overall sentence semantic understanding, coarser-grained sentence components are more useful. Therefore, these words should be retained as the optimal SREs. These optimal SREs may be scattered in different SRL sets. Replacing the original results can not only retain the natural entity relationship expression form of SRL but also extract high-quality knowledge.

[0178] (2) Starting from the fact that SREs between two punctuation marks have natural semantic relevance, it can be found that these semantic components may be scattered in different SRL sets. To more comprehensively reflect the semantics of the sentence, the SREs within the range of two punctuation marks (such as the words between two commas in a sentence) need to be added to the same SRL set. Therefore, from these two aspects, this application conducts post-optimization processing on the original SRL results, and its process is as Figure 12 shown and is specifically described as follows:

[0179] (1) Handling word inclusion: First, identify all different SREs in the sentence, which is based on the original SRL results. Arrange these valid SREs in the order of the token index to ensure obtaining a valid SRE sequence. For different SREs at the same token index position, only retain the SRE that appears first in the original SRL results. The first result can best reflect the role of the current word in the sentence because the SRL technology analyzes the predicates in order. Then, first remove the SREs that are included by other SREs, and conduct a mathematical interval comparison of the token indices of the valid SREs, only retaining the words with the widest coverage. After sorting, obtain the optimal SRE sequence. Then, replace the SREs in the processing results of each set of original SRLs with the optimal SREs according to the token index interval. Finally, perform duplicate removal of the internal SREs in each set of SRLs, as well as duplicate removal and nesting processing between multiple sets of SRLs to obtain a preliminary SRL optimization result.

[0180] (2) Interval merging process: First, determine the grouping intervals based on the positions of the word punctuation tokenization metrics. Then, compare the tokenization metrics of each ideal SRE with the grouping intervals and assign the SREs to the appropriate groups. Next, add the other SREs from the same group to the SRL results that contain the SREs of that group. Finally, repeat the previous step to add words to obtain more optimized SRL results.

[0181] (3) Eliminate invalid SRL groups: SRL groups that contain fewer than 2 SRL elements (partially retain 2) or have a total character length of less than 7 characters are considered invalid results and will be eliminated to obtain the optimal SRL of the results.

[0182] In this application, a standard knowledge graph is constructed and stored based on the Neo4j graph database, as shown in Figure 13 shown.

[0183] For a standard knowledge graph of specifications, the lower-level graph (clause concept layer) is built on top of the upper-level graph (specification clause layer), and there is a dependency relationship between the levels. Therefore, the corresponding graphs need to be built in sequence. Before creating nodes and relationships in Neo4j, it should be judged whether the current node or relationship already exists in the database to avoid duplicate addition.

[0184] There are mainly two ways to build a graph in Neo4j: One is to run Cypher statements based on the built-in Neo4j Browser, and CSV files can be batch-loaded through the LOAD CSV syntax to establish entities and relationships; the other is based on programming drivers to connect to the database and perform create, read, update, and delete operations on the graph data. In a Python environment, in addition to the Neo4j Python driver, the Py2neo module can also be used. Py2neo is a comprehensive Neo4j driver library and toolkit that can both flexibly create nodes and relationships by operating on Python variables and build graphs through Cypher statements. While the Neo4j Python driver can only adopt the latter method.

[0185] Different building methods have different applicability, advantages, and disadvantages. When generating a specification graph, different methods and procedures can be adopted according to different data. Generally speaking, for structured tabular data, the method of using LOAD CSV and Cypher statements is more efficient and convenient, but it is difficult to perform flexible data processing; for complex data that is difficult to organize into a table, the Python-Py2neo method can be adopted to achieve a good combination of data processing and graph construction. However, this method is difficult to take into account the merging of graph elements and batch construction, and may also cause a large network overhead due to frequent access to Neo4j. The code refactoring difficulty is also relatively large when the requirements change.

[0186] For the construction of the knowledge graph at the specification clause level, since the process of reading the specification text line by line, judging the chapter and clause, and then processing it into a structured table to build the graph is relatively complex, using Py2neo is more straightforward. Python - Py2neo integrates the graph building program into the file parsing program, and the specific process is as follows:

[0187] (1) Connect to the Neo4j database running in the background through the Graph object of the Py2neo module, and construct empty node lists, relationship lists, and subgraph containers.

[0188] (2) Read the specification text, judge the type of each clause (chapter, section, clause, composite clause, sub - clause) line by line, define nodes and relationships through Node and Relationship objects, and add them to the corresponding data containers.

[0189] (3) After the file traversal is completed, add the node list and relationship list containing graph data to the subgraph container to form a Subgraph object. Finally, use the Graph.create or Graph.merge method to batch build the graph in a transaction. Subgraph provides an efficient way to send multiple entities to the database in a single round - trip, which can significantly reduce network overhead. At the same time, Py2neo supports executing each operation in a separate transaction. For a transaction, either commit completely or roll back all when it fails. This feature is beneficial to the integrity and consistency of graph construction.

[0190] The advantages of the above - mentioned method are as follows: Through the Py2neo module, the graph building program can be seamlessly integrated with the file parsing program, avoiding the intermediate data conversion link; at the same time, using the transaction and batch creation functions of Neo4j, graph elements can be efficiently batch - built, reducing network overhead. In addition, Py2neo also provides a flexible way to define nodes and relationships, which is beneficial to the processing of complex data and graph construction.

[0191] This application realizes data processing and graph construction by adopting the Python - Py2neo method. In the specific implementation process, different construction methods will be flexibly selected and combined according to data characteristics and actual needs to obtain the best graph construction effect. This application mainly analyzes the "General Specification for Logistics Center Operations", which contains a total of 208 specification clauses. The established knowledge graph contains 672 nodes and 978 relationships. The specification knowledge graph is as Figure 14 shown.

[0192] II. Model training. Use the trained object detection model to perform object detection on the detection image to obtain the object detection results: the position, class label, and confidence of the object. Construct triples.

[0193] Object detection and triple construction are the foundation and starting point of the entire process. The main task of this step is to use the YOLOv8 model to perform object detection on the input image, obtain information such as the location, category, and confidence of each detected object, and construct triples (subject, predicate, object) based on the detection results to prepare for subsequent knowledge graph queries and risk judgments.

[0194] Specifically, first use the well-trained YOLOv8 model to infer the input image. The YOLOv8 model can efficiently detect various objects in the image and give the bounding box coordinates (x, y, w, h), category labels (such as people, vehicles, packages, etc.), and confidence scores of each object.

[0195] After obtaining reliable detection results, it is necessary to construct triples based on the category labels of each detected object. Triples are the basic units representing entity relationships in the knowledge graph, consisting of three parts: subject, predicate, and object. If the confidence is satisfied, a triple is constructed and generated.

[0196] Through the above steps, a set of preliminary triples is obtained, laying the foundation for subsequent risk assessment. These triples contain the basic information of the target entities and their mutual relationships in the input image.

[0197] To facilitate subsequent knowledge graph queries and risk judgments, this application can store the triples corresponding to each detected object in a data structure, and this application stores the data in a CSV format file. In this way, when making risk judgments, not only can the semantic information in the knowledge graph be utilized, but also information such as the location and confidence of visual targets can be combined for a more comprehensive and accurate assessment.

[0198] Through the step of object detection and triple construction, a mapping relationship between visual targets and entities in the knowledge graph is successfully established, laying a solid foundation for subsequent knowledge graph queries and risk judgments.

[0199] This application further optimizes the triple generation process to improve the construction speed and accuracy. The specific methods include introducing more context information and semantic rules, automatically correcting detection errors, and further enhancing the robustness of the model through machine learning techniques.

[0200] III. Design a suitable knowledge graph and use the automatically generated Cypher query language to find risk information related to the detected objects in it.

[0201] The design of the knowledge graph and the writing of Cypher query statements are the core links of the whole process. As an expressive data model, the knowledge graph can effectively represent complex semantic relationships and concepts, providing valuable background knowledge and context information for the risk judgment of visual targets. The design of the knowledge graph needs to be customized according to specific application scenarios and data sources.

[0202] In the scenario of this application, the knowledge graph should contain various entities and relationships related to visual target categories, such as people, whether they wear safety helmets, whether they wear reflective vests, etc.

[0203] After constructing the knowledge graph, the Cypher query language can be automatically generated and used to query information related to the detection target "a person sitting on a forklift" in it. The Cypher statement is the query language of the Neo4j graph database. It expresses and queries data in a graph form and has powerful pattern matching and traversal capabilities, enabling it to efficiently find nodes and relationship paths that meet the conditions in the knowledge graph. The query statement can filter conditions for nodes and relationships, thereby retrieving information related to the logistics operation scenario from the graph database, including worker information, work scenario characteristics, and the association relationship between workers and work scenarios.

[0204] When writing Cypher query statements, write Cypher query statements according to the subject and object in the constructed triples to find related nodes and relationships. For example, for the category label "a person sitting on a forklift", the triple (forklift, HAS_PERSON, person), write the following Cypher query:

[0205] MATCH(shovel{name:'forklift'})-[relation1]->(intermediate)-[relation2]-(person{name:'person'})

[0206] RETURN shovel,relation1,intermediate,relation2,person

[0207] This query will query and return the sub-graph related to "forklift" and "person" in the already constructed knowledge graph. As Figure 3 shown, while generating the sub-graph, a CSV file about the sub-graph information will also be generated at this stage, which helps with risk judgment. According to specific application scenarios, more complex queries can be written to integrate multiple information sources to obtain a more comprehensive risk assessment result.

[0208] By reasonably designing the knowledge graph and automatically generating Cypher query statements, this application can make full use of the semantic information in the knowledge graph, combine it with the visual object detection results, and achieve more intelligent and accurate risk judgment.

[0209] IV. Risk Judgment and Visualization Display.

[0210] The core idea of risk judgment is to combine the visual object detection results with the semantic information in the knowledge graph to comprehensively evaluate whether the target has potential risks. Specifically, according to the differences between the CSV file stored with the triple information generated during object detection and the CSV file corresponding to the sub-graph returned by the Cypher query, the association relationship between the detected target and the known risk entities is analyzed to perform risk scoring and judgment.

[0211] This application draws the bounding box of the detected target on the input image and superimposes and displays the risk judgment result in the form of text or icons for visualization display, presenting it to the user in an intuitive way for manual review and decision-making. See Figure 15 as shown.

[0212] The security warning method of this application integrates the knowledge graph and uses the YOLOv8 object detection model. Through word cloud diagrams and Python crawler technology, various unsafe behaviors are retrieved and a dataset of the operation site of the logistics park is built by itself. The YOLOv8 model is trained through the dataset. Through the real-time collected monitoring data, intelligent monitoring of the irregular behaviors at the logistics park site is realized and feedback is provided.

[0213] This application constructs an efficient and intelligent security protection system, combines the knowledge graph with image recognition technology, real-time monitors the security status in the park through image recognition technology, and integrates and analyzes a large amount of security data by combining knowledge graph technology, so as to achieve accurate identification and timely warning of potential security risks. This solution uses deep learning algorithms to automatically analyze and process the images collected by the monitoring cameras in the park, effectively identifying key information such as abnormal behaviors, potential safety hazards, and violations of regulations. At the same time, with the help of knowledge graph technology, these image recognition results are integrated with structured knowledge such as safety specifications, historical cases, and emergency plans in the park to form a dynamically updated security knowledge base. This knowledge base can not only provide strong support for security decision-making, but also continuously optimize the accuracy and timeliness of the model through continuous learning. This solution will realize the intelligent security warning function. Once potential security risks are detected, the warning mechanism will be immediately triggered, and warning information will be sent to the park management personnel and relevant departments through multiple channels and forms to ensure that potential safety hazards are disposed of in a timely manner.

[0214] Example Two.

[0215] Based on the design of the security warning method in Embodiment 1, this Embodiment 2 proposes a security warning system. The security warning system includes a data processing module, and the data processing module is used to execute the above-mentioned security warning method. The specific working process of the security warning system has been described in detail in the security warning method of Embodiment 1 and will not be elaborated here.

[0216] The security warning system of this embodiment uses a target detection model to perform target detection on the input image to obtain a target detection result, where the target detection result includes the position, category label, and confidence level of the target; constructs a triple according to the category label; queries the knowledge graph according to the constructed triple to obtain semantic information related to the entity; and obtains a risk judgment result according to the target detection result and the semantic information. The security warning system of this embodiment combines image recognition and the knowledge graph, comprehensively considers the position, category label, confidence level, and semantic information of the target, realizes accurate judgment of risks, obtains a relatively accurate risk judgment result, so as to facilitate timely and effective warning, and solves the technical problem of poor accuracy of risk judgment in the prior art.

[0217] Embodiment 3

[0218] Based on the design of the security warning system in Embodiment 2, this Embodiment 3 proposes an inspection robot. The inspection robot includes the above-mentioned security warning system.

[0219] The inspection robot mainly includes a camera, a processor, a display screen, an electric control chassis, etc. The security warning system is installed on the processor.

[0220] The inspection robot of this embodiment realizes accurate judgment of risks by installing a security warning system, obtains a relatively accurate risk judgment result, so as to facilitate timely and effective warning, and solves the technical problem of poor accuracy of risk judgment in the prior art. Moreover, the inspection robot of this embodiment can replace manual inspection to go to dangerous areas and reduce the inspection risk.

[0221] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, for those of ordinary skill in the art, it is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions required to be protected by the present invention.

Claims

1. A safety warning method, characterized in that, Including: Performing object detection on the input image using the trained object detection model to obtain the object detection result, where the object detection result includes the position, class label, and confidence of the object; Constructing a triple according to the class label; Querying the knowledge graph according to the constructed triple to obtain semantic information related to the entity; Obtaining a risk judgment result according to the semantic information and the object detection result.

2. The safety warning method according to claim 1, characterized in that, Before constructing the triple according to the class label, it further includes: Judging whether the confidence of the object reaches the set confidence level; If the confidence of the object reaches the set confidence level, constructing a triple according to the class label.

3. The safety warning method according to claim 1, wherein The object detection model is the YOLOv8 model.

4. The safety warning method according to claim 1, wherein, The security warning method further includes a model training process, specifically including: Performing label annotation on the images in the dataset, and dividing the labeled dataset into a training set, a validation set, and a test set; Training the neural network model based on the training set and the validation set; Testing the trained neural network model based on the test set to obtain the object detection model.

5. The security warning method according to claim 1, wherein Obtaining a risk judgment result according to the semantic information and the object detection result specifically includes: Respectively obtaining the risk scores corresponding to the semantic information, confidence, position, and class label; Performing weighted summation on each risk score according to its respective weight to obtain a total risk score; Obtaining the corresponding risk judgment result according to the total risk score.

6. The security warning method according to claim 5, wherein Obtaining the corresponding risk judgment result according to the total risk score specifically includes: If the total risk score is in the first score interval, the risk judgment result is the first risk level; If the total risk score is in the second score interval, the risk judgment result is the second risk level, and a reminder message is sent; If the total risk score is in the third score interval, the risk judgment result is the third risk level, and an alarm message is sent; Wherein, any value in the third score interval is greater than any value in the second score interval, Any value in the second score interval is greater than any value in the first score interval.

7. The security warning method according to claim 1, wherein Visualizing and displaying the object detection result and the risk judgment result on the input image.

8. The security warning method according to claim 5, wherein The position of the object is the bounding box coordinates of the object; Displaying the bounding box of the object on the input image, and displaying the confidence, class label, and total risk score on the bounding box.

9. Safety warning system, characterized in that, Including a data processing module, and the data processing module executes the security warning method according to any one of claims 1 to 8.

10. The inspection robot is characterized in that, Including the security warning system according to claim 9.

Images