Personnel state detection method and device, equipment, storage medium and computer program

Abstract

The invention relates to a personnel state detection method, device and equipment, a storage medium and a computer program, and the method comprises the steps: obtaining a visible light image collected by a visible light camera and a thermal imaging image collected by a thermal imaging camera, and carrying out the alignment processing of the visible light image and the thermal imaging image; a first detection result and a second detection result are obtained based on the visible light image and the thermal imaging image which are subjected to alignment processing, the first detection result and the second detection result are fused, a personnel detection result is obtained, and the personnel detection result comprises a personnel detection frame in the visible light image of the current frame; performing personnel tracking based on the personnel detection result to obtain a personnel tracking trajectory; and analyzing the state of the personnel according to the personnel detection result and the personnel tracking trajectory. The robustness and precision of detection and tracking are improved, so that the method can better adapt to complex and changeable industrial environments with low visibility and possible existence of multiple targets, and practical application requirements are met.

Description

Technical Field

[0001] The present disclosure relates to the field of visual detection technology, and in particular to a method, device, equipment, storage medium and computer program for detecting a person's state. Background Art

[0002] At present, the personnel status detection algorithms widely used in industrial environments mainly rely on a single sensor, that is, capturing personnel information through a visible light camera. The SSD (Single Shot MultiBox Detector) algorithm is usually used for personnel target detection, and combined with the inter-frame difference method for personnel tracking and status recognition.

[0003] Single sensor discrimination methods are generally limited by the defects of the corresponding data, resulting in problems such as failure to extract sufficient effective information or insufficient effective information. For example, the quality of data collected by visible light cameras is easily affected by the environment, especially in industrial environments with low visibility such as spray paint workshops. Smoke or insufficient light intensity will lead to a decrease in the quality of collected data and affect the subsequent accuracy of personnel status detection. Therefore, the existing personnel status detection algorithm has certain performance limitations when facing industrial environments with large lighting changes and complex backgrounds.

[0004] Although the SSD target detection method has an advantage in speed, its detection accuracy is relatively low, especially when dealing with small targets and complex backgrounds; at the same time, the inter-frame difference method is sensitive to illumination changes, noise and slight jitter, and is prone to false detection and missed detection. Summary of the invention

[0005] The present disclosure provides a personnel status detection method, device, equipment, storage medium and computer program to improve the robustness and accuracy of detection and tracking, so as to better adapt to complex and changeable industrial environments with low visibility and possible presence of multiple targets, and meet practical application needs.

[0006] In a first aspect, the present disclosure provides a method for detecting a person's status, comprising:

[0007] Acquire a visible light image captured by a visible light camera and a thermal imaging image captured by a thermal imaging camera, and align the visible light image and the thermal imaging image;

[0008] Based on the aligned visible light image and the thermal imaging image, respectively obtain a first detection result and a second detection result, and fuse the first detection result and the second detection result to obtain a person detection result, wherein the person detection result includes a person detection frame in the visible light image of the current frame;

[0009] Performing personnel tracking based on the personnel detection result to obtain a personnel tracking trajectory;

[0010] Analyze the status of personnel based on personnel detection results and personnel tracking trajectories.

[0011] In some embodiments, aligning the visible light image and the thermal imaging image includes:

[0012] The parameters of the visible light camera and the thermal imaging camera are obtained by using the Zhang Zhengyou calibration method, wherein the parameters include an intrinsic parameter matrix, an extrinsic parameter matrix and a distortion coefficient;

[0013] The relative position and posture of the visible light camera and the thermal imaging camera are calculated by using the parameters;

[0014] Based on the relative position and posture, the visible light image and the thermal imaging image are aligned using geometric transformation technology.

[0015] In some embodiments, obtaining a first detection result and a second detection result based on the aligned visible light image and the thermal imaging image respectively includes:

[0016] Inputting the aligned visible light image and thermal imaging image into a pre-trained target detection model to obtain a preliminary second detection result and a second detection result, respectively, wherein the preliminary second detection result and the second detection result include a person detection frame;

[0017] The non-maximum suppression technology is used to remove redundant personnel detection frames in the preliminary first detection result and the second detection result to obtain the final first detection result and the second detection result.

[0018] In some embodiments, based on the personnel detection result, a deepsort method is used to track personnel, including:

[0019] Based on the person tracking result of the k-1th visible light image, a Kalman filter algorithm is used to predict the person detection frame in the kth visible light image to obtain a person prediction frame of the kth visible light image;

[0020] Extracting a prediction feature vector of the person prediction box using a deep feature extractor;

[0021] A deep feature extractor is used to extract a detection feature vector of a person detection frame of the k-th visible light image;

[0022] Calculating the geometric distance between the person prediction frame and the person detection frame of the k-th visible light image, and the similarity between the prediction feature vector and the detection feature vector, and constructing a cost matrix based on the geometric distance and the similarity;

[0023] The cost matrix is ​​solved by using the Hungarian algorithm to optimally match the person prediction frame with the person detection frame, and the person detection frame of the k-th visible light image is associated with the previous person tracking result.

[0024] In some embodiments, analyzing the status of a person according to the person detection result and the person tracking trajectory includes:

[0025] Whether a person is in an abnormal state is analyzed based on the shape of the person detection frame, the person tracking trajectory, and the longitudinal change speed of the person detection frame.

[0026] In some embodiments, analyzing whether a person is in an abnormal state according to the shape of the person detection frame, the person tracking trajectory, and the longitudinal change speed of the person detection frame includes:

[0027] When the aspect ratio of the person detection frame is greater than the set value, the person is in a fainting state;

[0028] If the person tracking trajectory results show that the person has entered the set prohibited area, the person has intruded;

[0029] When the longitudinal change of the person detection frame in the continuous frame visible light image is greater than the set threshold, the person falls.

[0030] In a second aspect, the present disclosure provides a personnel status detection device, comprising:

[0031] An image acquisition module, used to acquire a visible light image acquired by a visible light camera and a thermal imaging image acquired by a thermal imaging camera, and align the visible light image and the thermal imaging image;

[0032] A personnel detection module, configured to obtain a first detection result and a second detection result respectively based on the aligned visible light image and the thermal imaging image, and to fuse the first detection result and the second detection result to obtain a personnel detection result, wherein the personnel detection result includes a personnel detection frame in the visible light image of the current frame;

[0033] A personnel tracking module, used to track personnel based on the personnel detection result to obtain a personnel tracking trajectory;

[0034] The personnel status analysis module is used to analyze the status of personnel based on personnel detection results and personnel tracking trajectories.

[0035] In a third aspect, the present disclosure provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the above aspects.

[0036] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method described in the above aspects when executed by a processor.

[0037] In a fifth aspect, the present disclosure provides a computer program product, including a computer program, which implements the steps of the method described in the above aspects when executed by a processor.

[0038] The present disclosure provides a personnel status detection method, device, equipment, storage medium and computer program, which obtains a visible light image collected by a visible light camera and a thermal imaging image collected by a thermal imaging camera, aligns the two, and respectively performs personnel target detection to obtain a first detection result and a second detection result, and fuses the two detection results to obtain a personnel detection result, and performs personnel tracking and personnel status analysis. It is suitable for personnel status detection in industrial environments with low visibility (for example, low visibility caused by smoke and dust when working in a spray paint workshop). Since visible light and thermal imaging data are fused to detect personnel targets, and the target detection results are tracked using a deepsort method, compared with the traditional method, the data quality of visible light cameras is reduced, and the target detection and tracking algorithms are sensitive to environmental changes. The information provided by the two types of data is fully utilized, and the advanced target detection and tracking methods are combined to significantly improve the robustness and accuracy of detection and tracking, so as to better adapt to complex and changeable industrial environments with low visibility and possible multiple targets, and meet practical application needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The present disclosure will be described in more detail below based on embodiments and with reference to the accompanying drawings:

[0040] Figure 1 A schematic diagram of the personnel status detection principle provided in an embodiment of the present disclosure;

[0041] Figure 2 The alignment processing effect diagram provided by the embodiment of the present disclosure, wherein (a) is a visible light image of the climate, and (b) is a thermal imaging image after alignment;

[0042] Figure 3 The image adjustment effect diagram of the data enhancement technology provided in the embodiment of the present disclosure, wherein (a) is the original image, (b) is the image after the contrast is changed, (c) is the image after the brightness is changed, and (d) is the image after the blur is changed;

[0043] Figure 4 A detection effect diagram of the YOLOv8 network model provided in an embodiment of the present disclosure;

[0044] Figure 5 A schematic diagram of a personnel tracking process provided by an embodiment of the present disclosure;

[0045] Figure 6 A schematic diagram of personnel tracking results provided by an embodiment of the present disclosure;

[0046] Figure 7 A schematic diagram of personnel status detection provided by an embodiment of the present disclosure;

[0047] Figure 8 A flowchart of a method for detecting a person's status is provided in an embodiment of the present disclosure.

[0048] In the drawings, the same reference numerals are used for the same components, and the drawings are not drawn to scale. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, and to fully understand and implement how the present disclosure applies technical means to solve technical problems and achieve the corresponding technical effects, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the present disclosure, not all of the embodiments. The embodiments of the present disclosure and the various features in the embodiments can be combined with each other without conflict, and the technical solutions formed are all within the scope of protection of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present disclosure.

[0050] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0051] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0052] At present, the personnel status detection algorithms widely used in industrial environments mainly rely on a single sensor, that is, capturing personnel information through a visible light camera. The SSD (Single Shot MultiBox Detector) algorithm is usually used for personnel target detection, and combined with the inter-frame difference method for personnel tracking and status recognition.

[0053] Single sensor discrimination methods are generally limited by the defects of the corresponding data, resulting in problems such as failure to extract sufficient effective information or insufficient effective information. For example, the quality of data collected by visible light cameras is easily affected by the environment, especially in industrial environments with low visibility such as spray paint workshops. Smoke or insufficient light intensity will lead to a decrease in the quality of collected data and affect the accuracy of subsequent personnel status detection.

[0054] Traditional object detection methods are usually divided into two steps: the first step is to generate candidate regions, and the second step is to classify and regress each candidate region. SSD combines the candidate region generation and classification and regression steps into one, directly extracting features from the input image and predicting bounding boxes and category probabilities. This architectural design enables it to run efficiently in real-time applications. SSD makes predictions on feature maps of different scales and is able to detect objects of different sizes, thereby enhancing the ability to detect multi-scale objects. The model complexity of SSD is moderate and suitable for resource-limited devices such as embedded systems and mobile devices.

[0055] The inter-frame difference method is a basic motion detection technology that detects moving targets in an image by comparing the differences between consecutive frames. This method is widely used because of its low computational complexity and simple implementation, especially for scenes with static backgrounds and stable lighting. In this environment, the inter-frame difference method can quickly respond to motion changes in the scene and effectively identify the movement trajectory and state changes of people.

[0056] The existing personnel status detection algorithm can realize efficient and real-time personnel status monitoring in scenes with static background and stable lighting.

[0057] Although the SSD target detection method has an advantage in speed, its detection accuracy is relatively low, especially when dealing with small targets and complex backgrounds; at the same time, the inter-frame difference method is sensitive to illumination changes, noise and slight jitter, and is prone to false detection and missed detection.

[0058] Therefore, the existing personnel status detection algorithms have certain performance limitations when facing industrial environments with large lighting changes and complex backgrounds.

[0059] Embodiment 1

[0060] Figure 8 A flow chart of a method for detecting a person's status provided in an embodiment of the present disclosure is provided. Figure 1 Figure 1 is a schematic diagram of the principle of personnel status detection. Figure 8 As shown, a method for detecting a person's status includes:

[0061] Step S11, obtaining a visible light image captured by a visible light camera and a thermal imaging image captured by a thermal imaging camera, and aligning the visible light image and the thermal imaging image.

[0062] Considering that the quality of visible light images is easily affected by environmental factors such as smoke, light intensity, etc., in order to improve the reliability of the algorithm in complex and changing environments, this embodiment introduces infrared thermal imaging images. Since the shooting angles of visible light cameras and infrared cameras are different, in order to better integrate the image data features of the two modalities, the two types of image data need to be aligned.

[0063] In some embodiments, the visible light image and the thermal imaging image are aligned, including:

[0064] Step S11a, using Zhang Zhengyou calibration method to obtain parameters of the visible light camera and the thermal imaging camera, the parameters including an intrinsic parameter matrix, an extrinsic parameter matrix and a distortion coefficient;

[0065] Step S11b, calculating the relative position and posture of the visible light camera and the thermal imaging camera using the above parameters;

[0066] Step S11c: Based on the relative posture, the visible light image and the thermal imaging image are aligned using geometric transformation technology.

[0067] This embodiment uses Zhang Zhengyou's calibration method to obtain the camera's intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient. Through these parameters, the relative position of the camera, that is, the rotation matrix and translation vector between the cameras, is calculated. The visible light image and the infrared thermal imaging image are aligned using geometric transformation technology. This image alignment process ensures data fusion in the same coordinate system, thereby improving the robustness and accuracy of subsequent target detection. In an example, the effect of the alignment process is as follows: Figure 2 As shown, (a) is the visible light image of the climate, and (b) is the aligned thermal imaging image.

[0068] Step S12: obtaining a first detection result and a second detection result based on the aligned visible light image and the thermal imaging image, respectively, and fusing the first detection result and the second detection result to obtain a person detection result, wherein the person detection result includes a person detection frame in the visible light image of the current frame.

[0069] In this embodiment, the target detection adopts the YOLOv8 network, and a data set is pre-built to train the YOLOv8 network. The input of the YOLOv8 network is a visible light image or a thermal imaging image, and the output is a detection frame covering the target (person). The first YOLOv8 network and the second YOLOv8 network are trained for the visible light image and the thermal imaging image respectively. When detecting the target, the aligned visible light image and the thermal imaging image are input into the corresponding first YOLOv8 network and the second YOLOv8 network respectively to obtain the corresponding person detection frame.

[0070] In order to improve the accuracy and robustness of target detection, data enhancement technology is used to expand the original image data when constructing the data set, and the model's adaptability to different environments is improved by adjusting the image brightness, grayscale, blur, etc. In an example, the image adjustment effect of data enhancement technology is as follows: Figure 3 As shown, (a) is the original image, (b) is the image after changing the contrast, (c) is the image after changing the brightness, and (d) is the image after changing the blur.

[0071] During the YOLOv8 network training process, the dropout layer is enabled to avoid overfitting and improve the generalization and scalability of the model. In an example, the detection effect of the YOLOv8 network model is as follows: Figure 4 shown.

[0072] In some embodiments, obtaining a first detection result and a second detection result based on the aligned visible light image and the thermal imaging image, respectively, includes:

[0073] Step S12a: Input the aligned visible light image and thermal imaging image into the pre-trained target detection model to obtain a preliminary second detection result and a second detection result, respectively. The preliminary second detection result and the second detection result include a person detection frame. The person detection frame indicates the location of the person. In some cases, the detection result also includes a category. For example, if the detected object is a person, the category is a person.

[0074] Step S12b: using non-maximum suppression technology, remove redundant personnel detection frames in the preliminary first detection result and the second detection result to obtain the final first detection result and the second detection result.

[0075] The visible light image and thermal image are normalized, the image size is adjusted, and then input into the YOLOv8 network to obtain the category and position information of the detection frame of the visible light image and infrared thermal image. The non-maximum suppression (NMS) technology is used to post-process the two detection frames, remove redundant detection frames, and output the final person detection frame position and category information.

[0076] In an example, the post-processing input: detection boxes, is composed of two detection boxes, each of which contains (x1, y1, x2, y2, confidence), where (x1, y1) and (x2, y2) are the coordinates of the upper left corner and lower right corner of the detection box, respectively. The confidence confidence represents the predicted credibility of each detection box, the IoU threshold is set to 0.3, and the confidence threshold is set to 0.7.

[0077] Correspondingly, the pseudo code for post-processing is as follows:

[0078] #Step 1: Remove detection boxes with confidence lower than 0.7

[0079] boxes=[box for box in boxes if box.confidence>=confidence_threshold]

[0080] #Step 2: Sort by confidence from high to low

[0081] boxes=sort_by_confidence(boxes)

[0082] #Step 3: Define a list to store the final retained detection box

[0083] keep_boxes = []

[0084] #Step 4: Loop through each detection box

[0085] while boxes is not empty:

[0086] #Take out the detection box with the highest confidence

[0087] current_box = boxes[0]

[0088] #Add the current frame to the reserved list

[0089] keep_boxes.append(current_box)

[0090] #Initialize a new list to store the temporarily retained detection boxes new_boxes=[]

[0091] #Step 5: Compare the remaining detection boxes and calculate IoU with the current box

[0092] for box in boxes[1:]:

[0093] iou=calculate_iou(current_box,box)

[0094] #If IoU is less than the threshold 0.3, keep the detection box

[0095] if iou <iou_threshold:

[0096] new_boxes.append(box)

[0097] #Update the list of pending boxes with the reserved boxes

[0098] boxes=new_boxes

[0099] #Return the retained detection box

[0100] return keep_boxes

[0101] #IoU calculation function, input two detection boxes, output IoU value function calculate_iou(box1,box2):

[0102] #Calculate the upper left and lower right corners of the overlapping area

[0103] x1_overlap=max(box1.x1,box2.x1)

[0104] y1_overlap=max(box1.y1,box2.y1)

[0105] x2_overlap=min(box1.x2,box2.x2)

[0106] y2_overlap=min(box1.y2,box2.y2)

[0107] # Calculate the area of ​​the overlapping region

[0108] overlap_area=max(0,x2_overlap-x1_overlap)*max(0,y2_overlap-y1_overlap)

[0109] #Calculate the area of ​​each detection box

[0110] box1_area=(box1.x2-box1.x1)*(box1.y2-box1.y1)

[0111] box2_area=(box2.x2-box2.x1)*(box2.y2-box2.y1)

[0112] #Calculate IoU value

[0113] iou=overlap_area / (box1_area+box2_area-overlap_area)

[0114] return

[0115] Step S13: tracking a person based on the person detection result to obtain a person tracking trajectory.

[0116] The deepsort method includes Kalman filter algorithm, deep feature extractor and Hungarian algorithm.

[0117] In some embodiments, Figure 5 As shown, based on the personnel detection results, the deepsort method is used to track personnel, which may include:

[0118] Step S13a: Based on the person tracking result of the k-1th visible light image frame, a Kalman filter algorithm is used to predict the person detection frame in the kth visible light image frame to obtain a person prediction frame of the kth visible light image frame.

[0119] Step S13b, using a deep feature extractor to extract a prediction feature vector of a person prediction frame;

[0120] Step S13c, using a deep feature extractor to extract a detection feature vector of a person detection frame of the k-th visible light image;

[0121] Step S13d, calculating the geometric distance between the person prediction frame and the person detection frame of the k-th visible light image, and the similarity between the prediction feature vector and the detection feature vector, and constructing a cost matrix based on the geometric distance and the similarity;

[0122] Step S13e: Use the Hungarian algorithm to solve the cost matrix to optimally match the person prediction frame with the person detection frame, and associate the person detection frame of the k-th visible light image with the previous person tracking result.

[0123] In actual application scenarios, similar targets may appear. By combining the motion prediction of Kalman filtering and the appearance features of deep features to construct feature vectors, multiple similar targets can be effectively distinguished. This method is not only suitable for static backgrounds, but also maintains good tracking performance in dynamic and complex backgrounds, and performs well when the target is occluded. The tracking results are shown in Figure 2. Figure 6 shown.

[0124] Step S14: Analyze the status of the personnel according to the personnel detection results and the personnel tracking trajectory.

[0125] In some implementations, analyzing the status of a person based on the person detection result and the person tracking trajectory includes: analyzing whether the person is in an abnormal state based on the shape of the person detection frame, the person tracking trajectory, and the longitudinal change speed of the person detection frame.

[0126] In some implementations, analyzing whether a person is in an abnormal state based on the shape of the person detection frame, the person tracking trajectory, and the longitudinal change speed of the person detection frame includes:

[0127] When the aspect ratio of the person detection frame is greater than the set value, the person is in a fainting state;

[0128] If the person tracking trajectory results show that the person has entered the set prohibited area, the person has intruded;

[0129] When the longitudinal change of the person detection frame in the continuous frame visible light image is greater than the set threshold, the person falls.

[0130] In one example, if Figure 7 As shown, person fainting detection: determine the aspect ratio of each person detection frame. If the aspect ratio is greater than 1, the person is considered to have fainted and an alarm is generated. Person intrusion detection: mark the prohibited area in the image in the form of polygonal data (Polygon), determine the trajectory of each person detection frame, and if it appears in the prohibited area, the person is considered to have invaded and an alarm is generated. Person fall detection: according to the trajectory of each person detection frame, calculate the change in the y value (image vertical coordinate) of each person within 3s. If it is greater than the set threshold, the person is considered to have fallen and an alarm is generated.

[0131] The method of this embodiment is a personnel status detection solution in an industrial environment with low visibility (for example, low visibility caused by smoke when working in a spray paint workshop). The personnel target (represented by a personnel detection frame) is extracted based on the fusion of detection results of visible light and thermal imaging image modal data, and target tracking is achieved by combining deep learning and target tracking algorithms. Personnel status recognition is achieved based on target detection and tracking results, and potential dangerous behaviors and abnormal states are identified.

[0132] Since visible light and thermal imaging data are integrated to detect personnel targets, and the deepsort method is used to track the target detection results, compared with the traditional method of using visible light cameras, the data quality is reduced, and the target detection and tracking algorithms are sensitive to environmental changes. This method makes full use of the information provided by the two types of data, and combines advanced target detection and tracking methods to significantly improve the robustness and accuracy of detection and tracking, so as to better adapt to complex and changeable industrial environments with low visibility and possible multiple targets, and meet practical application needs.

[0133] Embodiment 2

[0134] Based on the above embodiment, a personnel status detection device is provided, comprising:

[0135] An image acquisition module is used to acquire a visible light image acquired by a visible light camera and a thermal imaging image acquired by a thermal imaging camera, and align the visible light image and the thermal imaging image;

[0136] A personnel detection module, used to obtain a first detection result and a second detection result respectively based on the aligned visible light image and the thermal imaging image, and to fuse the first detection result and the second detection result to obtain a personnel detection result, wherein the personnel detection result includes a personnel detection frame in the visible light image of the current frame;

[0137] A personnel tracking module is used to track personnel based on personnel detection results and obtain personnel tracking trajectories;

[0138] The personnel status analysis module is used to analyze the status of personnel based on personnel detection results and personnel tracking trajectories.

[0139] In some embodiments, the visible light image and the thermal imaging image are aligned, including: using Zhang Zhengyou calibration method to obtain parameters of the visible light camera and the thermal imaging camera, the parameters including an intrinsic parameter matrix, an extrinsic parameter matrix and a distortion coefficient; using the above parameters, calculating the relative posture of the visible light camera and the thermal imaging camera; based on the relative posture, using geometric transformation technology to align the visible light image and the thermal imaging image.

[0140] In this embodiment, the target detection adopts the YOLOv8 network, and a data set is pre-built to train the YOLOv8 network. The input of the YOLOv8 network is a visible light image or a thermal imaging image, and the output is a detection frame covering the target (person). The first YOLOv8 network and the second YOLOv8 network are trained for the visible light image and the thermal imaging image respectively. When detecting the target, the aligned visible light image and the thermal imaging image are input into the corresponding first YOLOv8 network and the second YOLOv8 network respectively to obtain the corresponding person detection frame.

[0141] During the YOLOv8 network training process, the dropout layer is enabled to avoid overfitting and improve the generalization and scalability of the model.

[0142] In some embodiments, obtaining a first detection result and a second detection result based on the aligned visible light image and the thermal imaging image, respectively, includes: inputting the aligned visible light image and the thermal imaging image into a pre-trained target detection model, respectively obtaining a preliminary second detection result and a second detection result, wherein the preliminary second detection result and the second detection result include a person detection frame. Using a non-maximum suppression technique, redundant person detection frames in the preliminary first detection result and the second detection result are removed to obtain a final first detection result and a second detection result.

[0143] The visible light image and thermal image are normalized, the image size is adjusted, and then input into the YOLOv8 network to obtain the category and position information of the detection frame of the visible light image and infrared thermal image. The non-maximum suppression (NMS) technology is used to post-process the two detection frames, remove redundant detection frames, and output the final detection frame position and category information.

[0144] In some embodiments, based on the personnel detection result, using the deepsort method to track personnel may include: based on the personnel tracking result of the k-1th visible light image, using the Kalman filter algorithm to predict the personnel detection frame in the kth visible light image, and obtaining the personnel prediction frame of the kth visible light image. Using a deep feature extractor to extract the predicted feature vector of the personnel prediction frame; using the deep feature extractor to extract the detection feature vector of the personnel detection frame of the kth visible light image; calculating the geometric distance between the personnel prediction frame and the personnel detection frame of the kth visible light image, and the similarity between the predicted feature vector and the detection feature vector, and constructing a cost matrix based on the geometric distance and the similarity; using the Hungarian algorithm to solve the cost matrix so that the personnel prediction frame and the personnel detection frame are optimally matched, and the personnel detection frame of the kth visible light image is associated with the previous personnel tracking result.

[0145] In some implementations, analyzing the status of a person based on the person detection result and the person tracking trajectory includes: analyzing whether the person is in an abnormal state based on the shape of the person detection frame, the person tracking trajectory, and the longitudinal change speed of the person detection frame.

[0146] In some implementations, analyzing whether a person is in an abnormal state based on the shape of the person detection frame, the person tracking trajectory, and the longitudinal change speed of the person detection frame includes:

[0147] When the aspect ratio of the person detection frame is greater than the set value, the person is in a fainting state;

[0148] If the person tracking trajectory results show that the person has entered the set prohibited area, the person has intruded;

[0149] When the longitudinal change of the person detection frame in the continuous frame visible light image is greater than the set threshold, the person falls.

[0150] In one example, person fainting detection: determine the aspect ratio of each person detection frame. If the aspect ratio is greater than 1, the person is considered to have fainted and an alarm is generated. Person intrusion detection: mark the prohibited area in the image in the form of polygonal data (Polygon), determine the trajectory of each person detection frame, and if it appears in the prohibited area, the person is considered to have invaded and an alarm is generated. Person fall detection: based on the trajectory of each person detection frame, calculate the change in the y value (image vertical coordinate) of each person within 3 seconds. If it is greater than the set threshold, the person is considered to have fallen and an alarm is generated.

[0151] This embodiment is a personnel status detection solution in an industrial environment with low visibility (for example, low visibility caused by smoke when working in a spray paint shop). The personnel target (represented by a personnel detection frame) is extracted based on the fusion of the detection results of the two modal data of visible light and thermal imaging images, and target tracking is achieved by combining deep learning and target tracking algorithms. The personnel status is recognized based on the target detection and tracking results, and potential dangerous behaviors and abnormal states are identified.

[0152] Since visible light and thermal imaging data are integrated to detect personnel targets, and the deepsort method is used to track the target detection results, compared with the traditional method, the data quality of using visible light cameras is reduced, and the target detection and tracking algorithms are sensitive to environmental changes. This embodiment makes full use of the information provided by the two types of data, and combines advanced target detection and tracking methods to significantly improve the robustness and accuracy of detection and tracking, so as to better adapt to complex and changeable industrial environments with low visibility and possible multiple targets, and meet actual application needs.

[0153] Embodiment 3

[0154] On the basis of the above embodiments, this embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the above embodiments.

[0155] In some implementations of this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the above embodiment are implemented.

[0156] In some implementations of this embodiment, a computer program product is provided, including a computer program, which implements the steps of the method described in the above embodiment when executed by a processor.

[0157] The processor may include, but is not limited to, one or more processors or microprocessors, etc. Each processor may be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components to execute the methods in the above embodiments.

[0158] The computer-readable storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, and the computer-readable storage medium may include but is not limited to, for example, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (e.g., hard disk, floppy disk, solid-state drive, removable disk, CD-ROM, DVD-ROM, Blu-ray disc, etc.).

[0159] The computer-readable storage medium may also store at least one computer executable program / instruction, which may be, for example, a computer-readable instruction. The computer-readable storage medium includes, but is not limited to, for example, a volatile memory and / or a non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The computer-readable storage medium may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device runs the computer-readable instructions stored on the computer-readable storage medium, the various methods described above may be performed.

[0160] In addition, the computer device may also include (but not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (eg, keyboard, mouse, speaker, etc.), etc.

[0161] The processor may communicate with external devices via an I / O bus via a wired or wireless network.

[0162] In one embodiment, the at least one computer executable instruction may also be compiled into or constitute a software product / computer program product, wherein one or more computer executable instructions are executed by a processor to perform the various functions and / or method steps in the embodiments described in the present technology.

[0163] In the embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0164] It should be noted that in the present disclosure, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element limited by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0165] Although the embodiments disclosed in the present disclosure are as above, the above contents are only embodiments adopted for facilitating the understanding of the present disclosure and are not intended to limit the present disclosure. Any technician in the technical field to which the present disclosure belongs can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present disclosure, but the scope of patent protection of the present disclosure shall still be subject to the scope defined in the attached claims.

Claims

1. A method for detecting a person's status, characterized in that: include: Acquire a visible light image captured by a visible light camera and a thermal imaging image captured by a thermal imaging camera, and align the visible light image and the thermal imaging image; Based on the aligned visible light image and the thermal imaging image, respectively obtain a first detection result and a second detection result, and fuse the first detection result and the second detection result to obtain a person detection result, wherein the person detection result includes a person detection frame in the visible light image of the current frame; Performing personnel tracking based on the personnel detection result to obtain a personnel tracking trajectory; Analyze the status of personnel based on personnel detection results and personnel tracking trajectories.

2. The method according to claim 1, characterized in that The visible light image and the thermal imaging image are aligned, comprising: The parameters of the visible light camera and the thermal imaging camera are obtained by using the Zhang Zhengyou calibration method, wherein the parameters include an intrinsic parameter matrix, an extrinsic parameter matrix and a distortion coefficient; The relative position and posture of the visible light camera and the thermal imaging camera are calculated by using the parameters; Based on the relative position and posture, the visible light image and the thermal imaging image are aligned using geometric transformation technology.

3. The method according to claim 1, characterized in that The first detection result and the second detection result are obtained based on the aligned visible light image and the thermal imaging image, respectively, including: Inputting the aligned visible light image and thermal imaging image into a pre-trained target detection model to obtain a preliminary second detection result and a second detection result, respectively, wherein the preliminary second detection result and the second detection result include a person detection frame; The non-maximum suppression technology is used to remove redundant personnel detection frames in the preliminary first detection result and the second detection result to obtain the final first detection result and the second detection result.

4. The method according to claim 1, characterized in that: Based on the personnel detection result, the deepsort method is used to track personnel, including: Based on the person tracking result of the k-1th visible light image, a Kalman filter algorithm is used to predict the person detection frame in the kth visible light image to obtain a person prediction frame of the kth visible light image; Extracting a prediction feature vector of the person prediction box using a deep feature extractor; A deep feature extractor is used to extract a detection feature vector of a person detection frame of the k-th visible light image; Calculating the geometric distance between the person prediction frame and the person detection frame of the k-th visible light image, and the similarity between the prediction feature vector and the detection feature vector, and constructing a cost matrix based on the geometric distance and the similarity; The cost matrix is ​​solved by using the Hungarian algorithm to optimally match the person prediction frame with the person detection frame, and the person detection frame of the k-th visible light image is associated with the previous person tracking result.

5. The method according to claim 1, characterized in that The analyzing of the status of personnel according to the personnel detection results and the personnel tracking trajectory includes: Whether a person is in an abnormal state is analyzed based on the shape of the person detection frame, the person tracking trajectory, and the longitudinal change speed of the person detection frame.

6. The method according to claim 5, characterized in that The analyzing whether a person is in an abnormal state according to the shape of the person detection frame, the person tracking trajectory and the longitudinal change speed of the person detection frame includes: When the aspect ratio of the person detection frame is greater than the set value, the person is in a fainting state; If the person tracking trajectory results show that the person has entered the set prohibited area, the person has intruded; When the longitudinal change of the person detection frame in the continuous frame visible light image is greater than the set threshold, the person falls.

7. A personnel status detection device, characterized in that: include: An image acquisition module, used to acquire a visible light image acquired by a visible light camera and a thermal imaging image acquired by a thermal imaging camera, and align the visible light image and the thermal imaging image; A personnel detection module, configured to obtain a first detection result and a second detection result respectively based on the aligned visible light image and the thermal imaging image, and to fuse the first detection result and the second detection result to obtain a personnel detection result, wherein the personnel detection result includes a personnel detection frame in the visible light image of the current frame; A personnel tracking module, used to track personnel based on the personnel detection result to obtain a personnel tracking trajectory; The personnel status analysis module is used to analyze the status of personnel based on personnel detection results and personnel tracking trajectories.

8. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images