A lightweight human body detection and distance measurement method and system suitable for stage lamps

Abstract

The application discloses a lightweight human body detection and distance measurement method and system suitable for stage lamps, and is used for solving the technical problem that the existing human body detection method is greatly affected by the environment, is prone to "missed judgment", and has low detection precision. The method comprises the following steps: acquiring left and right visible light images and scene infrared images collected by a binocular camera, correcting the distortion of the images by using camera calibration parameters, adopting an improved YOLO11n model, outputting a plurality of bounding box regression distributions according to the corrected left visible light image and the corrected scene infrared image, generating a left image candidate detection box in combination with a preset anchor box, obtaining a left image accurate detection box through non-maximum suppression and performing pixel expansion, and finally outputting a left image target region of interest; then, a right image target region of interest is generated in combination with the corrected right visible light image, and the distance between the human body and the camera is detected through a stereo matching algorithm.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image processing technology, and in particular to a lightweight human body detection and ranging method and system suitable for stage lighting. Background Technology

[0002] Stage lighting fixtures, with laser lights as their core component, have become key equipment for creating immersive visual effects due to their high brightness and high directional optical characteristics.

[0003] However, the high power density of laser lights (especially those with a power of ≥300mW) poses significant safety risks to the human body: when the laser beam directly shines into the eyes, it can burn the photoreceptor cells of the retina, leading to temporary or permanent vision damage; even when shining on the skin, prolonged focusing can cause localized thermal burns, and the glare effect of strong lasers can interfere with the visual judgment of actors and staff, increasing the probability of stage accidents. Therefore, laser lights must strictly avoid areas with people during operation, ensuring that the projection range does not intersect with the area where people are moving; this is a core requirement for safe stage operation.

[0004] Existing human detection methods use infrared sensors to detect human body heat signals, convert them into electrical signals, and compare them with a preset "human infrared radiation intensity threshold" to complete the human body detection. However, stage scenes contain a large number of interference sources with infrared radiation characteristics similar to those of the human body. Furthermore, when the stage ambient temperature is high or low, the temperature difference between the human body and the environment narrows, and the intensity of the human infrared signal received by the sensor will be significantly weakened, easily leading to "missed detections" and resulting in low detection accuracy. Summary of the Invention

[0005] This invention provides a lightweight human body detection and ranging method and system suitable for stage lighting fixtures, which solves the technical problem that existing human body detection methods are prone to "missed detection" due to the great influence of the environment, resulting in low detection accuracy.

[0006] The first aspect of this invention provides a lightweight human body detection and ranging method suitable for stage lighting fixtures, comprising:

[0007] Acquire left and right visible light images and scene infrared images captured by the binocular camera;

[0008] Based on the camera calibration parameters of the binocular camera, distortion correction is performed on the left and right visible light images and the scene infrared image, and the corrected left and right visible light images and the corrected scene infrared image are output.

[0009] An improved YOLO11n model is used to generate the target region of interest in the left image based on the corrected left visible light image and the corrected scene infrared image in the corrected left and right visible light images.

[0010] Based on the target region of interest in the left image and the corrected right visible light image in the corrected left and right visible light images, the target region of interest in the right image is generated;

[0011] A stereo matching algorithm is used to detect human distance based on the target region of interest in the right image and the target region of interest in the left image, and outputs the distance between the human body and the stereo camera.

[0012] Optionally, the improved YOLO11n model includes a backbone network, a neck network, and a detection head based on adaptive multimodal fusion; the generation of the target region of interest in the left image using the improved YOLO11n model based on the corrected left visible light image and the corrected scene infrared image in the corrected left and right visible light images includes:

[0013] The corrected left visible light image and the corrected scene infrared image are adaptively fused using the backbone network based on adaptive multimodal fusion to output a target multimodal fusion feature map.

[0014] The target multimodal fusion feature map is fused using the neck network to generate a multi-scale fusion feature map.

[0015] The multi-scale fused feature map is used as the input to the detection head, and multiple bounding box regression distributions are output.

[0016] Based on the preset anchor boxes and the regression distribution of the multiple bounding boxes, multiple candidate detection boxes for the left image are generated;

[0017] A non-maximum suppression algorithm is used to filter multiple candidate detection boxes in the left image, outputting accurate detection boxes in the left image, and pixel expansion is performed on the accurate detection boxes in the left image to output the initial region of interest in the left image.

[0018] The number of initial regions of interest in the left image is counted, and the number of initial regions of interest in the left image is compared with a preset threshold for the number of regions of interest.

[0019] If the number of initial regions of interest in the left image is greater than the preset threshold for the number of regions of interest, then the spacing between adjacent initial regions of interest in the left image is calculated and compared with the preset pixel threshold respectively.

[0020] The adjacent initial regions of interest in the left image with a spacing smaller than the preset pixel threshold are merged to output the target region of interest in the left image;

[0021] If the number of initial regions of interest in the left image is less than or equal to the preset threshold for the number of regions of interest, then the initial regions of interest in the left image are taken as the target regions of interest in the left image.

[0022] Optionally, the backbone network based on adaptive multimodal fusion includes a convolutional module, a bottleneck convolutional module, a convolutional block with parallel spatial attention, a multimodal attention concatenation module, and a spatial pyramid pooling module; the target multimodal fusion feature map includes a first fusion feature map, a second fusion feature map, and a third fusion feature map; the step of using the backbone network based on adaptive multimodal fusion to adaptively fuse the corrected left visible light image and the corrected scene infrared image, and outputting the target multimodal fusion feature map, includes:

[0023] The corrected left visible light image and the corrected scene infrared image are convolved by two concatenated convolution modules to output the first left image convolution feature map and the first scene convolution feature map.

[0024] The bottleneck convolution module is used to perform cross-stage convolution on the first left image convolution feature map and the first scene convolution feature map respectively, and outputs the first left image cross-stage convolution feature map and the first scene cross-stage convolution feature map;

[0025] The first left image cross-stage convolutional feature map and the first scene cross-stage convolutional feature map are input into the multimodal attention splicing module for mid-term fusion, and the first left image mid-term fusion feature map is output. The first left image mid-term fusion feature map and the first scene cross-stage convolutional feature map are then convolved by the convolution module to output the second left image convolutional feature map and the second scene convolutional feature map.

[0026] The bottleneck convolution module is used to perform cross-stage convolution on the second left image convolution feature map and the second scene convolution feature map respectively, and outputs the second left image cross-stage convolution feature map and the second scene cross-stage convolution feature map;

[0027] The second left image cross-stage convolutional feature map and the second scene cross-stage convolutional feature map are input into the multimodal attention splicing module for mid-term fusion, and the first fused feature map is output. The first fused feature map and the second scene cross-stage convolutional feature map are then convolved by the convolution module to output the third left image convolutional feature map and the third scene convolutional feature map.

[0028] The bottleneck convolution module is used to perform cross-stage convolution on the third left image convolution feature map and the third scene convolution feature map respectively, and outputs the third left image cross-stage convolution feature map and the third scene cross-stage convolution feature map;

[0029] The third left image cross-stage convolutional feature map and the third scene cross-stage convolutional feature map are input into the multimodal attention splicing module for mid-term fusion, and the second fused feature map is output. The second fused feature map and the third scene cross-stage convolutional feature map are then convolved by the convolution module to output the fourth left image convolutional feature map and the fourth scene convolutional feature map.

[0030] The bottleneck convolution module is used to perform cross-stage convolution on the fourth left image convolution feature map and the fourth scene convolution feature map respectively, and outputs the fourth left image cross-stage convolution feature map and the fourth scene cross-stage convolution feature map;

[0031] The cross-stage convolutional feature map of the fourth left image and the cross-stage convolutional feature map of the fourth scene are input into the multimodal attention splicing module for mid-term fusion, and the mid-term fusion feature map of the second left image is output.

[0032] The spatial pyramid pooling module is used to pool the mid-term fusion feature map of the second left image, and the pooled feature map is output.

[0033] The pooled feature map is used as input to a convolutional block with parallel spatial attention to output a third fused feature map.

[0034] Optionally, the multimodal attention stitching module includes convolutional layers, ReLU activation function layers, and Sigmoid activation function layers; the step of inputting the first left image cross-stage convolutional feature map and the first scene cross-stage convolutional feature map into the multimodal attention stitching module for intermediate fusion, and outputting the first left image intermediate fused feature map, includes:

[0035] The first left image cross-stage convolutional feature map and the first scene cross-stage convolutional feature map are concatenated to output the concatenated feature map;

[0036] A convolutional layer is used to perform a convolution operation on the stitched feature map to generate a first convolutional feature map;

[0037] The first convolutional feature map is linearly transformed by a ReLU activation function layer, and the transformed feature map is output.

[0038] A convolutional layer is used to perform a convolution operation on the transformed feature map to output a second convolutional feature map;

[0039] The second convolutional feature map is used as the input to the Sigmoid activation function layer, and the output is a convolutional feature probability map.

[0040] The convolutional feature probability map is multiplied by the cross-stage convolutional feature map of the first left image to generate the mid-term fusion feature map of the first left image.

[0041] Optionally, the step of using a stereo matching algorithm to perform human distance detection based on the target region of interest in the right image and the target region of interest in the left image, and outputting the distance between the human body and the stereo camera, includes:

[0042] Multi-scale fusion feature extraction is performed on the target region of interest in the right image and the target region of interest in the left image respectively to generate the region of interest features in the left image and the right image.

[0043] Based on the region of interest features in the left and right images, a 3D cost volume is constructed.

[0044] Based on the features of the region of interest in the left image and the 3D cost volume, cost aggregation is performed to generate an aggregated cost volume;

[0045] Based on the aggregate cost body, determine the multiplication cost body;

[0046] Perform disparity regression on the multiplicative cost volume to generate initial disparity values ​​for the region of interest;

[0047] Based on the target region of interest in the left image, the disparity value of the initial region of interest is refined to generate the disparity values ​​of corresponding pixels in the left and right images;

[0048] The distance between the human body and the binocular camera is calculated based on the parallax values ​​of the corresponding pixels in the left and right images.

[0049] Optionally, the calculation process for the human body depth value is as follows:

[0050] ;

[0051] in, The depth value represents the distance between the human body and the binocular camera. The focal length of the binocular camera; The baseline for the binocular camera; The x-coordinate of the human body on the left camera's imaging plane; The x-coordinate of the human body on the imaging plane of the right camera; These are the disparity values ​​of corresponding pixels in the left and right images.

[0052] The second aspect of this invention provides a lightweight human body detection and ranging system suitable for stage lighting fixtures, comprising:

[0053] The acquisition module is used to acquire the left and right visible light images and scene infrared images captured by the binocular camera;

[0054] The correction module is used to perform distortion correction on the left and right visible light images and the scene infrared image based on the camera calibration parameters of the binocular camera, and output the corrected left and right visible light images and the corrected scene infrared image.

[0055] The left image region of interest generation module is used to generate the target region of interest in the left image based on the corrected left visible light image and the corrected scene infrared image in the corrected left and right visible light images using an improved YOLO11n model;

[0056] The right image region of interest generation module is used to generate the right image region of interest based on the left image target region of interest and the corrected right visible light image in the corrected left and right visible light images;

[0057] The human detection module is used to perform human distance detection based on the target region of interest in the right image and the target region of interest in the left image using a stereo matching algorithm, and outputs the distance between the human body and the stereo camera.

[0058] A computer device provided in a third aspect of the present invention includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the lightweight human body detection and ranging method applicable to stage lighting as described in any of the preceding claims.

[0059] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the steps of the lightweight human body detection and ranging method for stage lighting as described in any of the preceding claims.

[0060] The fifth aspect of the present invention provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein, when the program instructions are executed by a computer, the computer performs the steps of the lightweight human body detection and ranging method applicable to stage lighting as described in any of the preceding claims.

[0061] As can be seen from the above technical solutions, the present invention has the following advantages:

[0062] The above-mentioned technical solution of the present invention provides a lightweight human body detection and ranging method suitable for stage lighting, which acquires left and right visible light images and scene infrared images captured by a binocular camera; performs distortion correction on the left and right visible light images and scene infrared images based on the camera calibration parameters of the binocular camera, and outputs the corrected left and right visible light images and scene infrared images; uses an improved YOLOv11n model to generate the target region of interest in the left image based on the corrected left visible light image and the corrected scene infrared image in the corrected left and right visible light images; generates the target region of interest in the right image based on the target region of interest in the left image and the corrected right visible light image in the corrected left and right visible light images; and uses a stereo matching algorithm to generate the target region of interest in the right image and the target region of interest in the left image based on the target region of interest in the corrected left and right visible light images. The system performs human distance detection within the target region of interest (ROI) and outputs the distance between the human body and the binocular camera. Based on this approach, the invention utilizes an improved YOLO11n model to adaptively integrate effective features from visible light and infrared images. Even in scenarios prone to missed detection by traditional methods, such as direct stage lighting or fluctuating ambient temperatures, the system can still stably capture human features through the anti-interference properties of infrared images and the detailed supplementation from visible light images. Simultaneously, the distortion correction process ensures the accuracy of image pixel positions, avoiding human contour shifts caused by lens distortion and providing a reliable image foundation for subsequent feature extraction. The dynamically associated left and right ROIs further narrow the computational range of stereo matching, reducing interference from complex stage backgrounds and thus improving detection accuracy. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 The flowchart illustrates the steps of a lightweight human body detection and ranging method suitable for stage lighting fixtures provided in Embodiment 1 of the present invention.

[0065] Figure 2 This is a schematic diagram of the framework of the deep learning-based target detection method provided in Embodiment 1 of the present invention;

[0066] Figure 3 This is a schematic diagram of the underlying principle of the YOLO algorithm provided in Embodiment 1 of the present invention;

[0067] Figure 4 This is a block diagram of the YOLO11 algorithm provided in Embodiment 1 of the present invention;

[0068] Figure 5This is a schematic diagram of a human body under stage lighting according to Embodiment 1 of the present invention;

[0069] Figure 6 This is a schematic diagram showing the comparison of visible light and infrared images provided in Embodiment 1 of the present invention;

[0070] Figure 7 This is a schematic diagram of the multimodal fusion algorithm type provided in Embodiment 1 of the present invention;

[0071] Figure 8 This is a schematic diagram of the structure of the backbone network based on adaptive multimodal fusion provided in Embodiment 1 of the present invention;

[0072] Figure 9 This is a diagram of the dynamic ROI (Region of Interest) association design scheme provided in Embodiment 1 of the present invention;

[0073] Figure 10 This is a block diagram of the LightStereo algorithm provided in Embodiment 1 of the present invention;

[0074] Figure 11 This is a flowchart of the stereo matching algorithm provided in Embodiment 1 of the present invention;

[0075] Figure 12 This is a schematic diagram of the binocular ranging principle provided in Embodiment 1 of the present invention;

[0076] Figure 13 This is a roadmap of existing detection and ranging technologies provided in Embodiment 1 of the present invention;

[0077] Figure 14 This is a roadmap of a highly integrated detection and ranging algorithm provided in Embodiment 1 of the present invention;

[0078] Figure 15 Figure (a) shows the test visualization results of the dataset provided in Embodiment 1 of the present invention.

[0079] Figure 16 Figure (b) shows the visualization results of the dataset test provided in Embodiment 1 of the present invention.

[0080] Figure 17 This is a visualization of the dataset test results provided in Embodiment 1 of the present invention (c);

[0081] Figure 18 This is a structural block diagram of a lightweight human body detection and ranging system suitable for stage lighting provided in Embodiment 2 of the present invention. Detailed Implementation

[0082] This invention provides a lightweight human body detection and ranging method and system suitable for stage lighting fixtures, which solves the technical problem that existing human body detection methods are prone to "missed detections" due to the great influence of the environment, resulting in low detection accuracy.

[0083] This invention can be applied to the fields of computer vision and image processing, and relates to target detection and ranging technology based on deep learning for computer vision research.

[0084] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0085] Please see Figure 1 , Figure 1 This is a flowchart illustrating the steps of a lightweight human body detection and ranging method suitable for stage lighting fixtures, as provided in Embodiment 1 of the present invention.

[0086] This invention provides a lightweight human body detection and ranging method suitable for stage lighting fixtures, comprising:

[0087] Step 101: Acquire the left and right visible light images and scene infrared images captured by the binocular camera.

[0088] It should be noted that the acquisition includes the left and right visible light images (including the left and right visible light images) collected by the binocular camera after pre-calibration by a camera calibration tool (such as the Zhang Zhengyou calibration method), the corresponding scene infrared images collected by the infrared sensor, and the camera intrinsic parameters (focal length f, principal point coordinates, etc.) and extrinsic parameters (baseline b, i.e., the distance between the optical centers of the left and right cameras) obtained through the camera calibration process.

[0089] Step 102: Based on the camera calibration parameters of the binocular camera, perform distortion correction on the left and right visible light images and the scene infrared image, and output the corrected left and right visible light images and the corrected scene infrared image.

[0090] It should be noted that a distortion correction algorithm based on camera intrinsic parameters (such as the Brownian distortion model) is used to perform distortion elimination processing on the left and right visible light images and the scene infrared image respectively, correcting the pixel shift caused by lens optical distortion, and outputting the distortion-corrected left / right visible light images and the distortion-corrected infrared images, i.e., the corrected left and right visible light images and the corrected scene infrared images.

[0091] Step 103: Using the improved YOLO11n model, generate the target region of interest in the left image based on the corrected left visible light image and the corrected scene infrared image in the corrected left and right visible light images.

[0092] It should be noted that you should refer to [link / reference]. Figures 2-3 Deep learning-based object detection methods typically use deep neural networks to automatically learn features and perform object detection. Currently popular single-stage detection methods directly extract object location and category information from images, such as the YOLO (YouOnlyLookOnce) algorithm developed by the Ultralytics team, which can achieve faster detection speeds. One of the key features of the YOLO algorithm is its simplicity and speed. Figure 3 The diagram shows the underlying principle of the YOLO algorithm. The YOLO model models object detection as a regression problem. It divides the image into an S×S grid and predicts B bounding boxes, their confidence scores, and class C probabilities for each grid cell. The network is followed by two fully connected layers. Alternating 1×1 convolutional layers reduce the feature space of the preceding layers. As a one-stage detector, it treats object detection as a regression problem, directly predicting bounding boxes and class probabilities on the image. This end-to-end approach reduces computation and enables real-time object detection.

[0093] Further, please refer to Figure 4 The YOLO algorithm has been continuously updated and iterated. The new version, YOLO 11, was released by the Ultralytics team in 2024. This version introduced new features and improvements to further enhance performance and flexibility. Figure 4 The diagram shows the YOLO11 algorithm flowchart. The YOLO11 model structure is generally divided into three parts: Backbone, Neck, and Head. The Backbone performs initial feature extraction, the Neck performs multi-scale feature fusion to enhance the model's ability to extract information from images of different sizes, and the Head connects three detectors to detect large, medium, and small targets respectively, returning the final bounding boxes, categories, and confidence scores. Specifically, Upsample represents upsampling, Contact represents connection, CBS (Convolution-Batch Normalization-SiLU Module) is the Convolution-Batch Normalization-SiLU (Sigmoid Linear Unit) module (i.e., the convolution module), DSC is the Depthwise Separable Convolution module, and Conv2d is the 2D Convolution module.

[0094] Among the various YOLO11 models, there are multiple options for the number of module repetitions and the number of channels in each convolutional layer. This invention specifically selects the lightest YOLO11n model (You Only Look Once 11 Nano). The advantages of YOLO11n are as follows: 1) High accuracy: YOLO11n achieves high accuracy in object detection tasks. It employs an improved C3K2 module, containing multiple residual connections and selectable attention mechanisms, greatly improving model accuracy. 2) High efficiency: It is the lightest version of YOLO11, maintaining high accuracy while offering very high running speed. Real-time object detection can be achieved on embedded and mobile devices. 3) Scalability: YOLO11n is highly flexible in design and can be customized and expanded according to different needs. It provides rich model configuration options and can be optimized according to specific tasks and hardware conditions.

[0095] Furthermore, such as Figure 5 As shown, stage lighting can present both dim and bright conditions, posing a significant challenge to human detection and increasing the rates of false positives and false negatives. Figure 6 The image shown on the right is an infrared thermal image. This image is obtained by "measuring" the heat radiated outward by an object. It has low sensitivity to changes in lighting and is resistant to light interference, showing stable performance in detecting targets with significant heat sources (such as the human body). Therefore, fusing infrared and visible light images can complement each other's advantages and improve the accuracy of human body detection under stage lighting.

[0096] In the field of object detection, data fusion strategies are crucial, especially in integrating multimodal data and improving detection performance. Based on the timing and method of data integration, fusion strategies can be categorized into early-stage fusion, mid-stage fusion, and late-stage fusion. Each strategy has its unique applications and advantages in object detection tasks. Multimodal fusion algorithms include... Figure 7 As shown. Based on the different fusion locations in the detection network, multimodal target detection methods can be divided into three categories: early-stage fusion, mid-stage fusion (feature-level fusion), and late-stage fusion. For example... Figure 7 (a) Pre-fusion, which performs image fusion before feature extraction. The pre-fusion method first performs preliminary channel-level fusion of the infrared and visible light images, and then inputs the fused image into the detection model. This method allows the model to process all input data simultaneously in the initial feature extraction stage, thereby capturing the complex relationships between different data sources. Figure 7 (b) Intermediate fusion, which extracts features from infrared and visible light images separately and then fuses these features in a detection network. This method not only allows for independent feature extraction from different data sources but also captures higher-level interactions at the feature level. Figure 7(c) is post-processing fusion. In object detection, post-processing fusion means that after each data source has been processed independently by the model, the output results are integrated at the final decision level. Each of the three fusion methods has its own advantages and disadvantages. This invention conducts experimental comparisons of the three fusion methods and improves the algorithm based on the better fusion method.

[0097] Experiments were conducted on the LLVIP dataset (A Visible-infrared Paired Dataset for Low-light Vision) using the YOLO11n framework to compare the three fusion methods. The training device consisted of two NVIDIA GeForce RTX 3090 graphics cards, and the optimizer used was SGD. The training lasted for 130 epochs.

[0098] Table 1. Experimental Comparison of Three Fusion Methods

[0099]

[0100] As can be seen from Table 1, among the three fusion methods, the mid-term fusion structure is superior to the early-term fusion and late-term fusion in terms of recall (R), precision (P), and average precision (mAP@0.5). Therefore, this invention improves the model based on mid-term fusion and proposes an adaptive mid-term fusion method.

[0101] The original intermediate fusion process simply concatenates the visible light and infrared images at the channel level after passing them through convolution (Conv) and the C3K2 module. It then performs another convolution (Conv) and the C3K2 module (Cross Stage Partial 3 with 2×2 Kernel, the bottleneck convolution module) before final concatenation. This simple concatenation and separation may not fully utilize the complementary information of the two modalities. To compensate for this deficiency, an attention mechanism is introduced during the fusion process, enabling the network to adaptively select important features from the infrared image and weight them on the visible light image; this is the MAC (multimodal attention concat) module.

[0102] Based on the above, such as Figure 8As shown, this invention improves the backbone network in the existing YOLO11n model to obtain an improved YOLO11n model. The improved YOLO11n model includes a backbone network, a neck network, and a detection head based on adaptive multimodal fusion. The backbone network based on adaptive multimodal fusion includes a convolutional module (CBS), a convolutional block with parallel spatial attention (C2PSA, Cross Stage 2 with Parallel Spatial Attention), a multimodal attention concatenation module (MAC), a spatial pyramid pooling-fast module (SPPF), and a bottleneck convolutional module (C3K2, Cross Stage Partial 3 with 2×2 Kernel).

[0103] Specifically, step 103 may include the following sub-steps S31-S39:

[0104] Step S31: Adaptive multimodal fusion is used to perform adaptive multimodal fusion on the corrected left visible light image and the corrected scene infrared image, and the target multimodal fusion feature map is output.

[0105] The target multimodal fusion feature map includes a first fusion feature map, a second fusion feature map, and a third fusion feature map.

[0106] Furthermore, step S31 may include the following sub-steps S311-S3111:

[0107] Step S311: Perform convolution operations on the corrected left visible light image and the corrected scene infrared image through two concatenated convolution modules respectively, and output the first left image convolution feature map and the first scene convolution feature map.

[0108] Step S312: Use the bottleneck convolution module to perform cross-stage convolution on the first left image convolution feature map and the first scene convolution feature map respectively, and output the first left image cross-stage convolution feature map and the first scene cross-stage convolution feature map.

[0109] Step S313: Input the first left image cross-stage convolutional feature map and the first scene cross-stage convolutional feature map into the multimodal attention splicing module for mid-term fusion, output the first left image mid-term fusion feature map, and perform convolution operations on the first left image mid-term fusion feature map and the first scene cross-stage convolutional feature map respectively through the convolution module to output the second left image convolutional feature map and the second scene convolutional feature map;

[0110] The multimodal attention splicing module includes convolutional layers, ReLU activation function layers, and Sigmoid activation function layers.

[0111] Further, step S313 may include the following sub-steps S3131-S3136:

[0112] Step S3131: Concatenate the first left-side cross-stage convolutional feature map and the first scene cross-stage convolutional feature map to output the concatenated feature map;

[0113] Step S3132: Perform convolution operation on the stitched feature map using a convolutional layer to generate the first convolutional feature map;

[0114] Step S3133: Perform a linear transformation on the first convolutional feature map through a ReLU activation function layer to output the transformed feature map;

[0115] Step S3134: Perform convolution operation on the transformed feature map using a convolutional layer to output the second convolutional feature map;

[0116] Step S3135: Use the second convolutional feature map as the input to the Sigmoid activation function layer and output the convolutional feature probability map;

[0117] Step S3136: Multiply the convolutional feature probability map with the cross-stage convolutional feature map of the first left image to generate the mid-term fusion feature map of the first left image.

[0118] It should be noted that the cross-stage convolutional feature map of the first left image is used as Input1, and the cross-stage convolutional feature map of the first scene is used as Input2. They are both input into the multimodal attention concatenation module for processing to obtain the mid-term fusion feature map of the first left image.

[0119] It is worth mentioning that the processing of the first fused feature map, the second fused feature map, and the mid-section fused feature map of the second left image is the same as the processing of the mid-section fused feature map of the first left image, and will not be described in detail here.

[0120] Step S314: Use the bottleneck convolution module to perform cross-stage convolution on the second left image convolution feature map and the second scene convolution feature map respectively, and output the second left image cross-stage convolution feature map and the second scene cross-stage convolution feature map.

[0121] Step S315: Input the second left image cross-stage convolutional feature map and the second scene cross-stage convolutional feature map into the multimodal attention splicing module for mid-term fusion, output the first fused feature map, and perform convolution operations on the first fused feature map and the second scene cross-stage convolutional feature map through the convolution module respectively, output the third left image convolutional feature map and the third scene convolutional feature map;

[0122] Step S316: Use the bottleneck convolution module to perform cross-stage convolution on the third left image convolution feature map and the third scene convolution feature map respectively, and output the third left image cross-stage convolution feature map and the third scene cross-stage convolution feature map.

[0123] Step S317: Input the third left image cross-stage convolutional feature map and the third scene cross-stage convolutional feature map into the multimodal attention splicing module for mid-term fusion, output the second fusion feature map, and perform convolution operations on the second fusion feature map and the third scene cross-stage convolutional feature map through the convolution module respectively, output the fourth left image convolutional feature map and the fourth scene convolutional feature map;

[0124] Step S318: Use the bottleneck convolution module to perform cross-stage convolution on the fourth left image convolution feature map and the fourth scene convolution feature map respectively, and output the fourth left image cross-stage convolution feature map and the fourth scene cross-stage convolution feature map.

[0125] Step S319: Input the cross-stage convolutional feature map of the fourth left image and the cross-stage convolutional feature map of the fourth scene into the multimodal attention splicing module for mid-term fusion, and output the mid-term fusion feature map of the second left image;

[0126] Step S3110: Use the spatial pyramid pooling module to pool the mid-term fusion feature map of the second left image and output the pooled feature map;

[0127] Step S3111: Use the pooled feature map as input to the convolutional block with parallel spatial attention, and output the third fused feature map.

[0128] It should be noted that Input1 is the corrected left visible light image, and Input2 is the corrected scene infrared image. The two images undergo adaptive mid-stage fusion during the feature extraction phase. The fused image then undergoes multi-scale feature fusion in the Neck section before being input to the detection head for detection and result return. The adaptive mid-stage fusion, such as the MAC+Backbone part, first processes the two input images through two convolutions to increase the number of channels and downsample them, then uses the C3K2 module to further extract important features and enhance the receptive field. The features of the two images are then input into the MAC module. Specifically, the two inputs are first concatenated, and the features of the concatenated visible light and infrared images are then processed through convolution and linear transformation, finally using the Sigmoid function to obtain probability information, which is weighted onto Input1, i.e., the visible light image. Subsequent convolutions and downsampling by the C3K2 module are repeated before fusion. After four adaptive fusions, the SPPF and C2PSA modules are used to enhance the representation of important features, thus completing the initial feature extraction of MAC+Backbone.

[0129] It is worth mentioning that this attention-weighted fusion can effectively combine the information from the two modalities and then focus it on the visible light image, thereby reducing detection interference caused by changes in illumination in the visible light image.

[0130] Step S32: Perform multi-scale feature fusion on the target multimodal fusion feature map through the neck network to generate a multi-scale fusion feature map;

[0131] Step S33: Use the multi-scale fused feature map as input to the detection head and output multiple bounding box regression distributions;

[0132] Step S34: Generate multiple candidate detection boxes in the left image based on the preset anchor boxes and the regression distribution of multiple bounding boxes;

[0133] Step S35: Use the non-maximum suppression algorithm to filter multiple candidate detection boxes in the left image, output the accurate detection box in the left image, and expand the accurate detection box in the left image by pixels to output the initial region of interest in the left image.

[0134] Step S36: Count the number of initial regions of interest in the left image and compare the number of initial regions of interest in the left image with the preset threshold number of regions of interest;

[0135] Step S37: If the number of initial regions of interest in the left image is greater than the preset threshold for the number of regions of interest, calculate the spacing between adjacent initial regions of interest in the left image and compare them with the preset pixel threshold respectively.

[0136] Step S38: Merge adjacent initial regions of interest in the left image with a spacing smaller than a preset pixel threshold, and output the target region of interest in the left image;

[0137] Step S39: If the number of initial regions of interest in the left image is less than or equal to the preset threshold for the number of regions of interest, then the initial regions of interest in the left image are taken as the target regions of interest in the left image.

[0138] It should be noted that the obtained first, second, and third fusion feature maps are input into the neck network, which outputs a multi-scale fusion feature map (including the first-scale fusion feature map, the second-scale fusion feature map, and the third-scale fusion feature map). Then, the first-scale fusion feature map, the second-scale fusion feature map, and the third-scale fusion feature map are input into the detection head, which outputs multiple bounding box regression distributions (including bounding box regression distributions at three different scales) and category information. The processing process of the neck network for the target multi-modal fusion feature map and the processing process of the detection head for the target multi-scale fusion feature map can refer to the processing process of the neck network and the detection head in the existing YOLO11n model, and will not be elaborated further in this invention.

[0139] The detection head predicts the bounding box regression distribution at different scales based on fused feature maps. Then, it pre-sets anchor boxes of corresponding sizes and proportions for each scale of feature map grid (e.g., large anchor boxes for small scales). Next, it uses the bounding box regression distribution at each scale to predict the anchor box offset (center coordinate offset, width and height scaling factor). Then, it applies the offset to the pre-set anchor boxes, adjusting their position and size to generate preliminary candidate boxes. Finally, it maps the candidate boxes in the feature map coordinate system back to the original size of the left image, obtaining multiple candidate detection boxes in the left image that cover targets at different scales, providing a foundation for subsequent non-maximum suppression (NMS) to filter redundant boxes and for target classification.

[0140] Further, please refer to Figure 9 The images in the pair are the left and right images, respectively (images from the open-source real-world street view KITTI dataset, Karlsruhe Institute of Technology and Toyota Technological Institute at Chicago Dataset). Figure 9 In (a), it is evident that the human body in the right image has a leftward parallax relative to the left image, but this parallax disappears after ROI extraction. To address the issue of parallax disappearing after target region extraction from both images, a redesigned ROI extraction scheme is implemented. Target detection is performed only on the left image. After processing with a non-maximum suppression algorithm, a precise detection box is obtained in the left image. This box is then expanded to the left by d_max pixels, where d_max = (baseline * roi_width / 1.7), where baseline is the camera baseline (in meters), roi_width is the width of the target region (in pixels), and 1.7 (meters) is the average height of a human. The final target region, as shown in red in the figure, completely encompasses the human body positions in both images. Furthermore, target detection is performed only once, reducing computational load and avoiding inconsistencies between the left and right targets. If there are multiple ROI regions in the image, their proximity is considered when merging them for calculation. Figure 9 As shown in (b), when two ROI regions are smaller than a set value (128 pixels in this case), they are merged together for calculation, which can effectively avoid the depth estimation of duplicate regions. Specifically, if there are multiple initial ROIs in the image (i.e., the number of initial ROIs in the left image is greater than the preset threshold for the number of ROIs, which is set to 1), the distance between adjacent ROIs is calculated. If the distance is less than 128 pixels (the preset pixel threshold), ROI merging is performed (to avoid duplicate depth calculations). The output is the complete optimized ROI region (single or merged ROI set) containing the human body in the left and right images, which is the target ROI region in the left image.

[0141] It is worth mentioning that if there is only one initial region of interest in the left image, then that initial region of interest in the left image is used as the target region of interest in the left image; if the distance between adjacent initial regions of interest in the left image is greater than or equal to 128 pixels, then the independence of each initial region of interest in the left image is preserved, and each initial region of interest in the left image is used as the target region of interest in the left image for subsequent processing.

[0142] Step 104: Generate the target region of interest in the right image based on the target region of interest in the left image and the corrected right visible light image in the corrected left and right visible light images.

[0143] It should be noted that, firstly, relying on the "epipolar alignment" characteristic of stereo camera calibration (after calibration, the pixels of corresponding targets in the left and right images are located on the same horizontal epipolar line), the region of the target ROI in the left image is directly mapped to the right image. Figure 1 By aligning the regions, we obtain the target region of interest in the right figure, which provides the basis for aligning the left and right target regions for subsequent binocular disparity calculation and target depth estimation.

[0144] Step 105: Use a stereo matching algorithm to perform human distance detection based on the target region of interest in the right image and the target region of interest in the left image, and output the distance between the human body and the stereo camera.

[0145] The stereo matching algorithm used is the LightStereo lightweight stereo matching algorithm (stereo matching binocular depth estimation algorithm). This algorithm was selected after extensive experimental comparisons using LightStereo, released in 2024. Figure 10 The diagram shows the algorithm block diagram of LightStereo. Unlike traditional methods that rely on computationally intensive 4D costs, LightStereo uses 3D costs as a lightweight alternative. In the cost aggregation stage, a lightweight MSCA attention module is used for progressive disparity correction, and in the disparity refinement stage, left-image feature fusion is introduced to further refine the upsampled disparity map. This overall lightweight design achieves an excellent balance between speed and accuracy, with an error of 0.64 on the KITTI dataset and a runtime of only 46ms in local testing on an NVIDIA GeForce RTX 4060 device.

[0146] It should be noted that the LightStereo lightweight stereo matching algorithm is used to process the Region of Interest (ROI) in both the right and left images: First, lightweight convolution is used to extract ROI features and construct a 3D cost volume (replacing the traditional 4D cost volume to reduce computation). Then, MSCA (Multi-Scale Channel Attention, a lightweight attention module) is used to perform cost aggregation and progressive disparity correction. Finally, the disparity map is refined by feature fusion of the left image, and the target disparity map of the ROI region is output (disparity value d, in pixels). The ROI disparity value is converted into absolute human depth. Clustering is performed on the depth values ​​within the same ROI (outliers are removed) to retain stable depth values. The final human depth value (i.e., the absolute distance from the human body to the camera) corresponding to each human ROI is output. The system employs a distance threshold judgment logic (with a preset distance threshold, such as 3m, which can be adjusted according to the actual scenario) to compare the human body depth value with a safety threshold: if the depth value is less than the safety threshold, it determines that the human body has entered a dangerous illumination area; if the depth value is greater than or equal to the safety threshold, it determines that the human body is in a safe area; and outputs a "turn off stage lights" command (when the human body enters a dangerous area) or a "keep stage lights on" command (when the human body is in a safe area) to achieve active safety control of the lights.

[0147] Specifically, step 105 may include the following sub-steps S51-S57:

[0148] Step S51: Perform multi-scale fusion feature extraction on the target region of interest in the right image and the target region of interest in the left image respectively to generate the region of interest features in the left image and the right image.

[0149] Step S52: Construct a 3D cost volume based on the features of the region of interest in the left and right images;

[0150] Step S53: Perform cost aggregation based on the features of the region of interest in the left image and the 3D cost volume to generate an aggregated cost volume;

[0151] Step S54: Determine the multiplication cost volume based on the aggregation cost volume;

[0152] Step S55: Perform disparity regression on the multiplicative cost volume to generate initial disparity values ​​for the region of interest;

[0153] Step S56: Based on the target region of interest in the left image, refine the disparity value of the initial region of interest to generate the disparity values ​​of corresponding pixels in the left and right images;

[0154] Step S57: Calculate the distance between the human body and the binocular camera based on the parallax values ​​of the corresponding pixels in the left and right images.

[0155] It should be noted that the core idea of ​​stereo matching algorithms is to find the disparity value of corresponding points in two images. Most stereo matching algorithms follow this process: Figure 11 The algorithm consists of four basic steps: pixel matching cost calculation, cost aggregation, disparity calculation, and disparity map post-processing. To improve the accuracy of stereo matching algorithms, deep neural networks are introduced on the basis of traditional algorithms to construct an end-to-end stereo matching network. Most excellent networks achieve a balance between accuracy and efficiency.

[0156] Specifically, taking the feature processing of the region of interest (ROI) in the left image as an example, multi-scale feature extraction is performed on the ROI in the left image to obtain a multi-scale feature map. This feature map includes a quarter feature map, an eighth feature map, and a sixteenth feature map. Then, the one-sixteenth feature map is upsampled to obtain an upsampled eighth feature map. This upsampled eighth feature map is then concatenated with the first eighth feature map along the channel dimension to obtain an added eighth feature map. This added eighth feature map is then upsampled to obtain an upsampled quarter feature map. This upsampled quarter feature map is then concatenated with the first quarter feature map along the channel dimension to obtain the ROI features in the left image. Similarly, the ROI features in the right image can be obtained.

[0157] Furthermore, based on the baseline and focal length of the binocular camera, the effective disparity range is determined. The features of the region of interest in the right image are shifted along the horizontal epipolar direction according to different disparity values ​​to form multiple sets of shifted right image features. Then, the matching cost (such as cosine similarity, cross-correlation, or sum of squared element-wise differences) is calculated for the features of the left image and each set of shifted right image features to quantify the similarity between the two. Finally, the matching costs at all positions (H×W) and at all disparity values ​​are stacked according to the dimension of "height×width×disparity" to form a 3D cost volume (Cost), which provides the basic data structure for subsequent cost aggregation and disparity regression.

[0158] Furthermore, cost aggregation is performed based on the features of the region of interest (ROI) in the left image and the 3D cost volume to generate an aggregated cost volume. The multi-scale feature map of the ROI in the left image is used as the input to the MSCA attention module to obtain attention coefficients. The aggregated cost volume and the attention coefficients output by the MSCA attention module are multiplied to obtain a multiplied cost volume. The ROI in the left image is used as the input to the Context Network to obtain the features output by the Context Network. Disparity regression is performed on the multiplied cost volume to obtain the initial ROI region disparity value. The disparity of the initial ROI region disparity value is refined using the features output by the Context Network to obtain the target ROI region disparity value (i.e., the disparity value of the corresponding pixel in the left and right images).

[0159] Furthermore, this invention employs computer vision to implement the ranging function, specifically using a binocular depth estimation algorithm. Monocular depth estimation algorithms obtain real images through a single camera. Developing a monocular depth estimation algorithm for ranging requires training on a large dataset and yields relative distances with low accuracy. Binocular depth estimation primarily utilizes the parallax between the left and right cameras to determine the depth of an object, obtaining an absolute distance. The principle diagram of binocular ranging is shown below. Figure 12 As shown, to obtain the distance z of the target point P from the camera, two cameras are needed: the left camera L and the right camera R. f and b are the focal length and baseline of the camera, respectively (values ​​that are set when the camera is manufactured). z is the distance z of point P from the camera, which is the depth value.

[0160] Establish a coordinate system in the schematic diagram, connect point P with the left and right cameras, and construct two triangular proportional relationships as shown in formulas (1) and (2). The depth value is calculated by combining the two formulas as shown in formula (3). The depth is equal to the focal length f multiplied by the baseline b and then divided by the parallax value d. That is, the depth is inversely proportional to the parallax.

[0161] (1)

[0162] (2)

[0163] (3)

[0164] in, The depth value represents the absolute distance from the human body to the binocular camera. The focal length of the binocular camera; The baseline for the binocular camera; The x-coordinate of the human body on the left camera's imaging plane; The x-coordinate of the human body on the imaging plane of the right camera; is the disparity value of the corresponding pixels in the left and right images; x is the x-coordinate of the spatial point P (human body) in the camera coordinate system.

[0165] Determining the pixel-level disparity between the left and right images mainly uses stereo matching algorithms. Stereo matching is a core technology in computer vision for recovering three-dimensional structures from two-dimensional images. Essentially, it involves finding pixel-level correspondences (i.e., "matching points") between images from two perspectives (such as the left and right views of a binocular camera) and then calculating the disparity (the positional deviation of pixels in the left and right views).

[0166] For comparison of technical effects, existing technologies can be used as a reference. Laser lights, due to their high power density, can be harmful if directly irradiated onto the human body. Therefore, it is necessary to avoid projecting light onto people during operation. Current products mostly use manual exclusion zones to limit the illumination range of the lights, but this method cannot adapt to various unexpected situations. Therefore, there is an urgent need to develop a lightweight, deployable, and real-time active human detection algorithm—when a person enters the vicinity of the light's illumination area within a certain distance, the algorithm can send a command to turn off the lights in advance to avoid harm. Regarding the selection of human detection technology, while infrared sensors can detect human heat, their accuracy is low and they are easily affected by environmental interference; LiDAR combined with point cloud data has high detection accuracy but suffers from high cost and energy consumption; computer vision combined with deep learning algorithms offers high accuracy and wide application, making it a more reasonable choice. Among them, the YOLO algorithm developed by the Ultralytics team is particularly suitable for real-time detection needs—as a first-stage detector, it treats the target detection task as a regression problem, directly predicting bounding boxes and class probabilities on the image. The end-to-end design reduces computational load and effectively ensures detection speed. In addition, in terms of human body ranging technology, lidar ranging accuracy reaches the millimeter level but has high cost and energy consumption, infrared ranging has low cost but is greatly affected by the environment and has low accuracy, ultrasonic ranging is suitable for short distances and has centimeter-level accuracy, while vision-based ranging (including monocular and binocular ranging) has both lower cost and relatively high accuracy, and can be used as a supporting ranging solution to further improve the active human avoidance system.

[0167] Further investigation revealed that binocular depth estimation achieves high accuracy, meeting the requirements. Binocular depth estimation primarily utilizes the disparity between the left and right cameras to determine the depth of an object, yielding an absolute distance. Determining the pixel-level disparity between the left and right images mainly employs stereo matching algorithms. The core objective of stereo matching is to generate an accurate disparity map, which can be converted into a depth map using the focal length and baseline parameters of different cameras, thus achieving distance measurement. Traditional stereo matching algorithms primarily rely on manually designed feature extraction algorithms or optimization functions for disparity prediction, consisting of four basic steps: pixel matching cost calculation, cost aggregation, disparity calculation, and disparity map post-processing. Due to the powerful feature extraction and learning capabilities of convolutional neural networks in computer vision, recent stereo matching algorithms have incorporated deep learning to construct end-to-end stereo matching models, significantly improving the accuracy of stereo matching.

[0168] Figure 13The diagram shows the existing technology roadmap for combining object detection and ranging. The main process involves first performing stereo matching on the left and right images captured by a binocular camera to obtain pixel disparity values. Then, depth values ​​are calculated using the camera's focal length and baseline to obtain the depth value of the entire left image. Humans are then detected on the left image, and the corresponding depth values ​​are extracted from the bounding boxes returned by the human detection. Clustering is then used to filter stable depth values, and finally, the bounding boxes and depth values ​​are returned. In this approach, the YOLO algorithm for object detection only detects the left image and is a single-stage algorithm. On a test device with an NVIDIA GeForce RTX 4060 graphics card, it achieves real-time detection in 6ms. However, the stereo matching algorithm matches both left and right images, and most mainstream, high-accuracy algorithms take over 50ms in testing. Combining object detection with stereo matching would result in even longer response times, failing to achieve real-time detection and making deployment on mobile devices difficult. Furthermore, the separation of detection and depth estimation presents synchronization issues. Therefore, to address the shortcomings of the existing technology, innovative solutions that simultaneously optimize both efficiency and accuracy are needed.

[0169] Based on the above, the best existing solution for balancing speed and accuracy is to first perform full-image depth estimation, then perform human detection and extract the depth values ​​of the bounding boxes. The YOLO algorithm for object detection only detects the left image and is a single-stage algorithm. On a test device with an NVIDIA GeForce RTX 4060 graphics card, it achieves real-time detection in 6ms. However, stereo matching algorithms match both left and right images, and most mainstream, high-accuracy algorithms take over 50ms in testing. Combining object detection with stereo matching would result in even longer response times, failing to achieve real-time detection and making deployment on mobile devices even more difficult. Furthermore, separating detection and depth estimation presents synchronization issues.

[0170] To address the aforementioned shortcomings, this invention provides a lightweight human detection and ranging method suitable for stage lighting. It employs a model lightweighting strategy, such as using the YOLO11n detection network and the LightStereo depth estimation algorithm. Through algorithm integration and optimization, the human detection and depth estimation algorithms are highly integrated to reduce redundant computation and improve overall efficiency. A detection-driven depth estimation method is proposed, which first uses a lightweight detection model (YOLO11n) to quickly detect human targets in visible light images. After obtaining the detection box positions, depth estimation is performed only on the regions within the detection boxes, thus avoiding full-image computation, significantly reducing the image area to be processed, and improving processing speed. Furthermore, a joint detection and depth estimation network is designed. Through a shared feature extraction layer, it simultaneously outputs detection boxes and depth information, further reducing model parameters and computational load, and improving the synergistic effect of detection and depth estimation. The improved algorithm achieves a response time of <50ms, contributing to its operation on embedded devices.

[0171] To improve accuracy, a new technology of fusing multimodal data was introduced into the human body detection algorithm. Combining visible light and infrared images, this invention designs a multimodal adaptive fusion network to enhance the robustness of detection by using data from the two sensors, especially reducing false detections and missed detections under stage lighting (low light or strong light environment).

[0172] To address the time-consuming nature of existing technologies, this invention introduces an efficient matching strategy, employing a detection-driven depth estimation method. This method performs stereo matching only within the human body region after human detection, reducing computational load, minimizing background interference, and improving matching accuracy.

[0173] To address the issue that inconsistent human detection in the left and right images can affect subsequent stereo matching depth estimation, a dynamic ROI (Region of Interest) association scheme is designed. This scheme performs human detection only on the left image and then expands the pixels to the left to achieve a complete region that includes human bodies in both the left and right images.

[0174] In terms of algorithm optimization, multimodal data fusion was introduced. By combining visible light and infrared images, a multimodal adaptive fusion network was designed to enhance the robustness of detection and reduce false detections and false negatives by utilizing data from both sensors.

[0175] To address the time-consuming nature of existing technologies, this invention designs an efficient matching strategy and employs a detection-driven depth estimation method. Specific measures include designing a dynamic ROI association scheme, selecting and optimizing a lightweight model, and introducing a new technology that integrates multimodal data.

[0176] Specifically, this invention uses a lightweight convolutional neural network to extract image features, performs human detection based on YOLO11n, and integrates multimodal data enhancement features. It employs the LightStereo depth estimation algorithm to estimate the depth only for detected human regions. Based on the detection results and depth information, it quickly determines whether to turn off the lights. For example, Figure 14 As shown, this invention proposes a target detection-driven stereo matching technology approach, which highly integrates human detection algorithms and ranging algorithms. Considering the issue that performing target detection on both left and right images simultaneously could lead to inconsistent ROIs between the two images, this invention further improves the process by performing target detection only on the left image. The maximum disparity is estimated using the target bounding box size to find the corresponding target in the right image (i.e., the ROI association step), avoiding redundant calculations. Stereo matching is then performed only on the detected target regions, avoiding full-image calculations and background interference, thus significantly reducing the computational load of stereo matching.

[0177] Compared with existing techniques that first perform full-image depth estimation, then perform human detection and extract the depth value of the target box, the efficient matching strategy designed in this invention greatly reduces the amount of computation, contributing to deployment on mobile devices, and optimizes the accuracy of the algorithm, making a benchmark contribution to the fact that the accuracy will decrease when migrated to real-world application scenarios.

[0178] like Figures 15-17 The figures shown are comparison images from tests conducted on the KITTI dataset. Figure (a) shows the depth values ​​of the target bounding boxes extracted from the entire image using existing techniques for stereo matching. From top to bottom, the images are the original left image, disparity map, and depth map. Figure (b) shows the dynamic ROI extraction scheme designed in this invention, which performs stereo matching only on the target region. From top to bottom, the images are the original left image, ROI map, and disparity map. Figure (c) shows the added ROI merging step. From top to bottom, the images are the original left image, merged ROI map, and disparity map. It can be seen that the computational load is significantly reduced without affecting accuracy.

[0179] Table 2 Algorithm Test Time Results

[0180]

[0181] Table 3 Comparison of Fusion Methods

[0182]

[0183] Table 2 shows a comparison of the test times for the algorithm before and after the improvement. The test images have a resolution of 1242 pixels wide and 375 pixels high. It can be seen that the response time of the improved technique remains the same as that of the existing human detection algorithm, but the distance measurement time is significantly reduced. The test dataset contains 1-5 people, and the average time is taken here. The improved technique takes 20% of the running time of the existing technique. Although this time will increase with the number of people in the field of view, it will still be less than or equal to the running time of the existing technique overall.

[0184] The algorithm was trained for 130 epochs on the LLVIP dataset using the SGD (Stochastic Gradient Descent) optimizer. Key performance metrics are compared in Table 3. Our proposed adaptive fusion strategy achieves the highest overall performance evaluation metric, mAP@0.5, at 0.981. Precision (P) and recall (R) are also optimal. After reasonably controlling the number of channels and repetitions in the convolutional layers, the total computational cost (GFLOPS) is 6.2, a slight reduction compared to the non-fusion approach. This enables real-time human detection and is easily deployed on mobile devices.

[0185] In summary, this invention optimizes both efficiency and accuracy, contributing to real-time human detection and distance measurement under stage lighting.

[0186] In this embodiment of the invention, a lightweight human body detection and ranging method suitable for stage lighting is provided. The method acquires left and right visible light images and scene infrared images from a binocular camera; performs distortion correction on the left and right visible light images and scene infrared images based on the camera calibration parameters of the binocular camera, and outputs the corrected left and right visible light images and scene infrared images; uses an improved YOLOv11n model to generate a target region of interest (ROI) in the left image based on the corrected left visible light image and the corrected scene infrared image in the corrected left and right visible light images; generates a target ROI in the right image based on the target ROI in the left image and the corrected right visible light image in the corrected left and right visible light images; and uses a stereo matching algorithm to generate a target ROI in the right image based on the target ROI in the left and right visible light images. The system performs human distance detection within the target's region of interest (ROI) and outputs the distance between the human body and the binocular camera. Based on this approach, the invention utilizes an improved YOLO11n model to adaptively integrate effective features from visible light and infrared images. Even in scenarios prone to missed detection by traditional methods, such as direct stage lighting or fluctuating ambient temperatures, the system can still stably capture human features through the anti-interference properties of infrared images and the detailed supplementation from visible light images. Simultaneously, the distortion correction process ensures the accuracy of image pixel positions, avoiding human contour shifts caused by lens distortion and providing a reliable image foundation for subsequent feature extraction. The dynamically associated left and right ROIs further narrow the computational range of stereo matching, reducing interference from complex stage backgrounds and thus improving detection accuracy.

[0187] Please see Figure 18 , Figure 18 This is a structural block diagram of a lightweight human body detection and ranging system suitable for stage lighting provided in Embodiment 2 of the present invention.

[0188] This invention provides a lightweight human body detection and ranging system suitable for stage lighting fixtures, comprising:

[0189] The acquisition module 1801 is used to acquire the left and right visible light images and scene infrared images captured by the binocular camera;

[0190] The correction module 1802 is used to perform distortion correction on the left and right visible light images and scene infrared images based on the camera calibration parameters of the binocular camera, and output the corrected left and right visible light images and scene infrared images.

[0191] The left image region of interest generation module 1803 is used to generate the target region of interest in the left image based on the corrected left visible light image and the corrected scene infrared image in the corrected left and right visible light images using an improved YOLO11n model.

[0192] The region of interest generation module 1804 in the right image is used to generate the region of interest in the right image based on the target region of interest in the left image and the corrected right visible light image in the corrected left and right visible light images;

[0193] The human detection module 1805 is used to perform human distance detection based on the target region of interest in the right image and the target region of interest in the left image using a stereo matching algorithm, and outputs the distance between the human body and the stereo camera.

[0194] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system and modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0195] This invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program; when the computer program is executed by the processor, the processor performs the steps of the lightweight human body detection and ranging method applicable to stage lighting as described in any of the above embodiments.

[0196] This invention also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implement the steps of the lightweight human body detection and ranging method applicable to stage lighting as described in any of the above embodiments.

[0197] This invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of a lightweight human body detection and ranging method applicable to stage lighting fixtures as described in any of the above embodiments.

[0198] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A lightweight human body detection and ranging method suitable for stage lighting fixtures, characterized in that, include: Acquire left and right visible light images and scene infrared images captured by the binocular camera; Based on the camera calibration parameters of the binocular camera, distortion correction is performed on the left and right visible light images and the scene infrared image, and the corrected left and right visible light images and the corrected scene infrared image are output. An improved YOLO11n model is used to generate the target region of interest in the left image based on the corrected left visible light image and the corrected scene infrared image in the corrected left and right visible light images. The target region of interest in the right image is generated based on the target region of interest in the left image and the corrected right visible light image in the corrected left and right visible light images; A stereo matching algorithm is used to detect human distance based on the target region of interest in the right image and the target region of interest in the left image, and the distance between the human body and the stereo camera is output. The improved YOLO11n model includes a backbone network, a neck network, and a detection head based on adaptive multimodal fusion. The backbone network includes a convolutional module, a bottleneck convolutional module, a convolutional block with parallel spatial attention, a multimodal attention concatenation module, and a spatial pyramid pooling module. The target multimodal fusion feature map includes a first fusion feature map, a second fusion feature map, and a third fusion feature map. The backbone network based on adaptive multimodal fusion is used to adaptively fuse the corrected left visible light image and the corrected scene infrared image to output the target multimodal fusion feature map, including: The corrected left visible light image and the corrected scene infrared image are convolved by two concatenated convolution modules to output the first left image convolution feature map and the first scene convolution feature map. The bottleneck convolution module is used to perform cross-stage convolution on the first left image convolution feature map and the first scene convolution feature map respectively, and outputs the first left image cross-stage convolution feature map and the first scene cross-stage convolution feature map; The first left image cross-stage convolutional feature map and the first scene cross-stage convolutional feature map are input into the multimodal attention splicing module for mid-term fusion, and the first left image mid-term fusion feature map is output. The first left image mid-term fusion feature map and the first scene cross-stage convolutional feature map are then convolved by the convolution module to output the second left image convolutional feature map and the second scene convolutional feature map. Based on the second left convolutional feature map and the second scene convolutional feature map, a target multimodal fusion feature map is obtained.

2. The lightweight human body detection and ranging method for stage lighting fixtures according to claim 1, characterized in that, The improved YOLO11n model generates the target region of interest in the left image based on the corrected left visible light image and the corrected scene infrared image in the corrected left and right visible light images, including: The corrected left visible light image and the corrected scene infrared image are adaptively fused using the backbone network based on adaptive multimodal fusion to output a target multimodal fusion feature map. The target multimodal fusion feature map is fused using the neck network to generate a multi-scale fusion feature map. The multi-scale fused feature map is used as the input to the detection head, and multiple bounding box regression distributions are output. Based on the preset anchor boxes and the regression distribution of the multiple bounding boxes, multiple candidate detection boxes for the left image are generated; A non-maximum suppression algorithm is used to filter multiple candidate detection boxes in the left image, outputting accurate detection boxes in the left image, and pixel expansion is performed on the accurate detection boxes in the left image to output the initial region of interest in the left image. The number of initial regions of interest in the left image is counted, and the number of initial regions of interest in the left image is compared with a preset threshold for the number of regions of interest. If the number of initial regions of interest in the left image is greater than the preset threshold for the number of regions of interest, then the spacing between adjacent initial regions of interest in the left image is calculated and compared with the preset pixel threshold respectively. The adjacent initial regions of interest in the left image with a spacing smaller than the preset pixel threshold are merged to output the target region of interest in the left image; If the number of initial regions of interest in the left image is less than or equal to the preset threshold for the number of regions of interest, then the initial regions of interest in the left image are taken as the target regions of interest in the left image.

3. The lightweight human body detection and ranging method for stage lighting fixtures according to claim 2, characterized in that, The step of using the backbone network based on adaptive multimodal fusion to adaptively fuse the corrected left visible light image and the corrected scene infrared image to output a target multimodal fusion feature map further includes: The bottleneck convolution module is used to perform cross-stage convolution on the second left image convolution feature map and the second scene convolution feature map respectively, and outputs the second left image cross-stage convolution feature map and the second scene cross-stage convolution feature map; The second left image cross-stage convolutional feature map and the second scene cross-stage convolutional feature map are input into the multimodal attention splicing module for mid-term fusion, and the first fused feature map is output. The first fused feature map and the second scene cross-stage convolutional feature map are then convolved by the convolution module to output the third left image convolutional feature map and the third scene convolutional feature map. The bottleneck convolution module is used to perform cross-stage convolution on the third left image convolution feature map and the third scene convolution feature map respectively, and outputs the third left image cross-stage convolution feature map and the third scene cross-stage convolution feature map; The third left image cross-stage convolutional feature map and the third scene cross-stage convolutional feature map are input into the multimodal attention splicing module for mid-term fusion, and the second fused feature map is output. The second fused feature map and the third scene cross-stage convolutional feature map are then convolved by the convolution module to output the fourth left image convolutional feature map and the fourth scene convolutional feature map. The bottleneck convolution module is used to perform cross-stage convolution on the fourth left image convolution feature map and the fourth scene convolution feature map respectively, and outputs the fourth left image cross-stage convolution feature map and the fourth scene cross-stage convolution feature map; The cross-stage convolutional feature map of the fourth left image and the cross-stage convolutional feature map of the fourth scene are input into the multimodal attention splicing module for mid-term fusion, and the mid-term fusion feature map of the second left image is output. The spatial pyramid pooling module is used to pool the mid-term fusion feature map of the second left image, and the pooled feature map is output. The pooled feature map is used as input to a convolutional block with parallel spatial attention to output a third fused feature map.

4. The lightweight human body detection and ranging method for stage lighting fixtures according to claim 3, characterized in that, The multimodal attention stitching module includes convolutional layers, ReLU activation function layers, and Sigmoid activation function layers; the step of inputting the first left image cross-stage convolutional feature map and the first scene cross-stage convolutional feature map into the multimodal attention stitching module for mid-term fusion, and outputting the first left image mid-term fused feature map, includes: The first left image cross-stage convolutional feature map and the first scene cross-stage convolutional feature map are concatenated to output the concatenated feature map; A convolutional layer is used to perform a convolution operation on the stitched feature map to generate a first convolutional feature map; The first convolutional feature map is linearly transformed by a ReLU activation function layer, and the transformed feature map is output. A convolutional layer is used to perform a convolution operation on the transformed feature map to output a second convolutional feature map; The second convolutional feature map is used as the input to the Sigmoid activation function layer, and the output is a convolutional feature probability map. The convolutional feature probability map is multiplied by the cross-stage convolutional feature map of the first left image to generate the mid-term fusion feature map of the first left image.

5. The lightweight human body detection and ranging method for stage lighting fixtures according to claim 1, characterized in that, The method employs a stereo matching algorithm to perform human distance detection based on the target region of interest in the right image and the target region of interest in the left image, and outputs the distance between the human body and the stereo camera, including: Multi-scale fusion feature extraction is performed on the target region of interest in the right image and the target region of interest in the left image respectively to generate the region of interest features in the left image and the right image. Based on the region of interest features in the left and right images, a 3D cost volume is constructed. Based on the features of the region of interest in the left image and the 3D cost volume, cost aggregation is performed to generate an aggregated cost volume; Based on the aggregate cost body, determine the multiplication cost body; Perform disparity regression on the multiplicative cost volume to generate initial disparity values ​​for the region of interest; Based on the target region of interest in the left image, the disparity value of the initial region of interest is refined to generate the disparity values ​​of corresponding pixels in the left and right images; The distance between the human body and the binocular camera is calculated based on the parallax values ​​of the corresponding pixels in the left and right images.

6. The lightweight human body detection and ranging method for stage lighting fixtures according to claim 5, characterized in that, The calculation process for the distance between the human body and the binocular camera is as follows: ； in, The depth value represents the distance between the human body and the binocular camera. The focal length of the binocular camera; The baseline for the binocular camera; The x-coordinate of the human body on the left camera's imaging plane; The x-coordinate of the human body on the imaging plane of the right camera; These are the disparity values ​​of corresponding pixels in the left and right images.

7. A lightweight human body detection and ranging system suitable for stage lighting fixtures, applied to the lightweight human body detection and ranging method for stage lighting fixtures as described in claim 1, characterized in that, include: The acquisition module is used to acquire the left and right visible light images and scene infrared images captured by the binocular camera; The correction module is used to perform distortion correction on the left and right visible light images and the scene infrared image based on the camera calibration parameters of the binocular camera, and output the corrected left and right visible light images and the corrected scene infrared image. The left image region of interest generation module is used to generate the target region of interest in the left image based on the corrected left visible light image and the corrected scene infrared image in the corrected left and right visible light images using an improved YOLO11n model; The right image region of interest generation module is used to generate the right image region of interest based on the left image target region of interest and the corrected right visible light image in the corrected left and right visible light images; The human detection module is used to perform human distance detection based on the target region of interest in the right image and the target region of interest in the left image using a stereo matching algorithm, and outputs the distance between the human body and the stereo camera.

8. A computer device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the lightweight human body detection and ranging method for stage lighting as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the lightweight human body detection and ranging method for stage lighting as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, wherein when the program instructions are executed by a computer, the computer performs the lightweight human body detection and ranging method for stage lighting as described in any one of claims 1-6.

Images