Dizziness lamp control method and system based on human body depth estimation and target detection

Abstract

The application discloses a dizziness lamp control method and system based on human body depth estimation and target detection, comprising: acquiring a first data set and a second data set; training a pre-constructed depth estimation model according to the first data set, and then training a pre-constructed pedestrian detection model according to the second data set to obtain a human body depth estimation network; collecting a current monocular image, inputting the current monocular image into the human body depth estimation network to obtain human body depth information; and adjusting the brightness of the dizziness lamp according to the human body depth information. The application combines target detection and depth estimation to obtain human body depth information, can automatically and dynamically adjust the brightness of the dizziness lamp according to different target pedestrian distances, improves the single use time length of the dizziness lamp under the premise of ensuring the effect of the dizziness lamp, and can be widely applied to the technical field of light control.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of light control, in particular to a dizziness lamp control method and system based on human body depth estimation and target detection. BACKGROUND

[0002] The dizziness lamp is an auxiliary instrument for military or police use. The existing dizziness lamp flicker brightness is fixed brightness or manually adjusted. When the target is far away and the brightness is low, the dizziness lamp cannot play a role. When the target is close and the brightness is large, due to the limited battery capacity of the dizziness lamp, the single use time of the dizziness lamp is reduced. SUMMARY

[0003] To solve the above technical problems, the purpose of the present application is to provide a dizziness lamp control method and system based on human body depth estimation and target detection, which can improve the single use time of the dizziness lamp.

[0004] To achieve the above purpose, one aspect of the embodiment of the present application provides a dizziness lamp control method based on human body depth estimation and target detection, comprising the following steps:

[0005] Obtain a first data set and a second data set;

[0006] Train a pre-constructed depth estimation model according to the first data set, and then train a pre-constructed pedestrian detection model according to the second data set to obtain a human body depth estimation network;

[0007] Collect a current monocular image, input the current monocular image into the human body depth estimation network, and obtain human body depth information;

[0008] Adjust the brightness of the dizziness lamp according to the human body depth information.

[0009] In some embodiments, the dizziness lamp control method further comprises the step of pre-constructing the depth estimation model, and the pre-constructing the depth estimation model specifically comprises:

[0010] The first convolutional layer and the second convolutional layer are connected in series, and the first convolutional layer and the second convolutional layer connected in series are connected in parallel with two first convolutional layers to obtain a constructed RepConv module;

[0011] The RepConv module, the up-sampling module and the RepConv module are connected in series to obtain a constructed Upsample Block module;

[0012] The pre-set CDC module is improved based on enhanced spatial attention to obtain a constructed ESDC module;

[0013] According to the average pooling layer, the RepConv module, the ESDC module, the Upsample Block module, the up-sampling module, and a prediction head, the depth estimation model is constructed.

[0014] In some embodiments, the dizziness lamp control method further comprises a step of pre-constructing the pedestrian detection model, and the pre-constructing the pedestrian detection model specifically comprises:

[0015] The first convolutional layer and the second convolutional layer are connected in series, and the first convolutional layer and the second convolutional layer connected in series and the two first convolutional layers are connected in parallel to obtain a constructed RepConv module.

[0016] The RepConv module, a batch normalization layer, and a first activation function layer are connected in series to obtain a constructed RBR module.

[0017] The two RBR modules are connected in series to obtain a constructed DWB module.

[0018] The second convolutional layer, the batch normalization layer, and a second activation function layer are connected in series to obtain a constructed CBS module.

[0019] The second convolutional layer, the batch normalization layer, and a third activation function layer are connected in series to obtain a constructed CBL module.

[0020] The RBR module and the DWB module are connected in series, and the RBR module and the DWB module connected in series and the DWB module are connected in parallel to obtain a constructed SFB module.

[0021] According to the CBL module, the SFB module, the CBS module, the DWB module, an up-sampling module, and the RepConv module, the pedestrian detection model is constructed.

[0022] In some embodiments, the pre-constructed depth estimation model is trained according to the first data set, specifically comprising:

[0023] The parameters of the pedestrian detection model are frozen.

[0024] A first training loss function is constructed according to an image reconstruction loss and an image smoothing loss.

[0025] The first data set is input into the depth estimation model, and the depth estimation model is trained by the first training loss function until the depth estimation model reaches a preset convergence condition, and the training is stopped.

[0026] In some embodiments, the training of the pre-constructed pedestrian detection model according to the second data set to obtain a human body depth estimation network specifically comprises:

[0027] label filtering is performed on the second data set;

[0028] parameter freezing is performed on the depth estimation model;

[0029] a second training loss function is constructed according to a positioning loss, a classification loss and a confidence loss;

[0030] the second data set after label filtering is input into the pedestrian detection model, and the pedestrian detection model is trained through the second training loss function until the pedestrian detection model reaches a preset convergence condition, the training is stopped, and the human body depth estimation network is obtained.

[0031] In some embodiments, the first training loss function is:

[0032]

[0033] wherein L1 represents the first training loss function, L p represents the image reconstruction loss, L smooth represents the image smoothing loss, s represents an output of different scales, and μ is 0.001.

[0034] In some embodiments, the input of the current monocular image into the human body depth estimation network to obtain human body depth information specifically comprises:

[0035] the current monocular image is input into the human body depth estimation network;

[0036] the pedestrian coordinates are obtained from the current monocular image through the trained pedestrian detection model;

[0037] the depth map is obtained from the current monocular image through the trained depth estimation model;

[0038] the pedestrian coordinates are mapped into the depth map to obtain the human body depth information.

[0039] To achieve the above-mentioned purposes, another aspect of the embodiments of the present application proposes a dizziness lamp control system based on human body depth estimation and target detection, comprising:

[0040] a data set acquisition module configured to acquire a first data set and a second data set;

[0041] a network construction module configured to train a pre-constructed depth estimation model according to the first data set, and further train a pre-constructed pedestrian detection model according to the second data set, to obtain a human body depth estimation network;

[0042] a human body depth information acquisition module configured to acquire a current monocular image, input the current monocular image into the human body depth estimation network, and obtain human body depth information;

[0043] a light control module configured to adjust the brightness of the dazzling light according to the human body depth information.

[0044] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides an electronic device, which comprises a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing the connection communication between the processor and the memory, and the program is executed by the processor to realize the dazzling light control method based on human body depth estimation and target detection as described above.

[0045] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides a storage medium, which is a computer readable storage medium for computer readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to realize the dazzling light control method based on human body depth estimation and target detection as described above.

[0046] The beneficial effects of the present application are that the dazzling light control method and system based on human body depth estimation and target detection of the present application firstly acquires a first data set and a second data set, then trains a pre-constructed depth estimation model according to the first data set, further trains a pre-constructed pedestrian detection model according to the second data set, obtains a human body depth estimation network, then acquires a current monocular image, inputs the current monocular image into the human body depth estimation network, obtains human body depth information, and finally adjusts the brightness of the dazzling light according to the human body depth information. The present application combines target detection and depth estimation to obtain human body depth information, can automatically and dynamically adjust the brightness of the dazzling light according to different target pedestrian distances, and improves the single use time length of the dazzling light under the premise of ensuring the effect of the dazzling light. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following introduces the drawings needed to be used in the embodiments of the present application. It should be understood that the drawings introduced below are only for facilitating the clear description of some embodiments in the technical solutions of the present application, and other drawings can also be obtained according to these drawings without creative labor for those skilled in the art.

[0048] Figure 1 A step flow chart of a dizziness lamp control method based on human body depth estimation and target detection provided for an embodiment of the present application is shown in FIG. 14;

[0049] Figure 2 An original image example diagram of a KITTI Depth dataset provided for an embodiment of the present application is shown in FIG. 15;

[0050] Figure 3 An RGB image example diagram of a KITTI Depth dataset provided for an embodiment of the present application is shown in FIG. 16;

[0051] Figure 4 A structure schematic diagram of a depth estimation model provided for an embodiment of the present application is shown in FIG. 17;

[0052] Figure 5 A structure schematic diagram of an improved Upsampling Block module provided for an embodiment of the present application is shown in FIG. 18;

[0053] Figure 6 A structure schematic diagram of an original CDC module provided for an embodiment of the present application is shown in FIG. 19;

[0054] Figure 7 A structure schematic diagram of an improved ESDC module provided for an embodiment of the present application is shown in FIG. 20;

[0055] Figure 8 A structure schematic diagram of an ESA module provided for an embodiment of the present application is shown in FIG. 21;

[0056] Figure 9 A structure schematic diagram of a pedestrian detection model provided for an embodiment of the present application is shown in FIG. 22;

[0057] Figure 10 A structure schematic diagram of a RepConv module provided for an embodiment of the present application is shown in FIG. 23;

[0058] Figure 11 A structure schematic diagram of other modules used by a target detection module provided for an embodiment of the present application is shown in FIG. 24;

[0059] Figure 12 An example diagram of a current monocular image provided for an embodiment of the present application is shown in FIG. 25;

[0060] Figure 13 A target detection diagram of a current monocular image provided for an embodiment of the present application is shown in FIG. 26;

[0061] Figure 14 A depth diagram of a current monocular image provided for an embodiment of the present application is shown in FIG. 27;

[0062] Figure 15 A matching diagram of a current monocular image provided for an embodiment of the present application is shown in FIG. 28;

[0063] Figure 16 A structural schematic diagram of a dizziness lamp control system based on human body depth estimation and target detection is provided for an embodiment of the present application.

[0064] Figure 17 A hardware structural schematic diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all the implementations consistent with the embodiments of the present application, but are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0066] It can be understood that the terms "first", "second", and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0067] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0068] The dizziness lamp is an auxiliary instrument for military or police use. The existing dizziness lamp flicker brightness is fixed brightness or manual adjustment. When the target is far away and the brightness is low, the dizziness lamp cannot play a role; when the target is close and the brightness is large, due to the limited battery capacity of the dizziness lamp, the single use time of the dizziness lamp is reduced.

[0069] To this end, the embodiment of the present application provides a dizziness lamp control method based on human depth estimation and target detection. First, a first data set and a second data set are obtained. Then, a pre-constructed depth estimation model is trained according to the first data set, and a pre-constructed pedestrian detection model is trained according to the second data set to obtain a human depth estimation network. Then, a current monocular image is collected, and the current monocular image is input into the human depth estimation network to obtain human depth information. Finally, the brightness of the dizziness lamp is adjusted according to the human depth information. The dizziness lamp control method based on human depth estimation and target detection combines target detection and depth estimation to obtain human depth information, and can automatically and dynamically adjust the brightness of the dizziness lamp according to different target pedestrian distances, so as to improve the single use time of the dizziness lamp under the premise of ensuring the effect of the dizziness lamp. The dizziness lamp control method based on human depth estimation and target detection can also be applied to street lamp control, lighting lamp control and the like, but is not limited thereto.

[0070] Reference Figure 1 , Figure 1 A step flowchart of a dizziness lamp control method based on human depth estimation and target detection provided by the embodiment of the present application is provided. The embodiment of the present application provides a dizziness lamp control method based on human depth estimation and target detection. The method comprises steps S101 to S105.

[0071] S101, obtaining a first data set and a second data set;

[0072] In some optional embodiments, considering that the embodiment of the present application is used outdoors, the first data set adopts a KITTI Depth data set. The data set is mainly used for outdoor monocular depth estimation. The KITTI Depth data set is well-known for its high precision and diverse scenes, covering various environments such as cities, villages and highways. The KITTI Depth data set not only provides rich depth information, but also contains corresponding RGB images, so that multi-modal analysis can be performed between visual and depth information. Figure 2 The original image example of the KITTI Depth data set is as shown in Figure 3 The RGB image example of the KITTI Depth data set is as shown in

[0073] The second data set adopts a COCO data set. The COCO data set contains more than 330,000 images, of which 220,000 images are annotated in detail. These images cover 80 target categories, including pedestrians, cars and elephants.

[0074] S102, training a pre-constructed depth estimation model according to the first data set, and then training a pre-constructed pedestrian detection model according to the second data set to obtain a human depth estimation network;

[0075] Specifically, the embodiment of the present application adopts an end-to-end model design, and a human depth estimation network includes two sub-tasks: depth estimation and pedestrian detection. First, a depth estimation model is used to output depth information of the entire image, and a target detection is used to locate the human body position. According to the human body coordinates of the target detection and the depth map, the human depth information is obtained. In order to ensure the performance of the model, an end-to-end network structure is designed in the embodiment of the present application, and the input is an image, and the output is the human depth information in the image.

[0076] Further, as an optional implementation, the dizziness lamp control method further includes the step of pre-constructing a depth estimation model. The step of pre-constructing the depth estimation model can be divided into the following steps A1021 to A1024:

[0077] A1021, the first convolutional layer and the second convolutional layer are connected in series, and the first convolutional layer and the second convolutional layer connected in series are connected in parallel with the two first convolutional layers to obtain a constructed RepConv module;

[0078] A1022, the RepConv module, the up-sampling module and the RepConv module are connected in series to obtain a constructed Upsample Block module;

[0079] A1023, the preset CDC module is improved based on enhanced spatial attention to obtain a constructed ESDC module;

[0080] A1024, the depth estimation model is constructed according to the average pooling layer, the RepConv module, the ESDC module, the Upsample Block module, the up-sampling module and the prediction head.

[0081] In some optional embodiments, as Figure 4 The structure diagram of the depth estimation model, the depth estimation model of the present application is improved based on a self-supervised depth estimation network. The human depth estimation network based on the model usually has high calculation cost. The dizziness lamp chip adopts a low-power chip, and the data is processed at the acquisition end. Therefore, the self-supervised depth estimation network is optimized in the embodiment of the present application to adapt to the edge power chip and reduce the cost of the device.

[0082] Specifically, the self-supervised depth estimation network is improved as follows in the embodiment of the present application:

[0083] 1) Delete the LGFI Block module. Considering that the embodiment of the present application only needs to roughly estimate the depth and the deployment device has limited computing power, the module is deleted to reduce the calculation amount;

[0084] 2) Delete the PoseNet posture network to further reduce the calculation amount;

[0085] 3) Improve the Upsampling Block module by replacing the 3×3 convolutions with RepConv modules. The improved Upsampling Block module is as follows: Figure 5 As shown, Upsample represents the bilinear upsampling operation;

[0086] 4) Replace the CDC (Consecutive Dilated Convolutions) module with an ESDC (Enhance Spatial Dilated Convolutions) module using Enhanced Spatial Attention (ESA). The original CDC module is as follows: Figure 6 As shown, the improved ESDC module is as follows: Figure 7 As shown;

[0087] 5) Add an ESA module after connecting the original CDC module. The ESA module is as follows: Figure 8 As shown, the input data first passes through a first 1×1 convolutional layer to obtain shallow features. Then, it passes through a 3×3 convolutional layer with a stride of 2 (StridedConv), a max pooling layer, a 3×3 convolutional layer with padding of 1 (Conv Groups), and a bilinear interpolation upsampling layer to obtain deep features. The deep features are added to the shallow features through a jump connection, and then processed by a second 1×1 convolutional layer. The output of the second 1×1 convolutional layer is processed by a sigmoid activation function and then multiplied by the input through a jump connection to obtain the final output.

[0088] As an optional implementation, the glare lamp control method further includes a step of pre-building a pedestrian detection model, which can be specifically divided into the following steps B1021 to B1027:

[0089] B1021. Connect the first convolutional layer and the second convolutional layer in series, and then connect the series-connected first and second convolutional layers in parallel with the two first convolutional layers to obtain the constructed RepConv module.

[0090] B1022. Concatenate the RepConv module, the batch normalization layer, and the first activation function layer to obtain the constructed RBR module;

[0091] B1023. Connect the two RBR modules in series to obtain the constructed DWB module;

[0092] B1024, the second convolutional layer, the batch normalization layer and the second activation function layer are connected in series to obtain a constructed CBS module;

[0093] B1025, the second convolutional layer, the batch normalization layer and the third activation function layer are connected in series to obtain a constructed CBL module;

[0094] B1026, the RBR module and the DWB module are connected in series, and the connected RBR module and the DWB module are connected in parallel with the DWB module to obtain a constructed SFB module;

[0095] B1027, the pedestrian detection model is constructed according to the CBL module, the SFB module, the CBS module, the DWB module, the up-sampling module and the RepConv module.

[0096] In some optional embodiments, as Figure 9 is a structural diagram of a pedestrian detection model, the pedestrian detection model of the present application is improved based on a yolov5-lite network, the human body depth estimation network based on the model usually has high calculation cost, and the dizziness lamp core uses a low-power chip to process data at the acquisition end, therefore, the yolov5-lite network is optimized in the embodiments of the present application to adapt to the edge power chip and reduce the device cost.

[0097] Specifically, the yolov5-lite network is improved in the embodiments of the present application as follows:

[0098] 1) The 3x3 convolution in the original CBR (Conv+BN+ReLU, convolution+normalization+ReLU activation function) module is replaced by the RepConv module designed in the embodiments of the present application, the RepConv module is composed of a reparameterization module, and the structure of the RepConv module is as Figure 10 shown. The RepConv module contains two parallel 3x3 convolution layers (the first convolution layer) in training, a 1x1 convolution layer (the second convolution layer) and a 3x3 convolution layer (the first convolution layer) in series, and the RepConv module is equivalent to a 3x3 convolution layer in the inference stage by using the reparameterization technology.

[0099] 2) Part of the DWB structure is omitted to further reduce the calculation amount, and other modules used by the target detection module in the embodiments of the present application are as Figure 11As shown in the figure, the RBR module is composed of a Repconv module, a BN (batch normalization layer), and a ReLU (first activation function layer), the DWB module is composed of two RBR modules, the CBS module is composed of a 1x1 convolution layer, a BN (batch normalization layer), and a SiLU (second activation function layer), and the CBL module is composed of a 1x1 convolution layer, a BN (batch normalization layer), and a Leaky ReLU (third activation function layer).

[0100] Further, as an optional implementation, the step of training the pre-constructed depth estimation model according to the first data set can be specifically divided into the following steps S1021 to S1024:

[0101] S1021, training the pre-constructed depth estimation model according to the first data set, specifically including:

[0102] S1022, parameter freezing of the pedestrian detection model;

[0103] S1023, constructing a first training loss function according to the image reconstruction loss and the image smoothing loss;

[0104] S1024, inputting the first data set into the depth estimation model, training the depth estimation model through the first training loss function, and stopping the training until the depth estimation model reaches a preset convergence condition.

[0105] Further, as an optional implementation, the first training loss function is:

[0106]

[0107] wherein, L1 represents the first training loss function, L p represents the image reconstruction loss, L smooth represents the image smoothing loss, s represents an output of different scales, and μ is 0.001.

[0108] Specifically, the model training method of the embodiment of the application is a phased training, first, the parameters of the target detection model are frozen, and only the depth estimation module is trained. The depth estimation training includes two loss functions: an image reconstruction loss function and an image smoothing loss function, and the image reconstruction loss function is defined as follows:

[0109] L p (I d ,I t )=L p (F(I s ,D t ,K),I t )

[0110] wherein, I dwhich can be obtained by a function F, the function being designed to obtain the original image I s , the predicted depth D t and the camera intrinsic K, L p for I d , I t , the sum of the structural similarity SSIM and the L1 loss between them, the mathematical formula being as follows:

[0111]

[0112] wherein a is set to 0.85. And the mathematical formula of the image smoothing function is as follows:

[0113]

[0114] wherein, is the mean value normalized inverse depth, and thus the first training loss function is obtained as:

[0115]

[0116] wherein s represents the output of different scales, and μ is set to 0.001. After the depth estimation module is trained to reach a convergence condition (such as the number of iterations reaching a preset number of iterations), the model weight is saved, and the second stage training of the target detection model is started.

[0117] Further as an optional implementation, the pre-constructed pedestrian detection model is trained according to the second data set to obtain the human body depth estimation network, which can be specifically divided into the following steps S1025 to S1024:

[0118] S1025, label filtering is performed on the second data set;

[0119] S1026, the parameters of the depth estimation model are frozen;

[0120] S1027, a second training loss function is constructed according to the localization loss, the classification loss and the confidence loss;

[0121] S1028, the second data set after label filtering is input into the pedestrian detection model, and the pedestrian detection model is trained through the second training loss function until the pedestrian detection model reaches a preset convergence condition, the training is stopped, and the human body depth estimation network is obtained.

[0122] Specifically, when the target detection model is trained, the related parameters of the depth estimation model are frozen, and only the related parameters of the target detection model are trained, wherein the loss function of the target detection model includes the localization loss, the classification loss and the confidence loss, and the mathematical formula of the localization loss CIoU is as follows:

[0123]

[0124] Among them, L box Let IoU(b,b′) represent the localization loss, where IoU(b,b′) is the intersection ratio between the predicted bounding box b′ and the ground truth bounding box b, and ρ is the localization loss. 2 (b, b′) is the squared Euclidean distance between the center points of the predicted box and the ground truth box b, c is the diagonal length of the minimum bounding box (the smallest rectangle containing both the ground truth box b and the predicted box b′), v is the aspect ratio difference measure between the predicted box and the ground truth box b, and β is a parameter used to balance the loss by adjusting v and ρ.

[0125] The mathematical formula for the classification loss function is as follows:

[0126]

[0127] Among them, L cls Let y represent the classification loss. c For the one-hot encoding of the true label, if category c is the true category, y c =1, otherwise y c =0, y′ c This is the probability value for predicting category c, where C is the total number of categories.

[0128] The mathematical formula for the confidence loss function is as follows:

[0129]

[0130] Among them, L obj Indicates confidence loss, 1 obj (i) is an indicator function, which is 1 if the i-th box contains an object. obj (i) = 1, otherwise 1 obj (i) = 0, 1 noobj (i) is also an indicator function; if the i-th box does not contain an object, then 1 moobj (i) = 1, otherwise 1 noobj (i)=0,p′ i Let be the confidence prediction value for the i-th box.

[0131] Therefore, the second training loss function is as follows:

[0132] L=λ box L box +λ cls L cls +λ obj L obj

[0133] Where L2 represents the second training loss function, L box L represents the localization loss. cls L represents the classification loss. obj λ represents the confidence loss.box represents the positioning loss weight, which is set to 0.05 in the embodiment of the present application, λ cls represents the classification loss weight, which is set to 1.0 in the embodiment of the present application, λ obj represents the confidence loss weight, which is set to 1.0 in the embodiment of the present application. After the human body detection training reaches the convergence condition (such as the number of iterations reaches the preset number of iterations), the model weight is saved to obtain the human body depth estimation network.

[0134] S103, collect the current monocular image, input the current monocular image into the human body depth estimation network to obtain human body depth information;

[0135] In some optional embodiments, the human body depth estimation network with good training effect is deployed on the embedded chip of the dizziness lamp, and in actual use, the video stream data collected by the monocular camera is input into the algorithm program, the algorithm is inferred to obtain human body depth information, and the light is adjusted according to the human body depth information.

[0136] Further as an optional implementation, the current monocular image is input into the human body depth estimation network to obtain human body depth information, which specifically includes this step, and can be specifically divided into the following steps S1031 to S1034:

[0137] S1031, input the current monocular image into the human body depth estimation network;

[0138] It should be noted that the present application can complete human body depth estimation using images collected by a monocular camera, has lower model cost and better adaptability, and can have good performance in complex environments.

[0139] S1032, obtain pedestrian coordinates according to the current monocular image through the trained pedestrian detection model;

[0140] Specifically, referring to Figure 9The image processing flow of the target detection module as a whole is as follows: the input current monocular image is processed by the CBL module, then processed by the first SFB module and the second SFB module, the output result of the second SFB module is processed by the third and fourth SFB modules, the output result of the fourth SFB module is processed by the fifth and sixth SFB modules, then processed by the CBS module, the processing result of the CBS module is processed by the upsample block, the output of the upsample block and the output of the fourth SFB module are subjected to a first splicing operation, the output of the first splicing operation is processed by the first DWB module, the output of the first DWB module and the output of the second SFB are subjected to a second splicing operation, the output of the second splicing operation is processed by the second DWB module, the result of the second DWB module is processed by the first RepConv module to obtain a first output; at the same time, the outputs of the second DWB module and the first DWB module are subjected to a first addition operation, the first addition operation is processed by the third DWB module, the output of the third DWB module is processed by the second RepConv module to obtain a second output; at the same time, the output of the third DWB module and the output of the CBS module are added, and then processed by the fourth DWB module, the output of the fourth DWB module is processed by the third RepConv module to obtain a third output. Finally, the first, second and third outputs are subjected to a third splicing operation to obtain the final output (Output1) of the target detection module, i.e. the pedestrian coordinates.

[0141] S1033, obtaining a depth map according to the current monocular image by using the trained depth estimation model;

[0142] Specifically, referring to Figure 4The image processing flow of the entire depth estimation model is as follows: the current monocular image is first processed through an average pooling layer (AvgPool) and a first RepConv module, the two are input through a fourth Concat layer for splicing, and then input to a second RepConv module for processing. The result of the second RepConv module is input to a first ESDC module for processing. The result of the first ESDC module is spliced with the result of the second RepConv module through a fifth Concat layer, and then input to a third RepConv module for processing. The result of the third RepConv module is input to a second ESDC module for processing. The result of the second ESDC module is spliced with the result of the third RepConv module through a sixth Concat layer, and then input to a fourth RepConv module for processing. The result of the fourth RepConv module is input to a third ESDC module for processing. The results of the second and third ESDC modules are processed through a first upsampling module (UpsampleBlock), and then processed through a first prediction head (Prediction Head) to obtain a small-scale depth map. The result of the first upsampling module is processed through a second upsampling module (Upsample Block), and then processed through a second prediction head (Prediction Head) to obtain a medium-scale depth map. The result of the second upsampling module is processed through a third upsampling module (Upsample Block), and then processed through a third prediction head (Prediction Head) to obtain a large-scale depth map. Then, the small, medium and large scale depth maps are input into an image reconstruction module to obtain the final output (Output2), i.e., a depth map. The image reconstruction module is a bilinear interpolation operation, which selects one of the three scale depth maps for interpolation transformation to make the resolution correspond to the original image.

[0143] S104, adjusting the brightness of the dizziness lamp according to the human body depth information.

[0144] Specifically, as Figure 12 is an example image of the current monocular image, as Figure 13 is a target detection image of the current monocular image, as Figure 14 is a depth map of the current monocular image, as Figure 15 is a matching image of the current monocular image. In the depth map, the brighter the target color, the closer the target, and the darker the color, the farther the target. The corresponding light brightness is low, medium and high. For example, the left target is far away, so the light brightness is adjusted the most; the target in the middle region is medium, so the light brightness is medium; the right target is close, so the light brightness is the smallest.

[0145] In summary, the processing flow of the dizziness lamp control method based on human depth estimation and target detection according to the embodiment of the present application is as follows: including four parts of data set preparation, algorithm model construction, model training and testing, and light control. First, based on the online open source data set for training, the data set contains monocular depth estimation data set (i.e. the first data set) and pedestrian detection data set (i.e. the second data set); then construct a lightweight pedestrian detection model and a depth estimation model; then based on the data set, first train the depth estimation model parameters, freeze the pedestrian detection model parameters, and then freeze the network structure after the depth estimation model parameters converge, and then train the pedestrian detection model parameters; finally, save the trained human depth estimation network and deploy it on the dizziness lamp chip, use the monocular camera to collect image information, input the human depth estimation network, the human depth estimation network outputs the human depth information, and adjusts the light brightness of the dizziness lamp according to the human depth information.

[0146] The above describes the dizziness lamp control method based on human depth estimation and target detection according to the embodiment of the present application. It can be realized that the embodiment of the present application has the following advantages:

[0147] I. The combination of target detection and depth estimation can obtain human depth information, automatically and dynamically adjust the brightness of the dizziness lamp according to different target pedestrian distances, and improve the single use time of the dizziness lamp under the premise of ensuring the effect of the dizziness lamp;

[0148] II. Combined with the generalization ability of deep learning, the human depth estimation can be completed using the images collected by the monocular camera, the model cost is lower and the adaptability is stronger, and the model can have good performance in complex environments.

[0149] III. The self-supervised depth estimation network and yolov5-lite network are optimized to adapt to edge computing chips, while reducing the cost of the device.

[0150] Reference Figure 16 The embodiment of the present application also provides a dizziness lamp control system based on human depth estimation and target detection, comprising:

[0151] A data set acquisition module is configured to acquire a first data set and a second data set;

[0152] A network construction module is configured to train a pre-constructed depth estimation model according to the first data set, and then train a pre-constructed pedestrian detection model according to the second data set, to obtain a human depth estimation network;

[0153] A human depth information acquisition module is configured to acquire a current monocular image, input the current monocular image into the human depth estimation network, and obtain human depth information;

[0154] The light control module is used for adjusting the brightness of the dazzling light according to the human body depth information.

[0155] The contents in the above-mentioned dazzling light control method embodiment based on human body depth estimation and target detection are applicable to the present dazzling light control system embodiment based on human body depth estimation and target detection, the present dazzling light control system embodiment based on human body depth estimation and target detection has the same functions as the above-mentioned dazzling light control method embodiment based on human body depth estimation and target detection, and has the same beneficial effects as the above-mentioned dazzling light control method embodiment based on human body depth estimation and target detection.

[0156] The embodiment of the present application further provides an electronic device, which comprises a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned dazzling light control method based on human body depth estimation and target detection. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer and the like.

[0157] As Figure 17 Fig. 1 shows a hardware structure schematic diagram of an electronic device provided by the embodiment of the present application, referring to Figure 17 The embodiment of the present application provides an electronic device, which comprises:

[0158] The processor 1001 can be realized in the mode of a general CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit) or one or more integrated circuits, and is used for executing a related program to realize the technical scheme provided by the embodiment of the present application.

[0159] The memory 1002 can be realized in the mode of a ROM (Read Only Memory), a static storage device, a dynamic storage device or a RAM (Random Access Memory), and the like. The memory 1002 can store an operating system and other application programs, and when the technical scheme provided by the embodiment of the present application is realized by software or firmware, the related program codes are saved in the memory 1002 and are called and executed by the processor 1001 to realize the dazzling light control method based on human body depth estimation and target detection of the embodiment of the present application.

[0160] The input / output interface 1003 is used for realizing information input and output.

[0161] The communication interface 1004 is configured to realize the communication interaction between the device and other devices, and can realize the communication through a wired manner (for example, a USB, a network cable and the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth and the like).

[0162] The bus 1005 is configured to transmit information between various components (for example, the processor 1001, the memory 1002, the input / output interface 1003 and the communication interface 1004) of the device.

[0163] The processor 1001, the memory 1002, the input / output interface 1003 and the communication interface 1004 are connected to each other through the bus 1005 to realize the communication connection between the device.

[0164] The embodiment of the present application further provides a storage medium, the storage medium is a computer readable storage medium, and the storage medium is used for computer readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to realize the above-mentioned dizziness lamp control method based on human body depth estimation and target detection.

[0165] The memory is a non-transient computer readable storage medium, and can be used to store a non-transient software program and a non-transient computer executable program. In addition, the memory can include a high-speed random access memory, and can further include a non-transient memory, for example, at least one magnetic disk storage device, a flash memory device or other non-transient solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0166] The embodiment of the present application further discloses a computer program product or a computer program, and the computer program product or the computer program includes computer instructions stored in a computer readable storage medium. The processor of the computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method shown in the above. Figure 1

[0167] ​In alternative embodiments, the functions / operations in the flow diagrams can occur in sequences other than those depicted. For example, two operations shown in succession can in fact be executed substantially concurrently or the operations can sometimes be executed in the reverse order depending upon the functionality / operations involved. Such variations are contemplated to be within the scope of the present application. Embodiments presented and described in the flow diagrams are examples only and are used to provide an enabling teaching for the present application. The processes disclosed are not limited to the order or specific blocks described. Alternative embodiments are contemplated, in which the order of the blocks is changed and where some blocks are performed in parallel rather than sequentially.

[0168] Moreover, while the present application has been described in the context of functional modules, it is to be understood that one or more of the functions and / or features described above can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It will also be appreciated that detailed discussion of the actual implementation of each module is not necessary for an enabling understanding of the application. Rather, the actual implementation is most readily derived from the description of the functionality of the various functional modules in conjunction with the drawings. Accordingly, the disclosure of the present application is intended to be illustrative, but not limiting, of the scope of the application, which is set forth in the following claims. The claims, themselves, however, expressly include all available bases of recited matters in the claims. Moreover, since amounts, dosages, and other quantities are expressed herein in a range format, it is to be understood that every quantity between the stated limits, including endpoints, is also specifically intended.

[0169] The functions described above can be implemented in software and / or firmware using the system shown in the figures. The software implemented as a computer program, software, or firmware instructions can be stored in one or more computer-readable media used to program computer processors. Computer-readable media include computer storage media and communication media. Storage media, such as main memory or a system memory such as read-only memory (ROM), random access memory (RAM), cache, flash memory devices, optical storage such as CD-ROMs, other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other memory storage devices including one or more of the above or the like, can be employed in the implementation. Communication media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, other wireless media, or the like. Combinations of the above can also be employed to implement the disclosed techniques. Accordingly, computer-readable media of the present application generally include computer storage media and communication media.

[0170] The logic and / or steps represented in the flow diagrams or otherwise described herein, for example, can be considered as a sequence of executable instructions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or a combination thereof. For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be for example but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include an electrical connection, hard-wired connection, fiber-optic cable, portable storage media, RAM, ROM, EEPROM, tape, magnetic disk, optical disk, optical fiber, and / or micro-wave transmission medium, and the like.

[0171] It should be understood that various parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, various steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or a combination thereof, can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0172] Although the embodiments of the present application have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, alternatives, and variations can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

[0173] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present application, and these equivalent modifications or substitutions are included in the scope defined by the claims of the present application.

Claims

1. A dizziness lamp control method based on human depth estimation and target detection, characterized in that, The method comprises the following steps: obtaining a first data set and a second data set; training a pre-constructed depth estimation model according to the first data set, and then training a pre-constructed pedestrian detection model according to the second data set to obtain a human body depth estimation network; collecting a current monocular image, inputting the current monocular image into the human body depth estimation network to obtain human body depth information; adjusting the brightness of the dizziness lamp according to the human body depth information; The method further comprises the step of pre-constructing the depth estimation model, which specifically comprises: connecting a first convolutional layer and a second convolutional layer in series, connecting the first convolutional layer and the second convolutional layer in series with two first convolutional layers in parallel to obtain a constructed RepConv module; connecting the RepConv module, an up-sampling module, and the RepConv module in series to obtain a constructed UpsampleBlock module; improving a pre-set CDC module based on enhanced spatial attention to obtain a constructed ESDC module; constructing the depth estimation model according to an average pooling layer, the RepConv module, the ESDC module, the UpsampleBlock module, the up-sampling module, and a prediction head; The method further comprises the step of pre-constructing the pedestrian detection model, which specifically comprises: connecting a first convolutional layer and a second convolutional layer in series, connecting the first convolutional layer and the second convolutional layer in series with two first convolutional layers in parallel to obtain a constructed RepConv module; connecting the RepConv module, a batch normalization layer, and a first activation function layer in series to obtain a constructed RBR module; connecting two RBR modules in series to obtain a constructed DWB module; connecting the second convolutional layer, the batch normalization layer, and a second activation function layer in series to obtain a constructed CBS module; connecting the second convolutional layer, the batch normalization layer, and a third activation function layer in series to obtain a constructed CBL module; connecting the RBR module and the DWB module in series, connecting the RBR module and the DWB module in series with the DWB module in parallel to obtain a constructed SFB module; constructing the pedestrian detection model according to the CBL module, the SFB module, the CBS module, the DWB module, an up-sampling module, and the RepConv module.

2. The dizziness lamp control method based on human depth estimation and target detection according to claim 1, characterized in that, The training of the pre-constructed depth estimation model according to the first data set specifically comprises: parameter freezing of the pedestrian detection model; constructing a first training loss function according to an image reconstruction loss and an image smoothing loss; inputting the first data set into the depth estimation model, training the depth estimation model through the first training loss function until the depth estimation model reaches a pre-set convergence condition, and stopping the training. 3.The dizziness lamp control method based on human depth estimation and target detection of claim 1, wherein, The training of the pre-constructed pedestrian detection model according to the second data set comprises the following steps: label filtering is performed on the second data set; parameters of the depth estimation model are frozen; a second training loss function is constructed according to a positioning loss, a classification loss and a confidence loss; the second data set after label filtering is input into the pedestrian detection model, and the pedestrian detection model is trained through the second training loss function until the pedestrian detection model reaches a preset convergence condition, and the training is stopped to obtain the human body depth estimation network.

4. The dizziness lamp control method based on human depth estimation and target detection according to claim 2, characterized in that, The first training loss function is: wherein, represents the first training loss function, represents the image reconstruction loss, L smooth represents the image smoothing loss, represents the output of different scales, is 0.

001.

5. The dizziness lamp control method based on human depth estimation and target detection according to any one of claims 1 to 4, characterized in that, The input of the current monocular image into the human body depth estimation network to obtain human body depth information comprises the following steps: the current monocular image is input into the human body depth estimation network; human body coordinates are obtained from the current monocular image through the trained pedestrian detection model; a depth map is obtained from the current monocular image through the trained depth estimation model; the human body coordinates are mapped into the depth map to obtain the human body depth information.

6. A dizziness lamp control system based on human depth estimation and object detection, characterized in that, It comprises: a data set acquisition module for acquiring a first data set and a second data set; a network construction module for training a pre-constructed depth estimation model according to the first data set, and then training a pre-constructed pedestrian detection model according to the second data set to obtain a human body depth estimation network; a human body depth information acquisition module for collecting a current monocular image, inputting the current monocular image into the human body depth estimation network, and obtaining human body depth information; a light control module for adjusting the brightness of a dazzling lamp according to the human body depth information; The network construction module is also used for pre-constructing the depth estimation model, and the pre-constructing of the depth estimation model comprises the following steps: the first convolutional layer and the second convolutional layer are connected in series, and the first convolutional layer and the second convolutional layer connected in series and two first convolutional layers are connected in parallel to obtain a constructed RepConv module; the RepConv module, the up-sampling module and the RepConv module are connected in series to obtain a constructed UpsampleBlock module; a pre-set CDC module is improved based on enhanced spatial attention to obtain a constructed ESDC module; the depth estimation model is constructed according to an average pooling layer, the RepConv module, the ESDC module, the UpsampleBlock module, the up-sampling module and a prediction head; The network construction module is also used for pre-constructing the pedestrian detection model, and the pre-constructing of the pedestrian detection model comprises the following steps: the first convolutional layer and the second convolutional layer are connected in series, and the first convolutional layer and the second convolutional layer connected in series and two first convolutional layers are connected in parallel to obtain a constructed RepConv module; the RepConv module, a batch normalization layer and a first activation function layer are connected in series to obtain a constructed RBR module; The two RBR modules are connected in series to obtain a built DWB module; The second convolutional layer, the batch normalization layer and the second activation function layer are connected in series to obtain a built CBS module; The second convolutional layer, the batch normalization layer and the third activation function layer are connected in series to obtain a built CBL module; The RBR module and the DWB module are connected in series, and the RBR module and the DWB module connected in series and the DWB module are connected in parallel to obtain a built SFB module; The pedestrian detection model is built according to the CBL module, the SFB module, the CBS module, the DWB module, the up-sampling module and the RepConv module.

7. An electronic device, comprising: The electronic device includes a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing connection communication between the processor and the memory, and the program is executed by the processor to realize the steps of the dizziness lamp control method based on human body depth estimation and target detection according to any one of claims 1 to 5.

8. A storage medium, the storage medium being a computer-readable storage medium for computer-readable storage, characterized in that, The storage medium stores one or more programs executable by one or more processors to realize the steps of the dizziness lamp control method based on human body depth estimation and target detection according to any one of claims 1 to 5.

Images