Foot brake state detection method based on improved semantic segmentation technology and electronic equipment

Through the improved real-time semantic segmentation technology, the STDCNet network and FLD Decoder decoder are used to solve the problems of low detection efficiency and low accuracy in the existing technology, and fast and accurate foot brake status detection is achieved, suitable for driver examination videos efficient processing.

CN118942001APending Publication Date: 2024-11-12DUOLUN TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410957015.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

When the existing automatic judgment system detects the foot brake status of the co-pilot position during the driver's subject 3 exam, it is inefficient and has low accuracy, and cannot quickly and accurately segment image details.

Method used

Adopting improved real-time semantic segmentation technology, using lightweight STDCNet network and FLD Decoder decoder, multi-scale feature information is extracted through frame-by-frame video data processing, reducing calculation amount, improving detection accuracy, and integrating multi-level features through UAFM modules to Improve segmentation accuracy.

Benefits of technology

It realizes fast and accurate foot brake status detection, reduces calculation amount, improves detection efficiency and accuracy, and is suitable for the rapid processing needs of actual exam videos.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942001A_ABST
    Figure CN118942001A_ABST
Patent Text Reader

Abstract

The invention discloses a foot brake state detection method based on an improved semantic segmentation technology. The method comprises the following steps: collecting an original frame image containing a co-driver seat foot and a foot brake; preprocessing the original frame image to obtain a feature matrix containing foot and foot brake information; inputting the feature matrix into a trained STDCNet network model to obtain a feature matrix containing deep features; decoding is carried out through a decoder FLD Decoder, and a fusion feature matrix containing key semantic information is obtained; and processing the fusion feature matrix to obtain masks capable of visually distinguishing different target objects, and judging whether the feet of the co-driver position step on a foot brake or not. According to the invention, the technical problems of low detection efficiency and low detection precision of the existing automatic judgment system can be solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of motor vehicle driver examinations, and specifically relates to a foot brake state detection method and electronic equipment based on improved semantic segmentation technology. Background Art

[0002] The third subject test for drivers stipulates that the supervisor in the co-pilot position is not allowed to step on the brakes without authorization. Therefore, it is necessary to judge whether the supervisor's foot is on the foot brake to ensure that the supervisor does not step on the brake without authorization during the test. In the past, it was mainly based on manual inspection of the test process video to see whether the supervisor stepped on the foot brake in the third subject test, which was very inefficient. With the popularization of automatic judgment systems in driving skills tests, semantic segmentation algorithms have been introduced into automatic judgment systems to realize the detection of the co-pilot position's feet and foot brakes.

[0003] Conventional semantic segmentation models are large and computationally cumbersome, and are usually not sensitive to image details and the segmentation is not fine enough. For example, the YOLO series segmentation algorithm model in real-time semantic segmentation technology has the problem of being large and not suitable for high-precision detail segmentation. In practice, the detection of the video of the examination process requires rapid processing of a large number of images and segmentation of target details. Therefore, a fast and accurate detection method is particularly necessary to meet the actual situation. Summary of the invention

[0004] In order to solve the problems of low detection efficiency and low detection accuracy in the existing automatic determination system, the present invention provides a foot brake state detection method based on improved real-time semantic segmentation technology, which uses a lightweight network, can greatly reduce the amount of model calculation, and has higher detection accuracy and detection efficiency. Furthermore, the present invention also provides an electronic device and a computer-readable storage medium for implementing the above detection method.

[0005] A first aspect of the present invention discloses a foot brake state detection method based on an improved semantic segmentation technology, comprising:

[0006] Step 1: Collect video data including the passenger seat foot and foot brake, and obtain several continuous original frame images by intercepting the video data frame by frame;

[0007] Step 2: pre-process the original frame image to obtain a feature matrix and a MASK containing foot and foot brake information;

[0008] Step 3: Input the feature matrix containing foot and foot brake information into the trained STDCNet network model to obtain a feature matrix containing deep features;

[0009] Step 4: Input the feature matrix and the feature matrix obtained by the last two convolutions in the STDCNet network model into the decoder FLD Decoder for decoding, downsample and classify the fused feature matrix output after decoding, and obtain a fused feature matrix containing key semantic information;

[0010] Step 5: Process the fused feature matrix to obtain a mask that can visually distinguish different targets, and determine whether the passenger's foot is on the foot brake based on the mask.

[0011] The second aspect of the present invention discloses an electronic device, which includes a processor and a memory; the processor is used to call a computer program stored in the memory to execute the foot brake state detection method based on the improved semantic segmentation technology described in the first aspect of the present invention.

[0012] The third aspect of the present invention discloses a computer-readable storage medium, characterized in that the computer-readable storage medium is used to store program code, and the program code is used to execute the foot brake state detection method based on the improved semantic segmentation technology described in the first aspect of the present invention.

[0013] The present invention has the following beneficial effects:

[0014] The present invention is based on improved real-time semantic segmentation technology and adopts a lightweight network STDCNet network and an FLDDecoder decoder, which can not only greatly reduce the amount of calculation, but also quickly perform semantic segmentation on the image, thereby realizing fast and accurate foot brake status detection.

[0015] The present invention uses the STDCNet network model to extract multi-scale feature information of the input image, which can improve the detection accuracy; by gradually reducing the number of feature channels and thus reducing the amount of calculation, it can achieve rapid extraction of feature information of the target object's feet and foot brake.

[0016] The present invention uses a lightweight FLD Decoder to fuse the output of the STDCNet network model, and by reducing the channels of the feature matrix output by the STDCNet network model, the spatial size of the feature map is increased, thereby efficiently obtaining more useful semantic information. In addition, the UAFM module it contains can fuse multi-level features, which can further improve segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a structural diagram of the STDCNet network model;

[0018] Figure 2 It is a schematic diagram of the data processing process of the STDCNet network model;

[0019] Figure 3Schematic diagram of the data processing process of the STDCNet network model and the decoder FLD Decoder;

[0020] Figure 4 It is a schematic diagram of the data processing process of the decoder FLD Decoder;

[0021] Figure 5 It is an illustration of the feature map of three categories of objects: feet, foot brakes, and background. DETAILED DESCRIPTION

[0022] In order to further understand the present invention, the technical solution of the present invention is further explained below in conjunction with specific embodiments.

[0023] The embodiment of the present invention provides a foot brake state detection method based on an improved real-time semantic segmentation technology, which mainly includes the following steps:

[0024] Step 1: Collect video data including the co-pilot's foot and foot brake through a camera installed in the cab, and obtain several continuous original frame images by intercepting the video data frame by frame.

[0025] Specifically, by collecting video data including the passenger's feet and foot brake under different scene lighting environments, we can obtain more interfered samples in different scenes to improve the versatility of the detection method.

[0026] Step 2: Preprocess the original frame image to obtain the feature matrix and MASK containing the foot and foot brake information.

[0027] Specifically, use labelme image annotation software to annotate the feet and foot brakes in the original frame image, and generate an annotation file containing the feet and foot brakes. Generate a mask containing the position information of the feet and foot brakes based on the original frame image and the corresponding annotation file. The original frame image is h*w*3, and the mask is h*w. Each value in the mask represents the category of the target detection object. Here, the foot is assigned a value of 1, the foot brake is assigned a value of 2, and the rest is assigned a value of 0 for the background.

[0028] The original frame image and mask are subjected to various random image interference and unified size to obtain a 1*3*256*256 feature matrix and a 256*256 mask. The mask can be used in the training process of the STDCNet network. The interference and unified size processing can increase the training samples and facilitate the subsequent network to process the input feature matrix.

[0029] Step 3: Input the feature matrix containing foot and foot brake information into the STDCNet network model to obtain a feature matrix containing deep features.

[0030] like Figure 1As shown in Figure 2, the main structure of the STDCNet network model includes the DetailGuidance module ( Figure 1 The green part) and the context path module for feature information classification ( Figure 1 orange part of the image).

[0031] The training process of the STDCNet network model mainly includes: building the initial STDCNet network; inputting the feature matrix containing the foot and foot brake information and the corresponding MASK as training data into the STDCNet network, using the Pytorch framework, setting Epochs to 200, setting the Batch Size to 16, and using the GPU training environment; each Epoch is first trained with the training data to obtain a feature matrix output; using this feature matrix output and the input MASK to calculate the loss through the OhemCELoss function, and adjusting the parameters in the STDCNet network according to the loss to optimize the model; finally, an optimal model with the smallest loss is obtained through training, which is the required STDCNet network model.

[0032] like Figure 2 As shown in the figure, the feature matrix containing the foot and brake information is input into the STDCNet network model, firstly undergoes convolution processing with a kernel of 1 (implemented by Block1), and then undergoes convolution processing with a kernel of 3 three times (implemented by Block2, Block3, and Block4 in sequence). In each convolution process, a feature matrix is ​​extracted through high-level semantics. After four convolution processes, four feature matrices with sizes of 1*64*128*128, 1*128*64*64, 1*256*32*32, and 1*512*16*16 are obtained respectively. Finally, the four feature matrices are fused in a cascade manner to obtain a feature matrix containing deep features, which is the network output of the STDCNet network model. The size of the feature matrix is ​​1*1024*8*8. Among them, the feature matrix obtained by the two convolution processes of Block3 and Block4 and the feature matrix containing deep features, that is, the three feature matrices with sizes of 1*256*32*32, 1*512*16*16, and 1*1024*8*8 will be used as input for the next step.

[0033] Step 4: The feature matrix containing the deep features and the feature matrix obtained by the last two convolution processes are input into the lightweight decoder FLD Decoder for decoding. The fused feature matrix output after decoding is downsampled and classified to obtain a fused feature matrix containing key semantic information.

[0034] like Figure 3As shown in FIG. 1 , the FLD Decoder includes a pooling module SPPM (simple Pyramid Pooling Module, referred to as “SPPM module”) and two cascaded fusion modules UAFM (Unified Attention Fusion Module, referred to as “UAFM module”). The present invention utilizes the FLD Decoder to fuse the output of the STDCNet network model. The FLD Decoder reduces the channels of the feature matrix output by the STDCNet network model from high-level to low -level layer, thereby reducing the feature channels and increasing the feature map space size. The UAFM module mainly plays the role of fusing multi-level features and improving segmentation accuracy.

[0035] Specifically, the feature matrix of size 1*1024*8*8 output by the STDCNet network model is first input into the SPPM module to obtain semantic information of different scales and enhance the perception of objects of different scales, and a feature matrix of size 16*128*8*8 is obtained; then the feature matrix of size 1*512*16*16 obtained by the Block4 convolution process and the output result of the SPPM module are input into the first UAFM module to obtain a feature matrix of size 16*96*16*16, and then the feature matrix and the feature matrix of size 1*256*32*32 are input into the second UAFM module to obtain a feature matrix of size 16*64*32*32. Finally, the fused feature matrix output by the FLD Decoder is resized, downsampled and classified (predict) to obtain a fused feature matrix containing key semantic information.

[0036] It can be understood that for each UAFM module, there is an F high and F low Input, where the SPPM module is defined as F low . The upsampled F high and F low The input attention mechanism obtains the weight matrix α of the UAFM module. Specifically, the spatial attention mechanism is used to calculate the input Upsample (F high ) and F low Calculate the mean along the channel (Mea n ) and maximum value (Ma x ), four feature matrices are obtained, and then these four feature matrices are combined into one feature matrix, and finally the weight matrix α is obtained after convolution (conv) and Sigmod processing. Among them, the calculation formula of the weight matrix α is as follows:

[0037] α=Sigmoid(Conv(Fcat ))

[0038] In the formula,

[0039] F cat =Concat[Mean(Upsample(F high )), Mean(F low ),Max(Upsample(F high )), Max(F low )]

[0040] The feature matrix output by the SPPM module (i.e., high-level semantic information) and the feature matrix output by the STDCNet network model (i.e., low-level semantic information) are multiplied by the weight matrix α to obtain the fused feature matrix.

[0041] like Figure 4 As shown, taking the first UAFM module as an example, the inputs in the UAFM module are the STDCNet network model output F low (1*512*16*16) and SPPM module output F high (1*1024*8*8). First, low and F high Perform bilinear interpolation to make F high Upsample to and F low The same size; then the processed F high and F low The weight matrix α is obtained by the input attention mechanism; the two inputs are multiplied by the weight matrix α and (1-α) respectively, and the fused feature matrix F is output out (16*96*16*16).

[0042] The fused feature matrix F out (16*96*16*16) as the input F of the next UAFM module high , the F obtained by convolution processing of Block3 of STDCNet network model low (1*256*32*32) and F high (16*96*16*16) Repeat the above operation to obtain the fusion feature matrix F containing key semantic information output by the decoder out (16*64*32*32).

[0043] Among them, the fused feature matrix F out The resulting formula is as follows:

[0044]

[0045] Step 5: The fused feature matrix containing key semantic information output by the decoder FLD Decoder is processed by taking the maximum value to obtain a mask that can intuitively distinguish different targets. The IOU between the foot and the foot brake is calculated based on the mask, and based on this, it is determined whether the foot is on the foot brake.

[0046] The fusion feature matrix F containing key semantic information output by the decoder FLD Decoder out Take the maximum value to get a mask of size 256*256, such as Figure 5 As shown, it is a feature map containing different categories of targets (feet, foot brake, background).

[0047] According to the mask, the position information of the foot and the foot brake can be obtained. The area of ​​the foot and the foot brake can be calculated through the position information, and the IOU between the foot and the foot brake can be calculated based on this. The calculation formula is as follows:

[0048]

[0049] Where Area 脚 Indicates the area of ​​the foot, Area 脚刹 Indicates the area of ​​the foot brake.

[0050] If the calculated IOU exceeds a preset value (for example, 0.1), it can be determined that the foot is on the foot brake, otherwise it is determined that the foot brake is not on.

[0051] At this point, the automatic detection of the foot brake status is completed.

[0052] Furthermore, an electronic device is disclosed in an embodiment of the present invention, which mainly includes a processor and a memory. The processor is mainly used to call a computer program stored in the memory and execute the foot brake state detection method based on the improved real-time semantic segmentation technology provided in the above embodiment.

[0053] Furthermore, an embodiment of the present invention also provides a computer-readable storage medium, which is used to store program codes, and the program codes are used to execute the foot brake state detection method based on the improved real-time semantic segmentation technology provided by the present invention.

[0054] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the electronic device and program code in the computer-readable storage medium described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0055] Finally, it should be noted that although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments and application fields, and the above-mentioned specific embodiments are only illustrative and instructive, rather than restrictive. Under the guidance of this specification, a person of ordinary skill in the art can also make many forms without departing from the scope of protection of the claims of the present invention, all of which belong to the scope of protection of the present invention.

Claims

1. A foot brake state detection method based on improved semantic segmentation technology, characterized in that: include: Step 1: Collect video data including the passenger seat foot and foot brake, and obtain several continuous original frame images by intercepting the video data frame by frame; Step 2: pre-process the original frame image to obtain a feature matrix and a MASK containing foot and foot brake information; Step 3: Input the feature matrix containing foot and foot brake information into the trained STDCNet network model to obtain a feature matrix containing deep features; Step 4: Input the feature matrix and the feature matrix obtained by the last two convolutions in the STDCNet network model into the decoder FLD Decoder for decoding, downsample and classify the fused feature matrix output after decoding, and obtain a fused feature matrix containing key semantic information; Step 5: Process the fused feature matrix to obtain a mask that can visually distinguish different targets, and determine whether the passenger's foot is on the foot brake based on the mask.

2. The foot brake state detection method according to claim 1, characterized in that: In the step 1, video data including the passenger's foot and foot brake are collected under different scene lighting environments.

3. The foot brake state detection method according to claim 1, characterized in that: The pretreatment in step 2 includes: Use labelme image annotation software to annotate the feet and foot brake in the original frame image, and generate an annotation file containing the feet and foot brake; Generate a mask containing the position information of the foot and the foot brake according to the original frame image and the corresponding annotation file; The original frame image and mask are subjected to various random image interferences and unified sizes to obtain a 1*3*256*256 feature matrix and a 256*256 mask.

4. The foot brake state detection method according to claim 1, characterized in that: The STDCNet network model includes a DetailGuidance module for extracting feature information and a context path module for classifying feature information; The step 3 specifically includes: The feature matrix containing the foot and brake information is input into the STDCNet network model. It is first processed by convolution with a kernel of 1, and then by convolution with a kernel of 3 three times. In each convolution process, a feature matrix is ​​extracted through high-level semantics. After four convolutions, four feature matrices with sizes of 1*64*128*128, 1*128*64*64, 1*256*32*32, and 1*512*16*16 are obtained respectively. Finally, the four feature matrices are fused in a cascade manner to obtain a feature matrix containing deep features, wherein the size of the feature matrix containing deep features is 1*1024*8*8; Among them, three feature matrices with sizes of 1*256*32*32, 1*512*16*16, and 1*1024*8*8 are used as inputs of step 4.

5. The foot brake state detection method according to claim 1, characterized in that: The FLD Decoder includes an SPPM module and two cascaded UAFM modules; The step 4 specifically includes: First, the feature matrix of size 1*1024*8*8 output by the STDCNet network model is input into the SPPM module to obtain semantic information of different scales and enhance the perception of objects of different scales, and a feature matrix of size 16*128*8*8 is obtained; Then, the feature matrix of size 1*512*16*16 and the feature matrix of size 1*1024*8*8 output by the SPPM module are input into the first UAFM module, and the fused feature matrix of size 16*96*16*16 is obtained. Then, the feature matrix of size 16*96*16*16 and the feature matrix of size 1*256*32*32 are input into the second UAFM module, and the fused feature matrix of size 16*64*32*32 is obtained. Finally, the feature matrix of size 16*64*32*32 is unified in size, downsampled and classified to obtain a fused feature matrix containing key semantic information. Among them, the fused feature matrix F out The resulting formula is as follows:

6. The foot brake state detection method according to claim 5, characterized in that: The UAFM module includes F high and F low Input, where the SPPM module is defined as F low ; F high After upsampling and F low Input the attention mechanism to get the weight matrix α of the UAFM module; then multiply the two inputs with the weight matrix α and (1-α) respectively, and output the fused feature matrix; Among them, the calculation formula of the weight matrix α is as follows: α=Sigmoid(Conv(F cat )) In the formula, F cat =Concat[Mean(Upsample(F high )),Mean(F low ),Max(Upsample(F high )),Max(F low )]。 7. The foot brake state detection method according to claim 1, characterized in that: The step 5 specifically includes: The fusion feature matrix is ​​processed to obtain a maximum value to obtain a mask that can visually distinguish different targets; Obtaining the position information of the foot and the foot brake according to the mask, and then obtaining the area of ​​the foot and the foot brake; Calculate the IOU between the foot and the foot brake; The IOU is compared with a preset value, and if the IOU exceeds the preset value, it is determined that the passenger's foot is on the foot brake; The calculation formula of the IOU is as follows: Where Area 脚 Indicates the area of ​​the foot, Area 脚刹 Indicates the area of ​​the foot brake.

8. The foot brake state detection method according to claim 7, characterized in that: The preset value is 0.

1.

9. An electronic device, characterized in that: It comprises a processor and a memory; the processor is used to call the computer program stored in the memory to execute the foot brake status detection method based on the improved semantic segmentation technology as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the foot brake status detection method based on the improved semantic segmentation technology as described in any one of claims 1 to 8.

Citation Information

Patent Citations

  • Method for identifying behaviors of safeguard on subject 3 examination vehicle

    CN110956868A

  • System and method for monitoring behaviors of safety officer on driving test vehicle

    CN110971869A

  • Driver foot behavior violation detection method, device and equipment and storage medium

    CN115965946A

  • STDC network-based image real-time semantic segmentation method and device

    CN117292128A