A bus passenger safety monitoring method and system

Abstract

The present invention relates to a bus passenger safety monitoring method and system. The method comprises: capturing a passenger image sequence, inputting the passenger image sequence into an improved YOLOv8n model, and obtaining a seat belt recognition result and a passenger status recognition result; improving the YOLOv8n model comprises: adding a frequency domain enhancement unit and an edge feature extraction unit before a Backbone layer of the YOLOv8n model, for extracting frequency domain features and edge features, thereby superimposing the frequency domain features, edge features and original image features and inputting the features into the Backbone layer, adding an ECA attention unit in the Backbone layer, for performing channel attention weighting on an input feature map; judging whether a passenger is correctly wearing a seat belt based on the seat belt recognition result in combination with a seat belt buckle sensor detection result, and issuing corresponding safety warnings for the judgment result and the passenger status recognition result.

Description

Technical Field

[0001] The present invention relates to the technical field of safety monitoring, and in particular to a bus passenger safety monitoring method and system. Background Art

[0002] With the development of urbanization, public transportation systems have become increasingly robust, and more people are choosing to travel by public transportation. A passenger bus is a commercial vehicle typically carrying nine or more people (including the driver), typically with a square cabin, used to carry passengers and their carry-on luggage. Due to the large number of passengers and the large cabin space, drivers cannot always monitor the safety of each passenger, including seatbelt wear and physical condition.

[0003] The existing technical solution proposes a seat belt detection system and method that monitors the wearing of passengers' seat belts through pressure sensors, and the driver can monitor the system to observe the wearing of passengers' seat belts. However, this invention patent only monitors the wearing of passengers' seat belts. On the one hand, it fails to consider whether the passengers can wear the seat belts correctly. For example, when some passengers wrap the seat belts around their backs and buckle them, the seat belt detection device can only obtain the result of "the seat belt is worn" but not the result of "whether the passenger wears the seat belt correctly". On the other hand, this method does not take into account that during the long-term driving of the bus, the driver needs to pay attention to the physical and mental condition of the passengers during the ride in addition to the wearing of the seat belts. Therefore, there are certain deficiencies in both seat belt detection and passenger safety monitoring. Summary of the Invention

[0004] In order to solve the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a bus passenger safety monitoring method and system, which can help the driver monitor the safety status of passengers during driving.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A passenger safety monitoring method for a bus, comprising:

[0007] Capturing a passenger image sequence, inputting the passenger image sequence into an improved YOLOv8n model to obtain seatbelt recognition results and passenger status recognition results; the improved YOLOv8n model is trained using a passenger seatbelt dataset and a passenger status dataset and their corresponding labels; the improved YOLOv8n model is based on the YOLOv8n model by enhancing frequency domain features, edge feature processing, and incorporating a channel attention mechanism;

[0008] Based on the seat belt recognition result and the seat belt buckle sensor detection result, it is determined whether the passenger is wearing the seat belt correctly, and a corresponding safety warning is issued based on the determination result and the passenger status recognition result.

[0009] Optionally, capturing the passenger image sequence comprises:

[0010] The passenger image sequence is captured at a target frequency; the passenger image sequence is a two-dimensional image matrix sorted by passenger seats:

[0011]

[0012] Among them, image t represents a set of passenger image sequences, t represents the time when the current image sequence was acquired, and f(i, j) represents the passenger image in the i-th row and j-th column.

[0013] Optionally, by enhancing frequency domain features and edge feature processing based on the YOLOv8n model and adding a channel attention mechanism, the following are included:

[0014] A frequency domain enhancement unit and an edge feature extraction unit are added before the Backbone layer of the YOLOv8n model to extract frequency domain features and edge features, thereby superimposing the frequency domain features, the edge features and the original image features and inputting them into the Backbone layer. An ECA attention unit is added to the Backbone layer to perform channel attention weighting on the input feature map.

[0015] Optionally, extracting the frequency domain features includes:

[0016] Convert the time domain of the original image in the passenger image sequence to the frequency domain, and extract the high-frequency information of the frequency domain image:

[0017] F′(u,v)=F(u,v)·H(u,v);

[0018] Where H(u,v) represents the transfer function of the high-pass filter, F'(u,v) represents the frequency domain image after retaining the high-frequency information, · represents the point-by-point multiplication operation, F(u,v) represents the frequency domain image converted from the time domain to the frequency domain, u represents the horizontal coordinate of the frequency domain image, and v represents the vertical coordinate of the frequency domain image.

[0019] The frequency domain image from which high-frequency information is extracted is inversely transformed to convert the frequency domain image into a time domain image, thereby obtaining a time domain image containing high-frequency information, namely, the frequency domain feature:

[0020]

[0021] Among them, f'(x,y) represents the converted time domain image that retains high-frequency information, W represents the width of the image, H represents the height of the image, and F'(u,v) represents the frequency domain image that retains high-frequency information. It is used as input here. represents the complex exponential function, e represents the natural constant, j represents the imaginary unit, Represents the complex phase factor, ux represents the product of the abscissa u of the frequency domain image and the abscissa x of the time domain image, and ux represents the product of the ordinate v of the frequency domain image and the ordinate y of the time domain image.

[0022] Optionally, extracting the edge feature includes:

[0023] The original images in the passenger image sequence are converted into gray images through grayscale processing, and edge detection is performed on the gray images:

[0024]

[0025] Among them, f(x,y) represents the pixel value of the grayscale image, x represents the horizontal time domain component in the time domain coordinate, y represents the vertical time domain component in the time domain coordinate, G x Represents the gradient strength of the image in the x direction, G y Represents the gradient intensity of the image in the y direction, and * represents the convolution operation;

[0026] Calculate the edge gradient strength based on the edge detection result:

[0027]

[0028] Where G refers to the gradient strength;

[0029] Perform a convolution operation on the edge gradient intensity to obtain the edge feature:

[0030] F edge =Conv(G);

[0031] Among them, F edge Represents the edge features of the final output, and Conv represents the convolution operation.

[0032] Optionally, superimposing the frequency domain features, the edge features, and the original image features includes:

[0033] F concat =Concat(F orig +F fft +F edge );

[0034] Among them, F orig represents the original image features, F fft Represents the frequency domain characteristics, F edge represents edge features, F concatRepresents the features obtained after the superposition operation. Concat represents the operation of superimposing the frequency domain features, edge features and original image features. Here, an operation function named Concat is used to represent it. orig The number of channels representing the original image features, C fft The number of channels representing frequency domain features, C edge The number of channels representing edge features.

[0035] Optionally, performing channel attention weighting on the input feature map includes:

[0036] Calculate the channel attention and obtain the global features of the channel:

[0037]

[0038] Among them, X c Represents the cth channel of the input feature map X, H represents the height of the input feature map, W represents the width of the input feature map, X c (i, j) represents the pixel value with coordinates (i, j) in the input feature map, Z c represents the global average pooling value of the c-th channel, that is, the global feature of the channel;

[0039] Based on the global features, generate the attention weight of each channel:

[0040] W c =Conv(Z c ,k);

[0041] Among them, Z c represents the feature of the cth channel after global average pooling, k represents the size of the one-dimensional convolution kernel used for convolution operation, W c Represents the attention weight of the c-th channel generated after one-dimensional convolution;

[0042] Using the attention weight of each channel, the input feature map is weighted as follows:

[0043] Y c =X c W c ;

[0044] Among them, Y c Represents the feature map after channel attention weighting.

[0045] Optionally, determining whether the passenger is wearing a seat belt correctly includes:

[0046] If the seat belt recognition result is that the passenger is wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is not wearing the seat belt correctly, it is necessary to check for abnormalities in the seat belt; if the seat belt recognition result is that the passenger is wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is wearing the seat belt correctly, it is proved that the passenger is wearing the seat belt correctly; if the seat belt recognition result is that the passenger is not wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is wearing the seat belt correctly, it is necessary to check for abnormalities in the seat belt; if the seat belt recognition result is that the passenger is not wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is not wearing the seat belt correctly, it is proved that the passenger is not wearing the seat belt.

[0047] Optionally, the passenger status recognition result includes: sleeping, relaxing, neutral, tired, motion sickness, and vomiting.

[0048] To achieve the above object, the present invention further provides a bus passenger safety monitoring system, comprising:

[0049] an image acquisition module for capturing a sequence of passenger images;

[0050] A safety status recognition module is configured to input the passenger image sequence into an improved YOLOv8n model to obtain seatbelt recognition results and passenger status recognition results; the improved YOLOv8n model is trained using a passenger seatbelt dataset and a passenger status dataset and their corresponding labels; the improved YOLOv8n model is based on the YOLOv8n model by enhancing frequency domain features, edge feature processing, and adding a channel attention mechanism;

[0051] A seat belt detection module is used to determine whether the passenger is wearing the seat belt correctly based on the seat belt identification result and the seat belt buckle sensor detection result;

[0052] The safety warning module is used to issue corresponding safety warnings based on the determination results and the passenger status recognition results.

[0053] The beneficial effects of the present invention are:

[0054] The present invention improves YOLOv8n to enhance its robustness and accuracy for blurred images and dim environments, and uses passenger seat belt datasets and passenger status datasets for training to obtain accurate recognition results.

[0055] The present invention combines the seat belt buckle sensor and the algorithm recognition results to make a more comprehensive judgment on the passenger's seat belt wearing situation, which can better assist the driver in protecting the passenger's safety.

[0056] The present invention helps the driver to better check the safety status of all passengers by sorting the safety status of passenger seats through an electronic display screen, reduces the driver's distraction during driving, and ensures the safety of passengers during riding. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 This is a flow chart of a bus passenger safety monitoring method according to an embodiment of the present invention;

[0059] Figure 2 This is a schematic diagram of the improved structure of YOLOv8n according to an embodiment of the present invention;

[0060] Figure 3 The figure is a schematic diagram of a bus passenger safety monitoring system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0062] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] like Figure 1As shown, this embodiment discloses a bus passenger safety monitoring method, including: capturing a passenger image sequence, inputting the passenger image sequence into an improved YOLOv8n model, and obtaining a seat belt recognition result and a passenger status recognition result; the improved YOLOv8n model is trained using a passenger seat belt dataset and a passenger status dataset and their corresponding labels; the improved YOLOv8n model includes: adding a frequency domain enhancement unit and an edge feature extraction unit before the Backbone layer of the YOLOv8n model to extract frequency domain features and edge features, thereby superimposing the frequency domain features, the edge features and the original image features and inputting them into the Backbone layer, adding an ECA attention unit to the Backbone layer to perform channel attention weighting on the input feature map; judging whether the passenger is wearing the seat belt correctly based on the seat belt recognition result combined with the detection result of the seat belt buckle sensor, and issuing corresponding safety warnings for the judgment result and the passenger status recognition result.

[0064] Furthermore, capturing the passenger image sequence includes:

[0065] The passenger image sequence is captured at a target frequency; the passenger image sequence is a two-dimensional image matrix sorted by passenger seats:

[0066]

[0067] Among them, image t represents a set of passenger image sequences, t represents the time when the current image sequence was acquired, and f(i, j) represents the passenger image in the i-th row and j-th column.

[0068] Specifically:

[0069] In this embodiment, passenger images are collected by a camera module. The camera module is a monocular camera, which is placed in a front position above the seat, such as directly above the front seat, and the camera lens is tilted toward the passenger position so that the camera can collect passenger images that meet the recognition requirements. The passenger image should include the passenger's head and torso, that is, the image of the area above the passenger's thighs.

[0070] When the system starts running, the camera module is started to capture passenger image sequences at a certain frequency and output the image sequences to the safety status recognition module.

[0071] The passenger image sequence refers to a two-dimensional image matrix arranged according to the passenger seats. For example, if the seats in a bus are arranged in 3 rows and 4 columns, the passenger image sequence collected by the camera module each time should be:

[0072]

[0073] Among them, imaget It refers to a set of passenger image sequences, t refers to the time when the current image sequence is collected, and the content in the image sequence is f(i, j), which represents the passenger image in the i-th row and j-th column.

[0074] The neural network model used in this embodiment is the improved YOLOv8n algorithm, which is one of the YOLOv8 series algorithms. Compared with other versions of the algorithm in this series, the YOLOv8n algorithm has the smallest number of parameters and model size, and has low hardware requirements, making it suitable for deployment on terminal hardware. Although YOLOv8n can ensure high speed and small number of parameters, its accuracy is lower than that of other versions of the algorithm in this series. At the same time, in the environment of a bus, passengers may shake, which may cause the passenger images collected by the camera module to be unclear. At the same time, when a bus is driving at night or passing through a tunnel, the image brightness is prone to be insufficient, resulting in low detection accuracy. To address the above problems, in this embodiment, the YOLOv8n algorithm is improved to improve the generalization ability of the algorithm and improve the recognition ability in darker environments and in blurred images.

[0075] The frequency domain enhancement unit (FFT Enhancement) and edge feature extraction unit are added before the Backbone layer of YOLOv8n. The frequency domain features, edge features and original image features are superimposed and then passed to the Backbone layer. The ECA attention unit is added to each convolutional layer in the Backbone layer to improve the accuracy of target recognition. The improved YOLOv8n structure is as follows: Figure 2 As stated.

[0076] Furthermore, extracting the frequency domain features includes:

[0077] Convert the time domain of the original image in the passenger image sequence to the frequency domain, and extract the high-frequency information of the frequency domain image:

[0078] F′(u,v)=F(u,v)·H(u,v);

[0079] Where H(u,v) represents the transfer function of the high-pass filter, F'(u,v) represents the frequency domain image after retaining the high-frequency information, · represents the point-by-point multiplication operation, F(u,v) represents the frequency domain image converted from the time domain to the frequency domain, u represents the horizontal coordinate of the frequency domain image, and v represents the vertical coordinate of the frequency domain image.

[0080] The frequency domain image from which high-frequency information is extracted is inversely transformed to convert the frequency domain image into a time domain image, thereby obtaining a time domain image containing high-frequency information, namely, the frequency domain feature:

[0081]

[0082] Among them, f'(x,y) represents the converted time domain image that retains high-frequency information, W represents the width of the image, H represents the height of the image, and F'(u,v) represents the frequency domain image that retains high-frequency information. It is used as input here. represents the complex exponential function, e represents the natural constant, j represents the imaginary unit, Represents the complex phase factor, ux represents the product of the abscissa u of the frequency domain image and the abscissa x of the time domain image, and ux represents the product of the ordinate v of the frequency domain image and the ordinate y of the time domain image.

[0083] Furthermore, extracting the edge features includes:

[0084] The original images in the passenger image sequence are converted into gray images through grayscale processing, and edge detection is performed on the gray images:

[0085]

[0086] Among them, f(x,y) represents the pixel value of the grayscale image, x represents the horizontal time domain component in the time domain coordinate, y represents the vertical time domain component in the time domain coordinate, G x Represents the gradient strength of the image in the x direction, G y Represents the gradient intensity of the image in the y direction, and * represents the convolution operation;

[0087] Calculate the edge gradient strength based on the edge detection result:

[0088]

[0089] Where G refers to the gradient strength;

[0090] Perform a convolution operation on the edge gradient intensity to obtain the edge feature:

[0091] F edge =Conv(G);

[0092] Among them, F edge Represents the edge features of the final output, and Conv represents the convolution operation.

[0093] Furthermore, superimposing the frequency domain features, the edge features, and the original image features includes:

[0094] F concat =Concat(F orig +F fft +F edge );

[0095] Among them, F orig represents the original image features, Ffft Represents the frequency domain characteristics, F edge represents edge features, F concat Represents the features obtained after the superposition operation. Concat represents the operation of superimposing the frequency domain features, edge features and original image features. Here, an operation function named Concat is used to represent it. orig The number of channels representing the original image features, C fft The number of channels representing frequency domain features, C edge The number of channels representing edge features.

[0096] Furthermore, performing channel attention weighting on the input feature map includes:

[0097] Calculate the channel attention and obtain the global features of the channel:

[0098]

[0099] Among them, X c Represents the cth channel of the input feature map X, H represents the height of the input feature map, W represents the width of the input feature map, X c (i, j) represents the pixel value with coordinates (i, j) in the input feature map, Z c represents the global average pooling value of the c-th channel, that is, the global feature of the channel;

[0100] Based on the global features, generate the attention weight of each channel:

[0101] W c =Conv(Z c ,k);

[0102] Among them, Z c represents the feature of the cth channel after global average pooling, k represents the size of the one-dimensional convolution kernel used for convolution operation, W c Represents the attention weight of the c-th channel generated after one-dimensional convolution;

[0103] Using the attention weight of each channel, the input feature map is weighted as follows:

[0104] Y c =X c W c ;

[0105] Among them, Y c Represents the feature map after channel attention weighting.

[0106] Specifically:

[0107] In this embodiment, a frequency domain enhancement unit (FFT EnhancementModule) is added before the Backbone layer of YOLOv8n. The original image is converted from the time domain to the frequency domain through Fourier transform, and the frequency domain image is operated to retain the high-frequency information. Finally, an inverse transform is performed to convert the frequency domain image into a time domain image. This module is divided into three steps:

[0108] (1) Fourier transform:

[0109] Through Fourier transform, the time domain image is converted into a frequency domain image. The conversion process is as follows:

[0110]

[0111] Where f(x,y) refers to the pixel value of the time domain image, x is the horizontal time domain component in the time domain coordinate, y is the vertical time domain component in the time domain coordinate, F(u,v) refers to the pixel value of the frequency domain image, u is the horizontal frequency component in the frequency domain coordinate, v is the vertical frequency component in the frequency domain coordinate, W is the width of the input time domain image, H is the height of the input time domain image, j is the imaginary unit, e is the natural constant, and π is the mathematical constant, also known as pi.

[0112] (2) Retain high-frequency information:

[0113] In the process of retaining high-frequency information, a high-pass filter is used, which is defined as follows:

[0114]

[0115] Among them, H(u,v) refers to the transfer function of the high-pass filter, and D0 refers to the cutoff frequency of the filter, which is used to control the range of high-frequency information retained. Refers to the distance of the current frequency component. The farther away from the center, the higher the frequency.

[0116] (3) The process of retaining high-frequency information is as follows:

[0117] F′(u,v)=F(u,v)·H(u,v);

[0118] Here, F'(u,v) refers to the frequency domain image after retaining high-frequency information, and · refers to the point-by-point multiplication operation.

[0119] Inverse Fourier transform:

[0120] After the high-frequency information is retained, the frequency domain image needs to be converted into a time domain image through inverse Fourier transform. The process is as follows:

[0121]

[0122] Among them, f'(x,y) refers to the converted time domain image that retains high-frequency information.

[0123] An edge feature extraction unit is added before the Backbone layer of YOLOv8n. The edge features of the image are extracted through operators to improve the network's ability to perceive edge information. This module is divided into four steps:

[0124] (1) Grayscale processing:

[0125] In this embodiment, the color image needs to be converted into a gray image through grayscale processing. The grayscale processing uses a standard grayscale formula, which is as follows:

[0126] Y=0.299·R+0.587·G+0.114·B;

[0127] Where Y refers to the brightness of the grayscaled pixel, R refers to the red channel value in the color image, G refers to the green channel value in the color image, and B refers to the blue channel value in the color image. 0.299, 0.587, and 0.114 are the weights corresponding to the red, green, and blue channels, respectively. · refers to the multiplication operation.

[0128] (2) Calculate the gradient strength using the operator:

[0129] In this embodiment, the Sobel operator is used for edge detection, and its formula is as follows:

[0130]

[0131] Among them, f(x,y) refers to the pixel value of the grayscale image, x is the horizontal time domain component in the time domain coordinate, y is the vertical time domain component in the time domain coordinate, G x Refers to the gradient strength of the image in the x direction, G y It refers to the gradient strength of the image in the y direction, and * refers to the convolution operation.

[0132] (3) Then calculate the edge gradient strength:

[0133]

[0134] Here, G refers to the gradient strength.

[0135] (4) Edge feature convolution:

[0136] The convolution operation is performed on the extracted gradient strength. The process is as follows:

[0137] F edge =Conv(G);

[0138] Among them, F edgeRefers to the edge features of the final output, and Conv refers to the convolution operation, which is used to enhance the edge features.

[0139] Feature stitching method:

[0140] In this patent, the frequency domain features and edge features obtained by the frequency domain enhancement unit and the edge feature extraction unit need to be spliced ​​with the original image, and then the spliced ​​image is input into the subsequent Backbone layer for convolution operation. The splicing method is as follows:

[0141] The frequency domain features, edge features, and original image features are superimposed on the channel. After superposition, the height and width of the feature map do not change, but the number of channels increases. The number of channels is equal to the sum of the number of channels of the frequency domain features, edge features, and original image features:

[0142] F concat =Concat(F orig +F fft +F edge );

[0143] Among them, F orig is the original image feature, the size is (C orig ,H,W),F fft is the frequency domain feature, and the size is (C fft ,H,W),F edge is the edge feature, the size is (C edge ,H,W),F concat It is the feature obtained after the superposition operation, and its size is (C concat ,H,W), where C concat =C orig +C fft +C edge , C orig The number of channels representing the original image features, C fft The number of channels representing frequency domain features, C edge Represents the number of channels of edge features, H is the height of the feature map, and W is the width of the feature map.

[0144] ECA attention unit:

[0145] In this embodiment, an ECA attention unit is added to each convolutional layer of the Backbone layer to improve the YOLOv8n model's ability to perceive the target object. The working order and formula are as follows:

[0146] (1) Channel attention calculation:

[0147]

[0148] Among them, X cis the cth channel of the input feature map X, H is the height of the input feature map, W is the width of the input feature map, X c (i, j) is the pixel value with coordinates (i, j) in the input feature map, Z c It refers to the global average pooling value of the c-th channel, which represents the global feature of the channel.

[0149] (2) Local cross weight:

[0150] W c =Conv(Z c ,k);

[0151] Among them, Z c refers to the feature of the cth channel after global average pooling, k refers to the size of the one-dimensional convolution kernel used for convolution operation, which is used to control the channel cross-range, W c It refers to the attention weight of the c-th channel generated after one-dimensional convolution.

[0152] (3) Weighting the input feature map:

[0153] Y c =X c W c ;

[0154] Among them, Y c It refers to the feature map after channel attention weighting.

[0155] Furthermore, determining whether the passenger is wearing the seat belt correctly includes:

[0156] If the seat belt recognition result is that the passenger is wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is not wearing the seat belt correctly, it is necessary to check for abnormalities in the seat belt; if the seat belt recognition result is that the passenger is wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is wearing the seat belt correctly, it is proved that the passenger is wearing the seat belt correctly; if the seat belt recognition result is that the passenger is not wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is wearing the seat belt correctly, it is necessary to check for abnormalities in the seat belt; if the seat belt recognition result is that the passenger is not wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is not wearing the seat belt correctly, it is proved that the passenger is not wearing the seat belt.

[0157] Furthermore, the passenger status recognition result includes: sleeping, relaxing, neutral, tired, motion sickness, and vomiting.

[0158] Specifically:

[0159] The YOLOv8n algorithm in this embodiment is jointly trained using a passenger seatbelt dataset and a passenger status dataset, wherein the passenger seatbelt dataset refers to either a public dataset or a self-built dataset, and the dataset content is a real-life picture of a passenger or driver in a passenger car, and the corresponding label of the dataset is whether the passenger or driver is wearing a seatbelt correctly. When the passenger or driver in the picture is wearing a seatbelt correctly, the corresponding label is true, otherwise it is false; wherein the passenger status dataset refers to either a public dataset or a self-built dataset, and the dataset content is a facial picture of passengers while riding in a vehicle, reflecting the expression of the passenger's physical state and emotions in facial expressions while riding in a vehicle, corresponding to multiple states, including sleep, relaxation, neutral, tiredness, motion sickness, and vomiting.

[0160] Furthermore, issuing corresponding safety warnings for the determination result and the passenger status identification result includes:

[0161] Security warnings for the judgment results include:

[0162] If the passenger is wearing the seat belt correctly, no warning is required. If the seat belt needs to be checked for abnormalities, the electronic display will issue a warning, and the voice broadcast will issue a warning. If the passenger is not wearing the seat belt correctly, both the electronic display and the voice broadcast will issue a warning.

[0163] Specifically:

[0164] The seat belt detection module in this embodiment detects whether the seat belt buckle is used through a seat belt buckle sensor and outputs a detection result. The seat belt sensor is installed in the seat belt socket. If the seat belt buckle sensor detects that the seat belt is used, the detection result is true, otherwise it is false. Combined with the seat belt recognition result received from the safety status recognition module, the values ​​of the two parameters are combined to determine whether the passenger is wearing the seat belt correctly, and the judgment result is output to the safety warning module. The comprehensive judgment method of this module is shown in Table 1 below:

[0165] Table 1

[0166]

[0167] Providing a safety warning for the passenger status recognition result includes:

[0168] If the passenger status recognition result is sleep and the sleep duration is within the target duration, no warning is required; if the passenger status recognition result is sleep and the sleep duration is outside the target duration, the electronic display end will issue a warning, the voice broadcast end will issue a warning, and there will be no warning; if the passenger status recognition result is relaxed or neutral, no warning is required; if the passenger status recognition result is tired, the electronic display end will issue a warning, the voice broadcast end will issue a warning, and there will be no warning; if the passenger status recognition result is motion sickness or vomiting, both the electronic display end and the voice broadcast end will issue a warning.

[0169] Specifically:

[0170] The safety warning module in this embodiment receives the passenger status identification results from the safety status identification module and the passenger seatbelt wearing status results from the seatbelt detection module. It displays the safety status of the bus passengers according to the passenger seat distribution on an interactive electronic display screen installed at the driver's seat. Each passenger is associated with two indicators: the seatbelt wearing determination result and the passenger status. The indicator status on the display screen changes based on the passenger indicators. When a warning is needed, the module uses a voice broadcast device to alert the driver. The signals received from the seatbelt detection module are used to provide different levels of safety warnings to the driver depending on the situation. The specific determination method is shown in Table 2:

[0171] Table 2

[0172] Seat belt detection module results Warning level Electronic display warning Voice broadcast warning Passengers wearing seat belts correctly Safety No warning No warning Need to check Early Warning Early Warning No warning Passengers not wearing seat belts Danger Early Warning Early Warning

[0173] Based on the signals transmitted by the safety status recognition module, different levels of safety warnings are issued to the driver according to different situations. The specific judgment method is as follows. For the sleep state, different judgment conditions of different durations are set. Short duration means that a passenger is monitored to be in a sleep state for a period of 5 to 30 minutes. Long duration means that a passenger is monitored to be in a sleep state for a period of more than 30 minutes. When the duration exceeds 30 minutes, the warning level is changed and the corresponding warning is issued, as shown in Table 3.

[0174] Table 3

[0175]

[0176]

[0177] The electronic display screen should have interactive functions. When the driver finds the warning and completes the passenger safety check, the passenger's safety status can be changed through the interactive function so that the safety warning module no longer issues an warning, reducing invalid warnings and interference to the driver during driving.

[0178] like Figure 3As shown, this embodiment also provides a bus passenger safety monitoring system, including: an image acquisition module for capturing a passenger image sequence; a safety status recognition module for inputting the passenger image sequence into an improved YOLOv8n model to obtain a seat belt recognition result and a passenger status recognition result; the improved YOLOv8n model is trained using a passenger seat belt dataset and a passenger status dataset and their corresponding labels; the improved YOLOv8n model includes: adding a frequency domain enhancement unit and an edge feature extraction unit before the Backbone layer of the YOLOv8n model to extract frequency domain features and edge features, thereby superimposing the frequency domain features, the edge features and the original image features and inputting them into the Backbone layer, adding an ECA attention unit to the Backbone layer to perform channel attention weighting on the input feature map; a seat belt detection module for determining whether the passenger is wearing the seat belt correctly based on the seat belt recognition result combined with the detection result of the seat belt buckle sensor; a safety warning module for issuing corresponding safety warnings for the determination result and the passenger status recognition result.

[0179] Specifically, the camera module collects images of the area where each passenger is located during the driving of the bus in real time, and forms an image sequence according to the seat arrangement, and outputs the image sequence to the safety status recognition module. The safety status recognition module calls the pre-trained YOLOv8n neural network model to identify the safety status of the bus passengers, including: identifying the passenger's seat belt wearing status and identifying the passenger's status through the passenger's facial image, and outputs the recognition results to the seat belt detection module and the safety warning module respectively. The seat belt detection module uses the recognition result of the passenger's seat belt wearing status and the seat belt buckle sensor to jointly determine the passenger's seat belt wearing status, and outputs the passenger's seat belt wearing status to the safety warning module. When the safety warning module receives information that the passenger fails to wear the seat belt correctly or feels physically unwell, the module will remind the driver in real time through the interactive display screen or voice broadcast system to facilitate the driver to handle it in time. After processing, the driver can change the passenger's safety status and cancel the warning on the interactive interface of the safety warning module. As a driving assistance system, this system can not only ensure the safety of bus passengers during the ride, but also reduce the driver's energy consumption and improve work efficiency.

[0180] The present embodiment discloses a passenger safety monitoring method for a bus. When the bus starts, the bus passenger status monitoring system starts working, the camera module starts to collect passenger images and outputs the image sequence to the safety status recognition module, and the seat belt detection module starts to continuously detect the seat belt buckle sensor signal; after the safety status recognition module receives the image sequence, it starts to call the YOLOv8n algorithm to perform seat belt recognition and passenger status recognition, wherein the seat belt recognition result is output to the seat belt detection module, and the passenger status recognition result is output to the safety warning module; after receiving the seat belt recognition result, the seat belt detection module performs a comprehensive judgment in combination with the seat belt buckle sensor detection result, and outputs the judgment result to the safety warning module; the safety warning module receives the passenger After the status recognition result is obtained, the status indicator of the corresponding seat passenger is adjusted according to the recognition result. After receiving the judgment result of the seat belt detection module, the seat belt wearing judgment result indicator of the corresponding seat passenger is adjusted according to the judgment result. Then, according to the result, the driver is given an early warning reminder in the form of an electronic display or voice broadcast. After receiving the early warning signal, the driver uses the microphone and audio system inside the bus to give safety reminders to passengers in warning or dangerous states. If motion sickness or vomiting occurs, appropriate drugs can be provided to help relieve the symptoms. After ensuring that the passenger is in a safe state, the driver can use the interactive function of the electronic display to click on the screen to change the passenger's safety status, cancel the safety warning for the passenger, and complete a passenger status monitoring.

[0181] This embodiment proposes a passenger safety monitoring method and system for buses. Using a specifically improved visual recognition algorithm combined with physical sensors, the system monitors two key safety indicators for passengers. These indicators are then centrally displayed on the central control electronic display, making it easier for the driver to view and check passenger safety status. As a driver assistance system, this embodiment helps drivers monitor passenger safety status promptly, ensuring passenger safety during driving. It also reduces the driver's focus on passenger safety, improving work efficiency while ensuring safety.

[0182] In this embodiment, passenger image recognition technology is essential. The safety status recognition module obtains the passenger seatbelt detection and status recognition results, which are transmitted as input to the next module. Combined with the seatbelt buckle sensor, the passenger's seatbelt wearing status is determined. Therefore, image recognition technology is essential in this embodiment. This technology can be expanded to include other reference factors, such as weather and road traffic conditions, to derive control input parameters that are more appropriate for the current driving scenario.

[0183] In addition to being applicable to passenger buses, this embodiment can also be applied to scenarios where the status of people needs to be monitored, and more application scenarios can be developed, such as monitoring the status of children while riding in a vehicle, which helps to ensure the safety and status of children while riding in a vehicle.

[0184] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A bus passenger safety monitoring method, characterized in that: include: Capturing a passenger image sequence, inputting the passenger image sequence into the improved YOLOv8n model, and obtaining a seat belt recognition result and a passenger status recognition result; The improved YOLOv8n model is trained using the passenger seatbelt dataset and the passenger status dataset and their corresponding labels. The improved YOLOv8n model is based on the YOLOv8n model by enhancing frequency domain features, edge feature processing, and adding a channel attention mechanism: A frequency domain enhancement unit and an edge feature extraction unit are added before the Backbone layer of the YOLOv8n model to extract frequency domain features and edge features, thereby superimposing the frequency domain features, the edge features and the original image features and inputting them into the Backbone layer. An ECA attention unit is added to the Backbone layer to perform channel attention weighting on the input feature map; Based on the seat belt recognition result and the seat belt buckle sensor detection result, it is determined whether the passenger is wearing the seat belt correctly, and a corresponding safety warning is issued based on the determination result and the passenger status recognition result.

2. The bus passenger safety monitoring method according to claim 1, characterized in that: Capturing the passenger image sequence comprises: The passenger image sequence is captured at a target frequency; the passenger image sequence is a two-dimensional image matrix sorted by passenger seats: Among them, image t represents a set of passenger image sequences, t represents the time when the current image sequence was acquired, and f(i, j) represents the passenger image in the i-th row and j-th column.

3. The bus passenger safety monitoring method according to claim 1, characterized in that: Extracting the frequency domain features includes: Convert the time domain of the original image in the passenger image sequence to the frequency domain, and extract the high-frequency information of the frequency domain image: F'(u,v)=F(u,v)·H(u,v); Where H(u,v) represents the transfer function of the high-pass filter, F'(u,v) represents the frequency domain image after retaining the high-frequency information, · represents the point-by-point multiplication operation, F(u,v) represents the frequency domain image converted from the time domain to the frequency domain, u represents the horizontal coordinate of the frequency domain image, and v represents the vertical coordinate of the frequency domain image. The frequency domain image from which high-frequency information is extracted is inversely transformed to convert the frequency domain image into a time domain image, thereby obtaining a time domain image containing high-frequency information, namely, the frequency domain feature: Among them, f'(x,y) represents the converted time domain image that retains high-frequency information, W represents the width of the image, H represents the height of the image, and F'(u,v) represents the frequency domain image that retains high-frequency information. It is used as input here. represents the complex exponential function, e represents the natural constant, j represents the imaginary unit, π represents the circumference of a circle, Represents the complex phase factor, ux represents the product of the abscissa u of the frequency domain image and the abscissa x of the time domain image, and vy represents the product of the ordinate v of the frequency domain image and the ordinate y of the time domain image.

4. The bus passenger safety monitoring method according to claim 1, characterized in that: Extracting the edge features includes: The original images in the passenger image sequence are converted into gray images through grayscale processing, and edge detection is performed on the gray images: Among them, f(x,y) represents the pixel value of the grayscale image, x represents the horizontal time domain component in the time domain coordinate, y represents the vertical time domain component in the time domain coordinate, G x Represents the gradient strength of the image in the x direction, G y Represents the gradient intensity of the image in the y direction, and * represents the convolution operation; Calculate the edge gradient strength based on the edge detection result: Where G refers to the gradient strength; Perform a convolution operation on the edge gradient intensity to obtain the edge feature: F edge =Conv(G); Among them, F edge Represents the edge features of the final output, and Conv represents the convolution operation.

5. The bus passenger safety monitoring method according to claim 1, characterized in that: Superimposing the frequency domain features, the edge features, and the original image features includes: F concat =Concat(F orig +F fft +F edge ); Among them, F orig represents the original image features, F fft Represents the frequency domain characteristics, F edge represents edge features, F concat It represents the features obtained after the superposition operation, and Concat represents the operation of superimposing the frequency domain features, edge features and original image features.

6. The bus passenger safety monitoring method according to claim 1, characterized in that: Performing channel attention weighting on the input feature map includes: Calculate the channel attention and obtain the global features of the channel: Among them, X c Represents the cth channel of the input feature map X, H represents the height of the input feature map, W represents the width of the input feature map, X c (i, j) represents the pixel value with coordinates (i, j) in the input feature map, Z c represents the global average pooling value of the c-th channel, that is, the global feature of the channel; Based on the global features, generate the attention weight of each channel: W c =Conv(Z c ,k); Among them, Z c represents the feature of the cth channel after global average pooling, k represents the size of the one-dimensional convolution kernel used for convolution operation, W c Represents the attention weight of the c-th channel generated after one-dimensional convolution; Using the attention weight of each channel, the input feature map is weighted as follows: Y c =X c ·W c ; Among them, Y c Represents the feature map after channel attention weighting.

7. The bus passenger safety monitoring method according to claim 1, characterized in that: Determining whether a passenger is wearing a seat belt correctly includes: If the seat belt recognition result is that the passenger is wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is not wearing the seat belt correctly, it is necessary to check for abnormalities in the seat belt; if the seat belt recognition result is that the passenger is wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is wearing the seat belt correctly, it is proved that the passenger is wearing the seat belt correctly; if the seat belt recognition result is that the passenger is not wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is wearing the seat belt correctly, it is necessary to check for abnormalities in the seat belt; if the seat belt recognition result is that the passenger is not wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is not wearing the seat belt correctly, it is proved that the passenger is not wearing the seat belt.

8. The bus passenger safety monitoring method according to claim 1, characterized in that: The passenger status recognition results include: sleeping, relaxing, neutral, tired, motion sickness, and vomiting.

9. A passenger safety monitoring system for a bus, characterized in that: include: an image acquisition module for capturing a sequence of passenger images; The safety status recognition module is used to input the passenger image sequence into the improved YOLOv8n model to obtain seat belt recognition results and passenger status recognition results; the improved YOLOv8n model is trained using the passenger seat belt dataset and the passenger status dataset and their corresponding labels; the improved YOLOv8n model is based on the YOLOv8n model by enhancing frequency domain features, edge feature processing, and adding a channel attention mechanism: A frequency domain enhancement unit and an edge feature extraction unit are added before the Backbone layer of the YOLOv8n model to extract frequency domain features and edge features, thereby superimposing the frequency domain features, the edge features and the original image features and inputting them into the Backbone layer. An ECA attention unit is added to the Backbone layer to perform channel attention weighting on the input feature map; A seat belt detection module is used to determine whether the passenger is wearing the seat belt correctly based on the seat belt identification result and the seat belt buckle sensor detection result; The safety warning module is used to issue corresponding safety warnings based on the determination results and the passenger status recognition results.

Images