Passenger car passenger safety monitoring method and system

Abstract

The invention relates to a passenger car passenger safety monitoring method and system, and the method comprises the steps: capturing a passenger image sequence, inputting the passenger image sequence into an improved YOLOv8n model, and obtaining a safety belt recognition result and a passenger state recognition result; the step of improving the YOLOv8n model comprises the steps of adding a frequency domain enhancement unit and an edge feature extraction unit in front of a Backbone layer of the YOLOv8n model for extracting frequency domain features and edge features so as to superpose the frequency domain features, the edge features and original image features, inputting the superposed features into the Backbone layer, and adding an ECA attention unit into the Backbone layer for performing channel attention weighting on an input feature map; whether the passenger correctly wears the safety belt or not is judged according to the safety belt recognition result and the detection result of the safety belt buckle sensor, and corresponding safety early warning is conducted according to the judgment result and the passenger state recognition result.

Description

Technical Field

[0001] The present invention relates to the technical field of safety monitoring, and in particular to a bus passenger safety monitoring method and system. Background Art

[0002] With the development of urbanization, the public transportation system has become more and more sound, and more people have begun to choose to travel by public transportation. A bus usually refers to a commercial vehicle that can carry more than 9 people (including the driver's seat) and generally has a square compartment for carrying passengers and their carry-on luggage. Due to the large number of passengers and the large compartment space, during driving, the driver cannot always pay attention to the safety status of each passenger, including the passenger's seat belt wearing and physical condition.

[0003] The existing technical solution proposes a seat belt detection system and method to monitor the wearing of passengers' seat belts through pressure sensors, and the driver can monitor the system to observe the wearing of passengers' seat belts. However, this invention patent only monitors the wearing of passengers' seat belts. On the one hand, it fails to consider whether the passengers can wear the seat belts correctly. For example, when some passengers wrap the seat belts around their backs and buckle them, the seat belt detection device can only get the result of "the seat belts are worn" but not "whether the passengers wear the seat belts correctly". On the other hand, this method does not take into account that during the long-term driving of the bus, the driver needs to pay attention to the physical and mental conditions of the passengers during the ride in addition to the wearing of the seat belts. Therefore, there are certain deficiencies in seat belt detection and passenger safety monitoring. Summary of the invention

[0004] In order to solve the problems existing in the above-mentioned prior art, an object of the present invention is to provide a bus passenger safety monitoring method and system, which can help the driver monitor the safety status of passengers during driving.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A passenger safety monitoring method for a bus, comprising:

[0007] Capturing a passenger image sequence, inputting the passenger image sequence into an improved YOLOv8n model, and obtaining a seat belt recognition result and a passenger status recognition result; the improved YOLOv8n model is trained using a passenger seat belt dataset and a passenger status dataset and their corresponding labels; the improved YOLOv8n model is improved by enhancing frequency domain features and edge feature processing on the basis of the YOLOv8n model, and adding a channel attention mechanism;

[0008] According to the seat belt recognition result combined with the seat belt buckle sensor detection result, it is determined whether the passenger is wearing the seat belt correctly, and a corresponding safety warning is issued for the determination result and the passenger status recognition result.

[0009] Optionally, capturing the passenger image sequence comprises:

[0010] The passenger image sequence is captured at a target frequency; the passenger image sequence is a two-dimensional image matrix sorted according to passenger seats:

[0011]

[0012] Among them, image t represents a set of passenger image sequences, t represents the time when the current image sequence is collected, and f(i,j) represents the passenger image in the i-th row and j-th column.

[0013] Optionally, by enhancing the frequency domain features and edge feature processing based on the YOLOv8n model and adding a channel attention mechanism, the following are included:

[0014] A frequency domain enhancement unit and an edge feature extraction unit are added before the Backbone layer of the YOLOv8n model to extract frequency domain features and edge features, so as to superimpose the frequency domain features, the edge features and the original image features and input them into the Backbone layer; an ECA attention unit is added to the Backbone layer to perform channel attention weighting on the input feature map.

[0015] Optionally, extracting the frequency domain feature includes:

[0016] The time domain of the original image in the passenger image sequence is converted to the frequency domain, and the high-frequency information of the frequency domain image is extracted:

[0017] F′(u,v)=F(u,v)·H(u,v);

[0018] Where H(u,v) represents the transfer function of the high-pass filter, F'(u,v) represents the frequency domain image after retaining the high-frequency information, · represents the point-by-point multiplication operation, F(u,v) represents the frequency domain image converted from the time domain to the frequency domain, u represents the horizontal coordinate of the frequency domain image, and v represents the vertical coordinate of the frequency domain image;

[0019] The frequency domain image from which the high-frequency information is extracted is inversely transformed to convert the frequency domain image into a time domain image, thereby obtaining a time domain image containing the high-frequency information, namely, the frequency domain feature:

[0020]

[0021] Among them, f'(x,y) represents the converted time domain image with high-frequency information, W represents the width of the image, H represents the height of the image, and F'(u,v) represents the frequency domain image with high-frequency information retained. Here, it is used as input. represents the complex exponential function, e represents the natural constant, j represents the imaginary unit, represents the complex phase factor, ux represents the product of the abscissa u of the frequency domain image and the abscissa x of the time domain image, and ux represents the product of the ordinate v of the frequency domain image and the ordinate y of the time domain image.

[0022] Optionally, extracting the edge feature includes:

[0023] The original images in the passenger image sequence are converted into gray images by grayscale processing, and edge detection is performed on the gray images:

[0024]

[0025] Among them, f(x,y) represents the pixel value of the grayscale image, x represents the horizontal time domain component in the time domain coordinate, y represents the vertical time domain component in the time domain coordinate, G x Represents the gradient strength of the image in the x direction, G y Represents the gradient strength of the image in the y direction, and * represents the convolution operation;

[0026] Calculate the edge gradient strength according to the edge detection result:

[0027]

[0028] Where G refers to the gradient strength;

[0029] Perform a convolution operation on the edge gradient intensity to obtain the edge feature:

[0030] F edge =Conv(G);

[0031] Among them, F edge Represents the edge features of the final output, and Conv represents the convolution operation.

[0032] Optionally, superimposing the frequency domain feature, the edge feature and the original image feature includes:

[0033] F concat =Concat(F orig +F fft +F edge );

[0034] Among them, F orig represents the original image features, F fft represents the frequency domain feature, F edge represents the edge feature, F concatIt represents the features obtained after the superposition operation. Concat represents the operation of superimposing the frequency domain features, edge features and original image features. Here, an operation function named Concat is used to represent it. orig The number of channels representing the original image features, C fft The number of channels representing frequency domain features, C edge The number of channels representing edge features.

[0035] Optionally, performing channel attention weighting on the input feature map includes:

[0036] Calculate the channel attention and obtain the global features of the channel:

[0037]

[0038] Among them, X c represents the cth channel of the input feature map X, H represents the height of the input feature map, W represents the width of the input feature map, X c (i, j) represents the pixel value with coordinates (i, j) in the input feature map, Z c represents the global average pooling value of the cth channel, that is, the global feature of the channel;

[0039] Based on the global features, the attention weight of each channel is generated:

[0040] W c =Conv(Z c ,k);

[0041] Among them, Z c represents the feature of the cth channel after global average pooling, k represents the size of the one-dimensional convolution kernel used for the convolution operation, and W c Represents the attention weight of the cth channel generated after one-dimensional convolution;

[0042] Using the attention weight of each channel, the input feature map is weighted as follows:

[0043] Y c =X c ·W c ;

[0044] Among them, Y c Represents the feature map after channel attention weighting.

[0045] Optionally, determining whether the passenger wears the seat belt correctly includes:

[0046] If the seat belt recognition result is that the passenger is wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is not wearing the seat belt correctly, it is necessary to check the seat belt for abnormalities; if the seat belt recognition result is that the passenger is wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is wearing the seat belt correctly, it is proved that the passenger is wearing the seat belt correctly; if the seat belt recognition result is that the passenger is not wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is wearing the seat belt correctly, it is necessary to check the seat belt for abnormalities; if the seat belt recognition result is that the passenger is not wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is not wearing the seat belt correctly, it is proved that the passenger is not wearing the seat belt.

[0047] Optionally, the passenger status recognition result includes: sleeping, relaxing, neutral, tired, motion sickness, and vomiting.

[0048] To achieve the above object, the present invention also provides a passenger safety monitoring system for a bus, comprising:

[0049] An image acquisition module for capturing a sequence of passenger images;

[0050] A safety status recognition module is used to input the passenger image sequence into an improved YOLOv8n model to obtain a seat belt recognition result and a passenger status recognition result; the improved YOLOv8n model is obtained by training using a passenger seat belt dataset and a passenger status dataset and their corresponding labels; the improved YOLOv8n model is to enhance frequency domain features and edge feature processing on the basis of the YOLOv8n model, and to add a channel attention mechanism;

[0051] A seat belt detection module, used to determine whether the passenger is wearing the seat belt correctly based on the seat belt recognition result combined with the seat belt buckle sensor detection result;

[0052] The safety warning module is used to issue corresponding safety warnings to the determination results and the passenger status recognition results.

[0053] The beneficial effects of the present invention are:

[0054] The present invention improves YOLOv8n to improve the robustness and accuracy of blurred images and dim environments, and uses a passenger seat belt dataset and a passenger status dataset for training to obtain accurate recognition results.

[0055] The present invention combines the seat belt buckle sensor and the algorithm recognition result to make a more comprehensive judgment on the seat belt wearing situation of the passenger, which can better assist the driver in protecting the safety of the passengers.

[0056] The present invention helps the driver to better check the safety status of all passengers by sorting the safety status of passenger seats through an electronic display screen, reduces the driver's attention distraction during driving, and ensures the safety of passengers during riding. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0058] Figure 1 This is a flow chart of a bus passenger safety monitoring method according to an embodiment of the present invention;

[0059] Figure 2 A schematic diagram of the improved structure of YOLOv8n according to an embodiment of the present invention;

[0060] Figure 3 The figure is a schematic diagram of a passenger safety monitoring system for a bus according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] like Figure 1As shown, this embodiment discloses a passenger safety monitoring method for a bus, comprising: capturing a passenger image sequence, inputting the passenger image sequence into an improved YOLOv8n model, and obtaining a seat belt recognition result and a passenger status recognition result; the improved YOLOv8n model is obtained by training using a passenger seat belt data set and a passenger status data set and their corresponding labels; the improved YOLOv8n model comprises: adding a frequency domain enhancement unit and an edge feature extraction unit before the Backbone layer of the YOLOv8n model to extract frequency domain features and edge features, thereby superimposing the frequency domain features, the edge features and the original image features, and inputting them into the Backbone layer, adding an ECA attention unit to the Backbone layer to perform channel attention weighting on the input feature map; judging whether the passenger wears the seat belt correctly according to the seat belt recognition result combined with the detection result of the seat belt buckle sensor, and making corresponding safety warnings for the judgment result and the passenger status recognition result.

[0064] Further, capturing the passenger image sequence comprises:

[0065] The passenger image sequence is captured at a target frequency; the passenger image sequence is a two-dimensional image matrix sorted according to passenger seats:

[0066]

[0067] Among them, image t represents a set of passenger image sequences, t represents the time when the current image sequence is collected, and f(i,j) represents the passenger image in the i-th row and j-th column.

[0068] Specifically:

[0069] In this embodiment, passenger images are collected by a camera module. The camera module is a monocular camera, which is placed in a front position above the seat, such as directly above the front seat, and the camera lens is tilted toward the passenger position so that the camera can collect passenger images that meet the recognition requirements. The passenger image should include the passenger's head and torso, that is, the image of the area above the passenger's thighs.

[0070] When the system starts running, the camera module is started to capture passenger image sequences at a certain frequency and output the image sequences to the safety status recognition module.

[0071] The passenger image sequence refers to a two-dimensional image matrix arranged according to the passenger seats. For example, if the seats in a bus are arranged in 3 rows and 4 columns, the passenger image sequence collected by the camera module each time should be:

[0072]

[0073] Among them, imaget It refers to a set of passenger image sequences, t refers to the time when the current image sequence is collected, and the content in the image sequence is f(i,j), which represents the passenger image in the i-th row and j-th column.

[0074] The neural network model used in this embodiment is the improved YOLOv8n algorithm, which is one of the YOLOv8 series algorithms. Compared with other versions of the algorithm in this series, the parameter amount and model size of the YOLOv8n algorithm are the smallest, and the hardware requirements are low, which is suitable for deployment on terminal hardware. Although YOLOv8n can ensure a higher speed and a smaller parameter amount, its accuracy is lower than that of other versions of the algorithm in this series. At the same time, in the environment of the bus, the passengers are shaking, which may cause the passenger image collected by the camera module to be not clear enough. At the same time, when the bus is driving at night or passing through a tunnel, it is easy to have insufficient image brightness, resulting in low detection accuracy. In view of the above problems, in this embodiment, the YOLOv8n algorithm is improved to a certain extent to improve the generalization ability of the algorithm and improve the recognition ability in darker environments and blurred images.

[0075] A frequency domain enhancement unit (FFT Enhancement) and an edge feature extraction unit are added before the Backbone layer of YOLOv8n. The frequency domain features, edge features, and original image features are superimposed and then passed to the Backbone layer. An ECA attention unit is added to each convolutional layer in the Backbone layer to improve the accuracy of target recognition. The improved YOLOv8n structure is as follows: Figure 2 described.

[0076] Further, extracting the frequency domain features includes:

[0077] The time domain of the original image in the passenger image sequence is converted to the frequency domain, and the high-frequency information of the frequency domain image is extracted:

[0078] F′(u,v)=F(u,v)·H(u,v);

[0079] Where H(u,v) represents the transfer function of the high-pass filter, F'(u,v) represents the frequency domain image after retaining the high-frequency information, · represents the point-by-point multiplication operation, F(u,v) represents the frequency domain image converted from the time domain to the frequency domain, u represents the horizontal coordinate of the frequency domain image, and v represents the vertical coordinate of the frequency domain image;

[0080] The frequency domain image from which the high-frequency information is extracted is inversely transformed to convert the frequency domain image into a time domain image, thereby obtaining a time domain image containing the high-frequency information, namely, the frequency domain feature:

[0081]

[0082] Among them, f'(x,y) represents the converted time domain image with high-frequency information, W represents the width of the image, H represents the height of the image, and F'(u,v) represents the frequency domain image with high-frequency information retained. Here, it is used as input. represents the complex exponential function, e represents the natural constant, j represents the imaginary unit, represents the complex phase factor, ux represents the product of the abscissa u of the frequency domain image and the abscissa x of the time domain image, and ux represents the product of the ordinate v of the frequency domain image and the ordinate y of the time domain image.

[0083] Further, extracting the edge feature includes:

[0084] The original images in the passenger image sequence are converted into gray images by grayscale processing, and edge detection is performed on the gray images:

[0085]

[0086] Among them, f(x,y) represents the pixel value of the grayscale image, x represents the horizontal time domain component in the time domain coordinate, y represents the vertical time domain component in the time domain coordinate, G x Represents the gradient strength of the image in the x direction, G y Represents the gradient strength of the image in the y direction, and * represents the convolution operation;

[0087] Calculate the edge gradient strength according to the edge detection result:

[0088]

[0089] Where G refers to the gradient strength;

[0090] Perform a convolution operation on the edge gradient intensity to obtain the edge feature:

[0091] F edge =Conv(G);

[0092] Among them, F edge Represents the edge features of the final output, and Conv represents the convolution operation.

[0093] Furthermore, superimposing the frequency domain feature, the edge feature and the original image feature comprises:

[0094] F concat =Concat(F orig +F fft +F edge );

[0095] Among them, F orig represents the original image features, Ffft represents the frequency domain feature, F edge represents the edge feature, F concat It represents the features obtained after the superposition operation. Concat represents the operation of superimposing the frequency domain features, edge features and original image features. Here, an operation function named Concat is used to represent it. orig The number of channels representing the original image features, C fft The number of channels representing frequency domain features, C edge The number of channels representing edge features.

[0096] Further, performing channel attention weighting on the input feature map includes:

[0097] Calculate the channel attention and obtain the global features of the channel:

[0098]

[0099] Among them, X c represents the cth channel of the input feature map X, H represents the height of the input feature map, W represents the width of the input feature map, X c (i, j) represents the pixel value with coordinates (i, j) in the input feature map, Z c represents the global average pooling value of the cth channel, that is, the global feature of the channel;

[0100] Based on the global features, the attention weight of each channel is generated:

[0101] W c =Conv(Z c ,k);

[0102] Among them, Z c represents the feature of the cth channel after global average pooling, k represents the size of the one-dimensional convolution kernel used for the convolution operation, and W c Represents the attention weight of the cth channel generated after one-dimensional convolution;

[0103] Using the attention weight of each channel, the input feature map is weighted as follows:

[0104] Y c =X c ·W c ;

[0105] Among them, Y c Represents the feature map after channel attention weighting.

[0106] Specifically:

[0107] In this embodiment, a frequency domain enhancement unit (FFT EnhancementModule) is added before the Backbone layer of YOLOv8n. The original image is converted from the time domain to the frequency domain through Fourier transform, and the frequency domain image is operated to retain the high-frequency information. Finally, a reverse transform is performed to convert the frequency domain image into a time domain image. This module is divided into three steps:

[0108] (1) Fourier transform:

[0109] Through Fourier transform, the time domain image is converted into a frequency domain image. The conversion process is as follows:

[0110]

[0111] Among them, f(x,y) refers to the pixel value of the time domain image, x is the time domain component in the horizontal direction in the time domain coordinates, y is the time domain component in the vertical direction in the time domain coordinates, F(u,v) refers to the pixel value of the frequency domain image, u is the frequency component in the horizontal direction in the frequency domain coordinates, v is the frequency component in the vertical direction in the frequency domain coordinates, W is the width of the input time domain image, H is the height of the input time domain image, j is an imaginary unit, e refers to a natural constant, and π refers to a mathematical constant, also known as pi.

[0112] (2) Retain high-frequency information:

[0113] In the process of retaining high frequency information, a high pass filter is used, which is defined as follows:

[0114]

[0115] Among them, H(u,v) refers to the transfer function of the high-pass filter, and D0 refers to the cutoff frequency of the filter, which is used to control the range of high-frequency information retained. It refers to the distance of the current frequency component. The farther away from the center, the higher the frequency.

[0116] (3) The process of retaining high-frequency information is as follows:

[0117] F′(u,v)=F(u,v)·H(u,v);

[0118] Among them, F'(u,v) refers to the frequency domain image after retaining the high-frequency information, and · refers to the point-by-point multiplication operation.

[0119] Inverse Fourier transform:

[0120] After completing the high-frequency information retention, it is necessary to convert the frequency domain image into a time domain image through inverse Fourier transform. The process is as follows:

[0121]

[0122] Among them, f'(x,y) refers to the converted time domain image that retains high-frequency information.

[0123] An edge feature extraction unit is added before the Backbone layer of YOLOv8n. The edge features of the image are extracted through operators to improve the network's ability to perceive edge information. This module is divided into four steps:

[0124] (1) Grayscale processing:

[0125] In this embodiment, the color image needs to be converted into a gray image through grayscale processing. The grayscale processing uses a standard grayscale formula, which is as follows:

[0126] Y=0.299·R+0.587·G+0.114·B;

[0127] Where Y refers to the brightness value of the grayscale pixel, R refers to the red channel value in the color image, G refers to the green channel value in the color image, B refers to the blue channel value in the color image, 0.299, 0.587, and 0.114 are the weight values ​​corresponding to the red channel, green channel, and blue channel, respectively, and · refers to the multiplication operation.

[0128] (2) Use the operator to calculate the gradient strength:

[0129] In this embodiment, the Sobel operator is used for edge detection, and its formula is as follows:

[0130]

[0131] Among them, f(x,y) refers to the pixel value of the grayscale image, x is the horizontal time domain component in the time domain coordinate, y is the vertical time domain component in the time domain coordinate, G x Refers to the gradient strength of the image in the x direction, G y refers to the gradient strength of the image in the y direction, and * refers to the convolution operation.

[0132] (3) Then calculate the edge gradient strength:

[0133]

[0134] Here, G refers to the gradient strength.

[0135] (4) Edge feature convolution:

[0136] The extracted gradient strength is convolved as follows:

[0137] F edge =Conv(G);

[0138] Among them, F edgeIt refers to the edge features of the final output, and Conv refers to the convolution operation, which is used to enhance the edge features.

[0139] Feature stitching method:

[0140] In this patent, the frequency domain features and edge features obtained by the frequency domain enhancement unit and the edge feature extraction unit need to be spliced ​​with the original image, and then the spliced ​​image is input into the subsequent Backbone layer for convolution operation. The splicing method is as follows:

[0141] The frequency domain features, edge features and original image features are superimposed on the channel. After superposition, the height and width of the feature map do not change, but the number of channels increases, which is equal to the sum of the number of channels of the frequency domain features, edge features and original image features:

[0142] F concat =Concat(F orig +F fft +F edge );

[0143] Among them, F orig is the original image feature, the size is (C orig ,H,W),F fft is the frequency domain feature, and its size is (C fft ,H,W),F edge is the edge feature, and its size is (C edge ,H,W),F concat is the feature obtained after the superposition operation, and its size is (C concat ,H,W), where C concat =C orig +C fft +C edge , C orig The number of channels representing the original image features, C fft The number of channels representing frequency domain features, C edge Represents the number of channels of edge features, H is the height of the feature map, and W is the width of the feature map.

[0144] ECA Attention Unit:

[0145] In this embodiment, an ECA attention unit is added to each convolutional layer of the Backbone layer to improve the perception ability of the YOLOv8n model to the target object. The working order and formula are as follows:

[0146] (1) Channel attention calculation:

[0147]

[0148] Among them, X cis the cth channel of the input feature map X, H is the height of the input feature map, W is the width of the input feature map, X c (i, j) is the pixel value with coordinates (i, j) in the input feature map, Z c It refers to the global average pooling value of the cth channel, which represents the global feature of the channel.

[0149] (2) Local cross weight:

[0150] W c =Conv(Z c ,k);

[0151] Among them, Z c refers to the feature of the cth channel after global average pooling, k refers to the size of the one-dimensional convolution kernel used for convolution operation, which is used to control the channel crossover range, and W c It refers to the attention weight of the cth channel generated after one-dimensional convolution.

[0152] (3) Weighting the input feature map:

[0153] Y c =X c ·W c ;

[0154] Among them, Y c It refers to the feature map after channel attention weighting.

[0155] Further, determining whether the passenger is wearing the seat belt correctly includes:

[0156] If the seat belt recognition result is that the passenger is wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is not wearing the seat belt correctly, it is necessary to check the seat belt for abnormalities; if the seat belt recognition result is that the passenger is wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is wearing the seat belt correctly, it is proved that the passenger is wearing the seat belt correctly; if the seat belt recognition result is that the passenger is not wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is wearing the seat belt correctly, it is necessary to check the seat belt for abnormalities; if the seat belt recognition result is that the passenger is not wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is not wearing the seat belt correctly, it is proved that the passenger is not wearing the seat belt.

[0157] Furthermore, the passenger status recognition result includes: sleeping, relaxing, neutral, tired, motion sickness, and vomiting.

[0158] Specifically:

[0159] The YOLOv8n algorithm in this embodiment uses a passenger seat belt dataset and a passenger status dataset for joint training, wherein the passenger seat belt dataset refers to either a public dataset or a self-built dataset, and the dataset content is a real-life picture of a passenger or a driver in a passenger car, and the corresponding label of the dataset is whether the passenger or the driver wears the seat belt correctly. When the passenger or the driver in the picture wears the seat belt correctly, the corresponding label is true, otherwise it is false; wherein the passenger status dataset refers to either a public dataset or a self-built dataset, and the dataset content is a facial picture of passengers when riding in a vehicle, reflecting the expression of the passenger's physical state and emotions in facial expressions when riding in a vehicle, corresponding to multiple states, including sleep, relaxation, neutral, tiredness, motion sickness, and vomiting.

[0160] Furthermore, issuing corresponding safety warnings for the determination result and the passenger status recognition result includes:

[0161] Safety warnings for the determination results include:

[0162] If the passenger wears the seat belt correctly, no warning is needed. If the seat belt needs to be checked for abnormal conditions, the electronic display will issue a warning, and the voice broadcast will issue a warning. If the passenger does not wear the seat belt correctly, both the electronic display and the voice broadcast will issue a warning.

[0163] Specifically:

[0164] The seat belt detection module in this embodiment detects whether the seat belt buckle is used through the seat belt buckle sensor and outputs the detection result. The seat belt sensor is installed in the seat belt socket. If the seat belt buckle sensor detects that the seat belt is used, the detection result is true, otherwise it is false. In combination with the seat belt recognition result received from the safety status recognition module, the values ​​of the two parameters are combined to make a judgment on whether the passenger wears the seat belt correctly, and the judgment result is output to the safety warning module. The comprehensive judgment method of this module is as shown in Table 1 below:

[0165] Table 1

[0166]

[0167] Providing a safety warning for the passenger status recognition result includes:

[0168] If the passenger status recognition result is sleep and the sleep duration is within the target duration, no warning is required; if the passenger status recognition result is sleep and the sleep duration is outside the target duration, the electronic display end will issue a warning, the voice broadcast end will issue a warning, and there will be no warning; if the passenger status recognition result is relaxation or neutral, no warning is required; if the passenger status recognition result is tiredness, the electronic display end will issue a warning, the voice broadcast end will issue a warning, and there will be no warning; if the passenger status recognition result is motion sickness or vomiting, both the electronic display end and the voice broadcast end will issue a warning.

[0169] Specifically:

[0170] The safety warning module in this embodiment receives the passenger status recognition result from the safety status recognition module and the passenger seat belt wearing result from the seat belt detection module, and displays the safety status of the bus passengers according to the distribution of passenger seats through an electronic display screen with interactive functions installed at the driving position. Each passenger has two indicators, one is the seat belt wearing judgment result, and the other is the passenger status. The indicator status is changed on the display screen according to the passenger indicator. When a warning is needed, the module reminds the driver by calling the voice broadcast device. Among them, according to the signal transmitted from the seat belt detection module, the driver is given different degrees of safety warning according to different situations. The specific judgment method is shown in Table 2:

[0171] Table 2

[0172] Seat belt detection module results Warning level Electronic display warning Voice broadcast warning Passengers wearing seat belts correctly Safety No warning No warning Need to check Early Warning Early Warning No warning Passenger not wearing seat belt Danger Early Warning Early Warning

[0173] According to the signals transmitted from the safety status recognition module, the driver is given different degrees of safety warnings according to different situations. The specific judgment method is as follows. For the sleep state, judgment conditions of different durations are set. A short duration means that a passenger is monitored to be in a sleep state for 5 to 30 minutes. A long duration means that a passenger is monitored to be in a sleep state for more than 30 minutes. When it exceeds 30 minutes, the warning level is changed and a corresponding warning is issued, as shown in Table 3.

[0174] Table 3

[0175]

[0176]

[0177] The electronic display screen should have interactive functions. When the driver finds the warning and completes the passenger safety check, the passenger's safety status can be changed through the interactive function so that the safety warning module no longer warns, reducing invalid warnings and interference to the driver during driving.

[0178] like Figure 3As shown, this embodiment also provides a bus passenger safety monitoring system, including: an image acquisition module, used to capture a passenger image sequence; a safety status recognition module, used to input the passenger image sequence into an improved YOLOv8n model to obtain a seat belt recognition result and a passenger status recognition result; the improved YOLOv8n model is trained using a passenger seat belt data set and a passenger status data set and their corresponding labels; the improved YOLOv8n model includes: adding a frequency domain enhancement unit and an edge feature extraction unit before the Backbone layer of the YOLOv8n model to extract frequency domain features and edge features, thereby superimposing the frequency domain features, the edge features and the original image features, and inputting them into the Backbone layer, adding an ECA attention unit to the Backbone layer to perform channel attention weighting on the input feature map; a seat belt detection module, used to determine whether the passenger is wearing the seat belt correctly according to the seat belt recognition result combined with the seat belt buckle sensor detection result; a safety warning module, used to make corresponding safety warnings for the determination result and the passenger status recognition result.

[0179] Specifically, the camera module collects images of the area where each passenger is located during the driving of the bus in real time, and forms an image sequence according to the seat arrangement, and outputs the image sequence to the safety status recognition module. The safety status recognition module calls the pre-trained YOLOv8n neural network model to recognize the safety status of the bus passengers, including: recognition of the passenger's seat belt wearing status and recognition of the passenger's status through the passenger's facial image, and outputs the recognition results to the seat belt detection module and the safety warning module respectively. The seat belt detection module uses the recognition result of the passenger's seat belt wearing status and the seat belt buckle sensor to jointly determine the passenger's seat belt wearing status, and outputs the passenger's seat belt wearing status to the safety warning module. When the safety warning module receives that the passenger fails to wear the seat belt correctly or has physical discomfort, the module will remind the driver in real time through the interactive display screen or voice broadcast system, so that the driver can handle it in time. After processing, the driver can change the passenger's safety status in the interactive interface of the safety warning module and cancel the warning. As a driving assistance system, the system can not only ensure the safety of bus passengers during riding, but also reduce the driver's energy consumption and improve work efficiency.

[0180] The present embodiment discloses a passenger safety monitoring method for a bus. When the bus is started, the bus passenger status monitoring system starts working, the camera module starts to collect passenger images and outputs the image sequence to the safety status recognition module, and the seat belt detection module starts to continuously detect the seat belt buckle sensor signal; after the safety status recognition module receives the image sequence, it starts to call the YOLOv8n algorithm to perform seat belt recognition and passenger status recognition, wherein the seat belt recognition result is output to the seat belt detection module, and the passenger status recognition result is output to the safety warning module; after receiving the seat belt recognition result, the seat belt detection module performs a comprehensive judgment in conjunction with the seat belt buckle sensor detection result, and outputs the judgment result to the safety warning module; the safety warning module receives the passenger After the status recognition result is obtained, the status indicator of the corresponding seat passenger is adjusted according to the recognition result. After receiving the judgment result of the seat belt detection module, the seat belt wearing judgment result indicator of the corresponding seat passenger is adjusted according to the judgment result. Then, according to the result, the driver is given an early warning reminder in the form of an electronic display screen or voice broadcast. After receiving the early warning signal, the driver gives safety reminders to passengers in warning or dangerous states through the microphone and audio system inside the bus. If motion sickness or vomiting occurs, corresponding drugs can be provided to help relieve the symptoms. After ensuring that the passenger is in a safe state, the driver can use the interactive function of the electronic display screen to click on the screen to change the passenger's safety status, cancel the safety warning for the passenger, and complete a passenger status monitoring.

[0181] This embodiment proposes a passenger safety monitoring method and system for buses, which uses a targeted improved visual recognition algorithm and combines physical sensors to monitor the two major safety indicators of passengers, and displays them centrally through the central control electronic display screen, making it convenient for the driver to view and check the safety status of passengers. As a driving assistance system, this embodiment can help the driver pay attention to the safety status of passengers in a timely manner, ensure the safety of passengers during driving, and reduce the driver's energy allocation in passenger safety during work, thereby improving work efficiency while ensuring safety.

[0182] In this embodiment, the recognition technology of passenger images is indispensable. The safety status recognition module can obtain the passenger seat belt detection results and status recognition results, and transmit them to the next module as input, and combine the seat belt buckle sensor to obtain the passenger seat belt wearing status. Therefore, in this embodiment, image recognition technology is indispensable. There are certain expansion plans for this technology, and other reference factors can be added, such as weather, road traffic conditions and other external factors, to obtain control input parameters that are more in line with the current driving scene.

[0183] In addition to being applicable to passenger buses, this embodiment can also be applied to scenarios where the status of a person needs to be monitored, and more application scenarios can be developed, such as monitoring the status of children while riding in a vehicle, which helps to ensure the safety and status of children while riding in a vehicle.

[0184] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A passenger safety monitoring method for a bus, characterized in that: include: Capturing a passenger image sequence, inputting the passenger image sequence into the improved YOLOv8n model, and obtaining a seat belt recognition result and a passenger status recognition result; The improved YOLOv8n model is obtained by training using a passenger seat belt dataset and a passenger status dataset and their corresponding labels; the improved YOLOv8n model is to enhance frequency domain features and edge feature processing on the basis of the YOLOv8n model, and add a channel attention mechanism; According to the seat belt recognition result combined with the seat belt buckle sensor detection result, it is determined whether the passenger is wearing the seat belt correctly, and a corresponding safety warning is issued for the determination result and the passenger status recognition result.

2. The bus passenger safety monitoring method according to claim 1, characterized in that: Capturing the passenger image sequence comprises: The passenger image sequence is captured at a target frequency; the passenger image sequence is a two-dimensional image matrix sorted according to passenger seats: Among them, image t represents a set of passenger image sequences, t represents the time when the current image sequence is collected, and f(i,j) represents the passenger image in the i-th row and j-th column.

3. The bus passenger safety monitoring method according to claim 1, characterized in that: By enhancing the frequency domain features and edge feature processing based on the YOLOv8n model and adding the channel attention mechanism, the following are included: A frequency domain enhancement unit and an edge feature extraction unit are added before the Backbone layer of the YOLOv8n model to extract frequency domain features and edge features, so as to superimpose the frequency domain features, the edge features and the original image features and input them into the Backbone layer; an ECA attention unit is added to the Backbone layer to perform channel attention weighting on the input feature map.

4. The bus passenger safety monitoring method according to claim 3, characterized in that: Extracting the frequency domain features includes: The time domain of the original image in the passenger image sequence is converted to the frequency domain, and the high-frequency information of the frequency domain image is extracted: F'(u,v)=F(u,v)·H(u,v); Where H(u,v) represents the transfer function of the high-pass filter, F'(u,v) represents the frequency domain image after retaining the high-frequency information, · represents the point-by-point multiplication operation, F(u,v) represents the frequency domain image converted from the time domain to the frequency domain, u represents the horizontal coordinate of the frequency domain image, and v represents the vertical coordinate of the frequency domain image; The frequency domain image from which the high-frequency information is extracted is inversely transformed to convert the frequency domain image into a time domain image, thereby obtaining a time domain image containing the high-frequency information, namely, the frequency domain feature: Among them, f'(x,y) represents the converted time domain image with high-frequency information, W represents the width of the image, H represents the height of the image, and F'(u,v) represents the frequency domain image with high-frequency information retained. Here, it is used as input. represents the complex exponential function, e represents the natural constant, j represents the imaginary unit, π represents the circumference of a circle, represents the complex phase factor, ux represents the product of the abscissa u of the frequency domain image and the abscissa x of the time domain image, and vy represents the product of the ordinate v of the frequency domain image and the ordinate y of the time domain image.

5. The bus passenger safety monitoring method according to claim 3, characterized in that: Extracting the edge features includes: The original images in the passenger image sequence are converted into gray images through grayscale processing, and edge detection is performed on the gray images: Among them, f(x,y) represents the pixel value of the grayscale image, x represents the horizontal time domain component in the time domain coordinate, y represents the vertical time domain component in the time domain coordinate, G x Represents the gradient strength of the image in the x direction, G y Represents the gradient strength of the image in the y direction, and * represents the convolution operation; Calculate the edge gradient strength according to the edge detection result: Where G refers to the gradient strength; Perform a convolution operation on the edge gradient intensity to obtain the edge feature: F edge =Conv(G); Among them, F edge Represents the edge features of the final output, and Conv represents the convolution operation.

6. The bus passenger safety monitoring method according to claim 3, characterized in that: Superimposing the frequency domain feature, the edge feature and the original image feature comprises: F concat =Concat(F orig +F fft +F edge ); Among them, F orig represents the original image features, F fft represents the frequency domain feature, F edge represents the edge feature, F concat It represents the features obtained after the superposition operation. Concat represents the operation of superimposing the frequency domain features, edge features and original image features. Here, an operation function named Concat is used to represent it. orig The number of channels representing the original image features, C fft The number of channels representing frequency domain features, C edge The number of channels representing edge features.

7. The bus passenger safety monitoring method according to claim 3, characterized in that: Performing channel attention weighting on the input feature map includes: Calculate the channel attention and obtain the global features of the channel: Among them, X c represents the cth channel of the input feature map X, H represents the height of the input feature map, W represents the width of the input feature map, X c (i, j) represents the pixel value with coordinates (i, j) in the input feature map, Z c represents the global average pooling value of the cth channel, that is, the global feature of the channel; Based on the global features, the attention weight of each channel is generated: W c =Conv(Z c ,k); Among them, Z c represents the feature of the cth channel after global average pooling, k represents the size of the one-dimensional convolution kernel used for the convolution operation, and W c Represents the attention weight of the cth channel generated after one-dimensional convolution; Using the attention weight of each channel, the input feature map is weighted as follows: Y c =X c ·W c ; Among them, Y c Represents the feature map after channel attention weighting.

8. The bus passenger safety monitoring method according to claim 1, characterized in that: Determining whether a passenger is wearing a seat belt correctly includes: If the seat belt recognition result is that the passenger is wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is not wearing the seat belt correctly, it is necessary to check the seat belt for abnormalities; if the seat belt recognition result is that the passenger is wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is wearing the seat belt correctly, it is proved that the passenger is wearing the seat belt correctly; if the seat belt recognition result is that the passenger is not wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is wearing the seat belt correctly, it is necessary to check the seat belt for abnormalities; if the seat belt recognition result is that the passenger is not wearing the seat belt correctly, and the seat belt buckle sensor detection result is that the passenger is not wearing the seat belt correctly, it is proved that the passenger is not wearing the seat belt.

9. The bus passenger safety monitoring method according to claim 1, characterized in that: The passenger status recognition results include: sleeping, relaxing, neutral, tired, motion sickness, and vomiting.

10. A passenger safety monitoring system for a bus, characterized in that: include: An image acquisition module for capturing a sequence of passenger images; A safety status recognition module is used to input the passenger image sequence into an improved YOLOv8n model to obtain a seat belt recognition result and a passenger status recognition result; the improved YOLOv8n model is obtained by training using a passenger seat belt dataset and a passenger status dataset and their corresponding labels; the improved YOLOv8n model is to enhance frequency domain features and edge feature processing on the basis of the YOLOv8n model, and to add a channel attention mechanism; A seat belt detection module, used to determine whether the passenger is wearing the seat belt correctly based on the seat belt recognition result combined with the seat belt buckle sensor detection result; The safety warning module is used to issue corresponding safety warnings to the determination results and the passenger status recognition results.

Images