Multimodal dangerous environment perception and warning method for head-down users based on mobile terminals

Abstract

The present invention discloses a multi-modal mobile terminal-based dangerous environment perception and warning method for head-down people, comprising the following steps: Step 1: construct a user motion state classification module based on a parallel LSTM-CNN network to identify the current motion state of a mobile phone user; Step 2: use a monocular camera to collect images and estimate the depth information of the monocular image; for the depth information of the color image sequence and the depth map, predict the perspective expansion image; Step 3: analyze the motion state recognition result of step 1, construct a dangerous environment detection module based on a YOLOv5 lightweight network, and identify the dangerous environment; Step 4: construct a head-down dangerous state judgment module based on a decision tree model to classify the current dangerous state of the head-down people; Step 5: design a mobile phone user interface and establish an auditory-visual stereo warning method. The present invention can provide a dangerous warning when the head-down people use their mobile phones, effectively reducing the accident rate.

Description

Technical Field

[0001] The present invention relates to a multi-modal mobile terminal-based dangerous environment perception and early warning method for people with low head phones, belonging to the technical field of multi-sensor data fusion. Background Art

[0002] With the rapid development of the Internet and modern communication technology, smartphones have gradually become an indispensable part of people's daily lives. However, "never leaving the phone" has also become a common social phenomenon. Many people look down at their phones for a long time, often ignoring the surrounding environment, which not only easily causes various health problems, but also poses a great safety hazard. For example, playing with a phone while going up and down stairs can easily cause a fall, and playing with a phone while crossing the road can cause a traffic accident.

[0003] Current mobile terminals do not have early warning measures when used in the above-mentioned dangerous scenarios, and are unable to provide users with sufficient safety warnings, which can easily lead to users developing a mentality of luck and dependence, or even not caring about their own dangerous behaviors. At present, there is no environmental detection system for extremely dangerous situations, and there are very few software that can perform possible detections on pedestrians' walking status, and the design angles are single. Most of them are zebra crossing detection or the distinction between running and stillness. More comprehensive behavior detection and walking environment detection have not yet been introduced. Therefore, it is necessary to design a new method that can monitor in real time, obtain the surrounding environment efficiently and timely, and perform detection and judgment, with a detection efficiency of up to 90% or more. It can also provide sound warnings to remind people to pay attention to the surrounding environment to avoid dangerous situations. Summary of the invention

[0004] In view of the problems existing in the above-mentioned prior art, the present invention provides a multimodal dangerous environment perception and warning method for "head-down people" based on mobile terminals, which utilizes the sensors built into the mobile phone to detect the user's behavior status and environmental conditions in real time without adding any other equipment; and establishes a "head-down people" danger warning system based on the sensor detection data, further strengthens the user's attention to the walking environment through text and voice prompts, effectively reduces the accident rate, and ensures the safety of the user.

[0005] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a multi-modal mobile terminal-based dangerous environment perception and warning method for head-down people, characterized in that it includes the following steps:

[0006] Step 1: Use the built-in acceleration sensor of the mobile phone to collect three-directional acceleration time series data; extract the acceleration state features from the three-directional acceleration time series data, and build a user motion state classification module based on the parallel LSTM-CNN network (holding the mobile phone and walking slowly, putting the mobile phone in the pocket and walking slowly, leaving the mobile phone still, walking slowly and looking at the mobile phone, going up and down the stairs and looking at the mobile phone, and standing still and looking at the mobile phone) to identify the current motion state of the mobile phone user;

[0007] Step 2: Use a monocular camera to capture images and estimate the depth information of the monocular image; based on the depth information of the color image sequence and the depth map, predict the view extension image;

[0008] Step 3: Analyze the motion state recognition results of step 1. When the user is in a non-stationary state, turn on the rear-view camera of the mobile phone to collect image data of the user's surrounding environment; based on the image data of the user's surrounding environment, build a dangerous environment detection module based on the YOLOv5 lightweight network to identify potential dangerous environments around the mobile phone user (zebra crossings, stair areas, slippery roads, low lighting);

[0009] Step 4: Based on the mobile phone user motion state recognition results and surrounding dangerous environment recognition results output from Step 1 and Step 3, a dangerous state judgment module for the head-down users is constructed based on a decision tree model to classify the current dangerous state of the head-down users.

[0010] Step 5: Design a mobile user interface based on the real-time recognition results of the mobile phone user's motion status, the recognition results of the dangerous environment around the user, and the judgment results of the dangerous status of the "phone addicts", and establish an auditory-visual three-dimensional warning method.

[0011] Furthermore, the step 1 is specifically as follows:

[0012] Step 11: For the built-in gravity acceleration sensor of the mobile phone, collect acceleration time series data of 13 different users at a sampling frequency of 20 Hz in six states: holding the mobile phone and walking slowly, putting the mobile phone in the pocket and walking slowly, leaving the mobile phone still, walking slowly and looking at the mobile phone, walking up and down the stairs and looking at the mobile phone, and standing still and looking at the mobile phone. Collect 10 seconds of data in each state;

[0013] Step 12: Label the collected data according to different users and states; a, b, c, d, e, f, g, h, i, j, k, l, and m are used to represent 13 different users, and 1, 2, 3, 4, 5, and 6 are used to represent six states: holding a mobile phone and walking slowly, putting a mobile phone in a pocket and walking slowly, leaving the mobile phone still, walking slowly and looking at the mobile phone, walking up and down stairs and looking at the mobile phone, and standing still and looking at the mobile phone. The specific expressions are:

[0014]

[0015] in, It represents the time series data of the x, y, and z axes acceleration collected by the gravity acceleration sensor when user a is walking slowly with a mobile phone in hand. X 200 ={x1, x2, ...X 200},Y 200 ={y1, y2, ...y 200}, Z 200 ={z1, z2, ... z 200};

[0016] Step 13: Input data for x, y, and z axis acceleration and manually extract statistical features T imput , including the average value in each state: μ x , μ y , μ z , variance: σ x , σ y , σ z and the Pearson correlation coefficient between the two axes: ρ xy , ρ xz , ρ yz . The calculation formula of Pearson correlation coefficient is as follows:

[0017]

[0018] Step 14: The x, y, and z axis acceleration data are superimposed in time to generate a signal image. Then, a two-dimensional discrete Fourier transform (DFT) is applied to the signal image and its amplitude is selected as the active image F. imput . The formula for the two-dimensional discrete Fourier transform is as follows:

[0019]

[0020] Step 15: Collect raw data S from the gravity acceleration sensor input , statistic T imput , frequency image F imput For input data, a user state recognition network based on LSTM-CNN is constructed, data features are extracted through the DWConv deep convolution module, and the maximum pooling layer and downsampling layer are used to enhance the robustness of the model and reduce the risk of overfitting.

[0021] Furthermore, the step 2 is specifically as follows:

[0022] Step 21: First, the depth of the monocular image is estimated using the decoder structure based on the Laplacian pyramid to accurately explain the relationship between the encoded features and the final output; the Laplacian residual of the input color image is L k , specifically expressed as:

[0023] L k =I k -U p (I k+1 ),k=1,2,3,4 (4)

[0024] Where k is the horizontal index in the Laplace pyramid, I k is obtained by downsampling the original input image, U p represents the upsampling function;

[0025] The depth residual obtained at the kth pyramid level is R k , specifically expressed as:

[0026] R k =B k ([x k ,L k ,U p (R k+1 )])+L k ,k=1,2,3,4 (5)

[0027] Among them, [x k ,L k ,U p (R k+1 )] is in series, B k It consists of stacked convolutional blocks;

[0028] Starting from the highest level of the Laplacian pyramid, the depth map is gradually reconstructed, which is specifically expressed as:

[0029] D k =R k +U p (D k+1 ),k=1,2,3,4 (6);

[0030] Step 22: Then, a standard fully convolutional architecture is used to predict the predicted pixel depth and normal map for a single image in two independent streams respectively; on each plane, given the required relative pose, i.e. 3D rotation and translation, and the intrinsic parameters of the camera, a homography can be calculated from its depth and normal to model m different planes and segment the input image into m regions; use the first 4 convolutional modules of VGG to apply the reverse warping process from the input image and the seed region mask to predict m planar region masks in the new view, combine these segmentation masks with the relative pose to produce their own homography transformations, and generate new views based on the warped input image combination.

[0031] Furthermore, the step 3 is specifically as follows:

[0032] Step 31: First, build the YOLOv5s 6.0 lightweight target detection network under the pytorch framework. The network consists of four modules: Input module, Backbone network module, Neck network module and Head module;

[0033] Step 32: Then, make four kinds of image datasets: stage (stairs), wetroad (slippery road), zcrossing (zebra crossing), and night (night), with 2500, 1500, 1500, and 1500 images respectively, and 1000 negative samples. Divide the datasets into training set, validation set, and test set in a ratio of 7:2:1;

[0034] Step 33: Use the labelImg tool to manually label the training set of the four categories of images, then perform 200 rounds of training to generate the best.pt model, and verify the effectiveness of the trained model using the validation set and test set data.

[0035] Furthermore, the Input module includes three parts: Mosaic data enhancement, adaptive anchor frame calculation, and image size processing; the standard size of the input image is 640×640×3. After Mosaic data enhancement, any four images in the data set are randomly scaled, cropped, and arranged for splicing, and then adaptive frame calculation is performed. The initial frames set by YOLOv5s are [70, 28, 74, 100, 88, 213], [147, 67, 181, 106, 189, 303], and [261, 132, 513, 281, 544, 395]. When the network is trained, the output prediction frame is based on the initial frame, and then compared with the real frame groundtruth, and the difference calculated between the two is reversely updated, thereby iteratively updating the network parameters;

[0036] The Backbone network module consists of 1 Fcous structure, 5 Conv modules, 21 C3 modules, and 1 SPPF module. The Focus structure takes a 640×640×3 image as input, uses a slicing operation, and first converts it into a 320×320×12 feature map, then undergoes a 3×3 convolution operation, outputs 32 channels, and finally becomes a 320×320×32 feature map; the Conv module includes convolution (Conv2d), Batch Normalization, and activation function (SiLU); the C3 module contains 3 standard convolution layers and multiple Bottleneck modules; the SPPF module uses multiple small-size pooling kernels in cascade to replace the single large-size pooling kernel in the SPP module;

[0037] The Neck network module is composed of a FPN+PAN network structure. The FPN structure from top to bottom processes the information transfer process through an upsampling operation to obtain a predicted feature map; the feature pyramid from bottom to top is applied in the PAN structure.

[0038] The Head (prediction end) module consists of 4 Conv modules, 2 nn.Upsample modules, 4 Concat (concatenation layer) modules, 12 C3 modules, and 1 Detect module, which is mainly responsible for multi-scale target detection on the feature maps extracted by the backbone network.

[0039] Furthermore, the step 4 is specifically as follows:

[0040] Step 41: First, determine three features of the input data: whether the user is exercising, whether the user is playing with a mobile phone, and whether the user is in a dangerous environment, and establish a decision tree model based on these features;

[0041] Step 42: Then, "whether the user is exercising" is selected as the root node of the decision tree. This feature can best classify whether the mobile phone user is in a dangerous state;

[0042] Step 43: Finally, the Gini coefficients of the two features "whether the user is playing with the phone" and "whether the user is in a dangerous environment" are calculated to determine the internal nodes and leaf nodes of the decision tree model; the calculation formula of the Gini coefficient is as follows:

[0043]

[0044] Among them, p k Represents the probability of occurrence of a certain category.

[0045] Furthermore, the step 5 is specifically as follows:

[0046] The user interface is divided into two parts, upper and lower. The lower part detects the user's activity status in real time, and displays real-time data detected by the x, y, and z three-axis gravity acceleration sensors for six situations: walking slowly with the phone in hand, walking slowly with the phone in the pocket, leaving the phone still, walking slowly and looking at the phone, going up and down stairs and looking at the phone, and standing still and looking at the phone. The upper part of the interface displays the user's environment in real time, and at the same time determines the user's dangerous state. When it is detected that the user is using the phone in dangerous environments such as stairs, zebra crossings, at night, and slippery roads, the interface provides early warnings to the "head-down" users through text display and voice broadcast.

[0047] The beneficial effect of the present invention is that the present invention has developed a multi-modal dangerous environment perception system method for head-down people based on mobile terminals. By real-time monitoring of the walking environment, and judging the state according to the detection data of the acceleration sensor, and comprehensively deriving the possibility of the state result, an early warning is issued in time. At the same time, the present invention can not only monitor in real time, but also obtain the surrounding environment efficiently and timely and perform detection and judgment, and the detection efficiency is as high as 90% or more, and it can also provide sound warnings to remind people to pay attention to the surrounding environment to avoid the occurrence of dangerous conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of the steps of a multi-modal mobile terminal-based dangerous environment perception and warning method for head-down people proposed by the present invention;

[0049] Figure 2 This is an example diagram of the three-axis acceleration time series collected by the built-in acceleration sensor of the mobile phone under the six mobile phone user motion states proposed by the present invention;

[0050] Figure 3 A process diagram of generating a moving image in the frequency space proposed by the present invention;

[0051] Figure 4 This is the network structure diagram of the mobile phone user motion state detection based on LSTM-CNN proposed by the present invention;

[0052] Figure 5 The accuracy and loss curve of the training and testing of the LSTM-CNN mobile phone user motion state detection network model proposed by the present invention;

[0053] Figure 6 A diagram of the image viewing angle expansion process based on a monocular image sequence proposed by the present invention;

[0054] Figure 7 This is a network architecture diagram of the dangerous scene detection module based on the YOLOv5 lightweight network proposed in the present invention;

[0055] Figure 8 This is a model diagram for determining the dangerous state of the group of people looking down at their phones based on a decision tree proposed by the present invention;

[0056] Fig. 9 The user interface design and the auditory-visual stereo warning design diagram proposed by the present invention DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below through the accompanying drawings and embodiments. However, it should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the scope of the present invention.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which the present invention belongs. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0059] like Figure 1 As shown, a multi-modal mobile terminal-based dangerous environment perception and warning method for head-down people includes the following steps:

[0060] Step 1: Use the built-in acceleration sensor of the mobile phone to collect three-directional acceleration time series data; extract the acceleration state features from the three-directional acceleration time series data, build a user motion state classification module based on the parallel LSTM-CNN network, and identify the current motion state of the mobile phone user; the specific contents are as follows:

[0061] First, we collected acceleration time series data of 13 different users at a sampling frequency of 20 Hz in six states: holding a mobile phone while walking slowly, putting the mobile phone in a pocket while walking slowly, leaving the mobile phone still, walking slowly while looking at the mobile phone, walking up and down stairs while looking at the mobile phone, and standing still while looking at the mobile phone. We collected 10 seconds of data in each state. Figure 2 Shown are examples of data collection under six conditions.

[0062] Next, the collected data is labeled according to different users and states. A, B, C, D, E, F, G, H, I, J, K, L, and M represent 13 different users, and 1, 2, 3, 4, 5, and 6 represent six states: holding a mobile phone and walking slowly, putting the mobile phone in the pocket and walking slowly, leaving the mobile phone still, walking slowly and looking at the mobile phone, walking up and down stairs and looking at the mobile phone, and standing still and looking at the mobile phone. The specific expressions are:

[0063]

[0064] in, It represents the time series data of the x, y, and z axes acceleration collected by the gravity acceleration sensor when user a is walking slowly with a mobile phone in hand. X 200 ={x1,x2,…x 200},Y 200 ={y1,y2,y 200}, Z 200 ={z1,z2,…z 200}.

[0065] Then, input the acceleration data of the x, y, and z axes and manually extract the statistical features T imput , including the average value in each state: μ x , μ y , μz , variance: σ x , σ y , σ z and the Pearson correlation coefficient between the two axes: ρ xy , ρ xz , ρ yz . The calculation formula of Pearson correlation coefficient is as follows:

[0066]

[0067] Next, the x, y, and z axis acceleration data are superimposed in time to generate a signal image. Then, a two-dimensional discrete Fourier transform (DFT) is applied to the signal image and its amplitude is selected as our active image F. imput , the process is as follows Figure 3 As shown. The formula for the two-dimensional discrete Fourier transform is as follows:

[0068]

[0069] Finally, the original data S collected by the gravity acceleration sensor input , statistic T imput , frequency image F imput For input data, a user status recognition network based on LSTM-CNN is constructed, data features are extracted through the DWConv deep convolution module, and the maximum pooling layer and downsampling layer are used to enhance the robustness of the model and reduce the risk of overfitting. The network structure is as follows Figure 4 shown.

[0070] Step 2: Use a monocular camera to capture images and estimate the depth information of the monocular image; predict the perspective expansion image based on the depth information of the color image sequence and the depth map; the specific contents are as follows:

[0071] First, the depth of the monocular image is estimated using a decoder structure based on the Laplacian pyramid, which accurately explains the relationship between the encoded features and the final output. The Laplacian residual of the input color image is L k , specifically expressed as:

[0072] L k =I k -U p (I k+1 ),k=1,2,3,4 (4)

[0073] Where k is the horizontal index in the Laplace pyramid. k is obtained by downsampling the original input image, U p represents the upsampling function.

[0074] The depth residual obtained at the kth pyramid level is R k , specifically expressed as:

[0075] R k =B k ([x k , L k , U p (R k+1 )])+L k , k = 1, 2, 3, 4 (5)

[0076] Among them, [x k , L k , U p (R k+1 )] is connected in series. B k It consists of stacked convolutional blocks.

[0077] Starting from the highest level of the Laplacian pyramid, the depth map is gradually reconstructed, which is specifically expressed as:

[0078] D k =R k +U p (D k+1 ), k = 1, 2, 3, 4 (6)

[0079] Then, the predicted pixel depth and normal map for a single image are predicted in two separate streams using a standard fully convolutional architecture, as Figure 6 shown.

[0080] In each plane, given the desired relative pose, i.e. 3D rotation and translation, and the intrinsic parameters of the camera, a homography can be computed from its depth and normal to model m different planes, segmenting the input image into m regions. The first 4 convolutional modules of VGG are used to apply the reverse warping process from the input image and the seed region mask to predict m planar region masks in the new view. These segmentation masks are combined with the relative pose to produce their own homography transformations and combined to generate new views based on the warped input image.

[0081] Step 3: Analyze the motion state recognition results of step 1. When the user is in a non-stationary state, turn on the rear-view camera of the mobile phone to collect image data of the user's surrounding environment; for the image data of the user's surrounding environment, build a dangerous environment detection module based on the YOLOv5 lightweight network to identify the potential dangerous environment around the mobile phone user (zebra crossing, stair area, slippery road surface, low lighting); the specific content is:

[0082] First, build the YOLOv5s 6.0 lightweight target detection network under the pytorch framework. The network consists of four modules: Input module, Backbone network module, Neck network module and Head module. Figure 7 As shown;

[0083] The Input module includes three parts: Mosaic data enhancement, adaptive anchor frame calculation, and image size processing. The standard size of the input image is 640×640×3. After Mosaic data enhancement, any four images in the data set are randomly scaled, cropped, and arranged for splicing. Then, through adaptive frame calculation, the initial frames set by YOLOv5s are [70,28,74,100,88,213], [147,67,181,106,189,303], and [261,132,513,281,544,395]. When the network is trained, the output prediction frame is based on the initial frame, and then compared with the real frame groundtruth. The difference between the two is calculated and the reverse update is performed, thereby repeatedly iterating and updating the network parameters.

[0084] The Backbone network module consists of 1 Fcous structure, 5 Conv modules, 21 C3 modules, and 1 SPPF module. The Focus structure takes a 640×640×3 image as input, uses a slicing operation, and first converts it into a 320×320×12 feature map. Then, after a 3×3 convolution operation, it outputs 32 channels and finally becomes a 320×320×32 feature map. The Conv module includes convolution (Conv2d), Batch Normalization, and activation function (SiLU). The C3 module contains 3 standard convolutional layers and multiple Bottleneck modules. The SPPF module uses multiple small-sized pooling kernels in cascade to replace the single large-sized pooling kernel in the SPP module.

[0085] The Neck network module consists of a FPN+PAN network structure. The FPN structure from top to bottom processes the information transfer process through upsampling operations to obtain the predicted feature map. The feature pyramid from bottom to top is applied in the PAN structure.

[0086] The Head module consists of 4 Conv modules, 2 nn.Upsample modules, 4 Concat (concatenation layer) modules, 12 C3 modules, and 1 Detect module. It is mainly responsible for multi-scale target detection on the feature maps extracted by the backbone network.

[0087] Then, we made four kinds of image datasets: stage (stairs), wetroad (slippery road), zcrossing (zebra crossing), and night (night). The number of images is 2500, 1500, 1500, and 1500 respectively, and there are 1000 negative samples. The dataset is divided into training set, validation set, and test set in a ratio of 7:2:1. We manually labeled the training set of the four types of images with the labelImg tool, and then performed 200 rounds of training to generate the best.pt model. We then used the validation set and test set data to test the effect of the trained model.

[0088] Step 4: Based on the mobile phone user motion status recognition results and surrounding dangerous environment recognition results output from Step 1 and Step 3, a dangerous status judgment module for head-down users based on a decision tree model is constructed to classify the current dangerous status of head-down users. The specific contents are as follows:

[0089] First, determine the three features of the input data: whether the user is exercising, whether the user is playing with the phone, and whether the user is in a dangerous environment, and build a decision tree model based on these features.

[0090] Then, "whether the user is exercising" is selected as the root node of the decision tree. This feature can best classify whether the mobile phone user is in a dangerous state.

[0091] Finally, the Gini coefficients of the two features "whether the user is playing with the phone" and "whether the user is in a dangerous environment" are calculated to determine the internal nodes and leaf nodes of the decision tree model. Figure 8 The decision tree model shown. The calculation formula of Gini coefficient is as follows:

[0092]

[0093] Among them, p k Represents the probability of occurrence of a certain category.

[0094] Step 5: Design the mobile user interface and establish an auditory-visual three-dimensional warning method based on the real-time recognition results of the mobile phone user's motion status, the recognition results of the dangerous environment around the user, and the judgment results of the dangerous status of the mobile phone users. The specific contents are as follows:

[0095] The user interface is divided into two parts, upper and lower. Fig. 9The lower part detects the user's activity status in real time and displays the real-time data detected by the x, y, and z three-axis gravity acceleration sensors in six situations: holding a mobile phone and walking slowly, holding a mobile phone in a pocket and walking slowly, leaving the mobile phone still, walking slowly and looking at the mobile phone, walking up and down stairs and looking at the mobile phone, and standing still and looking at the mobile phone; the upper part of the interface displays the user's environment in real time and determines the user's dangerous state. When it is detected that the user is using the mobile phone in dangerous environments such as stairs, zebra crossings, at night, and on slippery roads, it provides early warnings to the head-down users in the form of text display and voice broadcast.

[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent substitution or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-modal mobile terminal-based dangerous environment perception and warning method for head-down people, characterized in that: The following steps are involved: Step 1: Use the built-in acceleration sensor of the mobile phone to collect three-directional acceleration time series data; extract acceleration state features from the three-directional acceleration time series data, and build a user motion state classification based on the parallel LSTM-CNN network. The user motion state classification includes holding the mobile phone and walking slowly, putting the mobile phone in the pocket and walking slowly, leaving the mobile phone still, walking slowly and looking at the mobile phone, going up and down the stairs and looking at the mobile phone, and standing still and looking at the mobile phone. The current motion state of the mobile phone user is identified; Step 2: Use a monocular camera to capture images and estimate the depth information of the monocular image; based on the depth information of the color image sequence and the depth map, predict the view extension image; Step 3: Analyze the motion state recognition results of step 1. When the user is in a non-stationary state, turn on the rear-view camera of the mobile phone to collect image data of the user's surrounding environment; based on the image data of the user's surrounding environment, build a dangerous environment detection module based on the YOLOv5 lightweight network to identify potential dangerous environments around the mobile phone user, including zebra crossings, stair areas, slippery roads, and low lighting; Step 4: Based on the mobile phone user motion state recognition results and surrounding dangerous environment recognition results output from Step 1 and Step 3, a dangerous state judgment module for the head-down users is constructed based on a decision tree model to classify the current dangerous state of the head-down users. Step 5: Design a mobile user interface based on the real-time recognition results of the mobile phone user's motion status, the recognition results of the dangerous environment around the user, and the judgment results of the dangerous status of the "phone addicts", and establish an auditory-visual three-dimensional warning method.

2. According to the mobile terminal-based multi-modal dangerous environment perception and warning method of claim 1, it is characterized in that: The step 1 is specifically as follows: Step 11: For the built-in gravity acceleration sensor of the mobile phone, collect acceleration time series data of 13 different users at a sampling frequency of 20 Hz in six states: holding the mobile phone and walking slowly, putting the mobile phone in the pocket and walking slowly, leaving the mobile phone still, walking slowly and looking at the mobile phone, walking up and down the stairs and looking at the mobile phone, and standing still and looking at the mobile phone. Collect 10 seconds of data in each state; Step 12: Label the collected data according to different users and states; a, b, c, d, e, f, g, h, i, j, k, l, and m are used to represent 13 different users, and 1, 2, 3, 4, 5, and 6 are used to represent six states: holding a mobile phone and walking slowly, putting a mobile phone in a pocket and walking slowly, leaving the mobile phone still, walking slowly and looking at the mobile phone, walking up and down stairs and looking at the mobile phone, and standing still and looking at the mobile phone. The specific expressions are: in, It represents the time series data of the x, y, and z axes acceleration collected by the gravity acceleration sensor when user a is walking slowly with a mobile phone in hand. X 200 ={x1, x2…x 200 },Y 200 ={y1, y2, ...y 200 }, Z 200 ={z1, z2, ... z 200 }; Step 13: Input data for x, y, and z axis acceleration and manually extract statistical features T imput , including the average value in each state: μ x μ y μ z , variance: σ x , σ y , σ z and the Pearson correlation coefficient between the two axes: ρ xy , ρ xz , ρ yz ; Among them, the calculation formula of Pearson correlation coefficient is as follows: Step 14: Superimpose the x, y, and z axis acceleration data in time to generate a signal image, then apply the two-dimensional discrete Fourier transform DFT to the signal image and select its amplitude as the active image F imput ; The formula for the two-dimensional discrete Fourier transform is as follows: Step 15: Collect raw data S from the gravity acceleration sensor input , statistic T imput , frequency image F imput For input data, a user state recognition network based on LSTM-CNN is constructed, data features are extracted through the DWConv deep convolution module, and the maximum pooling layer and downsampling layer are used to enhance the robustness of the model and reduce the risk of overfitting.

3. According to the multi-modal mobile terminal-based dangerous environment perception and warning method of claim 1, it is characterized in that: The step 2 is specifically as follows: Step 21: First, the depth of the monocular image is estimated using the decoder structure based on the Laplacian pyramid to accurately explain the relationship between the encoded features and the final output; the Laplacian residual of the input color image is L k , specifically expressed as: L k =I k -U p (I k+1 ),k=1,2,3,4 (4) Where k is the horizontal index in the Laplace pyramid, I k is obtained by downsampling the original input image, U p represents the upsampling function; The depth residual obtained at the kth pyramid level is R k , specifically expressed as: R k =B k ([x k ,L k ,U p (R k+1 )])+L k ,k=1,2,3,4 (5) Among them, [x k , L k , U p (R k+1 ) is in series, B k It consists of stacked convolutional blocks; Starting from the highest level of the Laplacian pyramid, the depth map is gradually reconstructed, which is specifically expressed as: D k =R k +U p (D k+1 ),k=1,2,3,4 (6); Step 22: Then, a standard fully convolutional architecture is used to predict the predicted pixel depth and normal map for a single image in two independent streams respectively; on each plane, given the required relative pose, i.e. 3D rotation and translation, and the intrinsic parameters of the camera, a homography can be calculated from its depth and normal to model m different planes and segment the input image into m regions; use the first 4 convolutional modules of VGG to apply the reverse warping process from the input image and the seed region mask to predict m planar region masks in the new view, combine these segmentation masks with the relative pose to produce their own homography transformations, and generate new views based on the warped input image combination.

4. According to the mobile terminal-based multi-modal dangerous environment perception and warning method of claim 1, it is characterized in that: The step 3 is specifically as follows: Step 31: First, build the YOLOv5s 6.0 lightweight target detection network under the pytorch framework. The network consists of four modules: Input module, Backbone network module, Neck network module and Head prediction module; Step 32: Then, make four kinds of image datasets: stage stairs, wetroad slippery road, zcrossing zebra crossing, and night, with 2500, 1500, 1500, and 1500 images respectively, and 1000 negative samples. Divide the dataset into training set, validation set, and test set in a ratio of 7:2:1; Step 33: Use the labelImg tool to manually label the training set of the four categories of images, then perform 200 rounds of training to generate the best.pt model, and verify the effectiveness of the trained model using the validation set and test set data.

5. According to the mobile terminal-based multi-modal dangerous environment perception and warning method for head-down people according to claim 4, it is characterized in that: The Input module includes three parts: Mosaic data enhancement, adaptive anchor frame calculation, and image size processing. The standard size of the input image is 640×640×3. After Mosaic data enhancement, any four images in the data set are randomly scaled, cropped, and arranged for splicing. Then, through adaptive frame calculation, the initial frames set by YOLOv5s are [70, 28, 74, 100, 88, 213], [147, 67, 181, 106, 189, 303], and [261, 132, 513, 281, 544, 395]. When the network is trained, the output prediction frame is based on the initial frame, and then compared with the real frame groundtruth, and the difference between the two is calculated. The reverse update is performed, thereby repeatedly iteratively updating the network parameters. The Backbone network backbone network module consists of 1 Fcous structure, 5 Conv modules, 21 C3 modules, and 1 SPPF module; the Focus structure inputs a 640×640×3 image, uses a slicing operation, first converts it into a 320×320×12 feature map, and then undergoes a 3×3 convolution operation, outputs 32 channels, and finally converts it into a 320×320×32 feature map; the Conv module includes convolution Conv2d, Batch Normalization, and activation function SiLU; the C3 module includes 3 standard convolution layers and multiple Bottleneck modules; The SPPF module uses multiple small-sized pooling cores in cascade to replace the single large-sized pooling core in the SPP module; The neck network module of the Neck network is composed of a network structure of FPN+PAN. The FPN structure from top to bottom processes the information transfer process through an upsampling operation to obtain a predicted feature map; the feature pyramid from bottom to top is applied in the PAN structure; The Head prediction module consists of 4 Conv modules, 2 nn.Upsample modules, 4 Concat splicing layer modules, 12 C3 modules, and 1 Detect module, which is mainly responsible for multi-scale target detection on the feature maps extracted by the backbone network.

6. According to the mobile terminal-based multi-modal dangerous environment perception and warning method of claim 1, it is characterized in that: The step 4 is specifically as follows: Step 41: First, determine three features of the input data: whether the user is exercising, whether the user is playing with a mobile phone, and whether the user is in a dangerous environment, and establish a decision tree model based on these features; Step 42: Then, "whether the user is exercising" is selected as the root node of the decision tree. This feature can classify whether the mobile phone user is in a dangerous state; Step 43: Finally, calculate the Gini coefficients of the two features "whether the user is playing with the phone" and "whether the user is in a dangerous environment" to determine the internal nodes and leaf nodes of the decision tree model; the calculation formula of the Gini coefficient is as follows: Among them, p k Represents the probability of occurrence of a certain category.

7. According to the mobile terminal-based multi-modal dangerous environment perception and warning method of claim 1, it is characterized in that: The step 5 is specifically as follows: The user interface is divided into two parts, upper and lower. The lower part detects the user's activity status in real time, and displays real-time data detected by the x, y, and z three-axis gravity acceleration sensors for six situations: walking slowly with the phone in hand, walking slowly with the phone in the pocket, leaving the phone still, walking slowly and looking at the phone, going up and down the stairs and looking at the phone, and standing still and looking at the phone. The upper part of the interface displays the user's environment in real time, and at the same time determines the user's dangerous state. When it is detected that the user is using the phone in dangerous environments such as stairs, zebra crossings, at night, or on slippery roads, the interface provides an early warning to the "head-down" users through text display and voice broadcast.

Images