Online car-hailing passenger position rapid positioning method based on voice interaction and visual perspective

Abstract

The invention provides an online car-hailing passenger position rapid positioning method based on voice interaction and visual perspective, and relates to the technical field of intelligent traffic, and the method comprises the following steps: arranging a sensor, a camera, a metal reflective curtain and a projector on a car; voice information of a driver is collected, and an intelligent voice wake-up model is designed; constructing a scenarized speech recognition model; detecting a pedestrian target for the panoramic image by using a sound-image multi-level cross-modal detection model, and obtaining appearance and posture feature detection information of the pedestrian target; a static feature matching method and a dynamic feature recognition model are designed, and passenger feature information matching is achieved; based on a matching result, determining position information of the passenger; acquiring a passenger direction taking the vehicle as a reference origin through position estimation in the panoramic image; according to the direction estimation result, a projector in the vehicle is controlled, the passenger information is projected on a curtain in the corresponding direction in the vehicle, and meanwhile voice information is used for giving a prompt.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a method for quickly locating the position of a passenger in an online car-hailing vehicle based on voice interaction and visual perspective. Background Art

[0002] With the rapid development and improvement of online car-hailing platforms, online car-hailing has become a common means of transportation for people. However, when taking an online car-hailing, due to inaccurate positioning or passengers' unfamiliarity with the surrounding situation, online car-hailing often encounters the phenomenon of "people looking for cars and cars looking for people" when picking up passengers, which not only wastes the time of drivers and passengers, but also affects the passage of vehicles, causing congestion on surrounding roads. In addition, large online car-hailing systems have multiple blind spots and cannot quickly locate passengers. The current system has obvious shortcomings in intelligent interaction, cannot respond to passenger needs in real time, and cannot effectively monitor potential traffic risks. So, in the increasingly congested traffic, how can drivers find their passengers in the vast sea of ​​people, how can they understand the situation outside the car more clearly, and reduce the probability of safety accidents. The current in-vehicle system cannot solve this type of problem well. Therefore, it is of great significance to develop a method that can realize intelligent monitoring and passenger service through voice interaction.

[0003] The common methods of locating passengers for online ride-hailing vehicles can be divided into two types: telephone communication positioning and GPS positioning. The defects of these methods are quite obvious: (1) Telephone communication positioning: The online ride-hailing driver communicates with the passenger by phone about the passenger's waiting position, but this method is not suitable for daily life because the passenger is not familiar with the surrounding situation and it is difficult to describe the specific location, or because the surrounding traffic is too large to distinguish. (2) GPS positioning: The online ride-hailing driver obtains the passenger's actual location information through the passenger's mobile phone positioning, but this method has inaccurate positioning information and unstable signals, making it difficult for the driver to determine the passenger's actual location, and difficult to apply in practice. Summary of the invention

[0004] The purpose of the present invention is to solve the technical problems existing in the online car-hailing passenger positioning method in the prior art.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for quickly locating the position of a passenger in an online car-hailing vehicle based on voice interaction and visual perspective comprises the following steps:

[0007] S1: Deploy sensors, cameras, metal reflective screens and projectors on the online car-hailing vehicle, wherein there are five cameras, which are respectively located at the right front door, right rear door, front roof, rear roof and rear of the vehicle; the metal reflective screens are located at the front door, rear door and rear seat backrest, and the projector is located obliquely above the reflective screens;

[0008] S2: Use the vehicle microphone to continuously collect the driver's voice information in the background, use Transformer as the voice wake-up word detection model and extract voice features to design an intelligent voice wake-up model to realize the "one-word wake-up" function of voice interaction;

[0009] S3: Build a scenario-based speech recognition model to automatically identify the appearance and movement features of passengers by recognizing the driver's speech information, and save the text information of the above features;

[0010] S4: Use the camera to obtain a panoramic image outside the vehicle, use the "sound-image" multi-level cross-modal detection model to detect pedestrian targets in the panoramic image, and obtain the appearance and posture feature detection information of pedestrian targets;

[0011] S5: Design a passenger multi-attribute "static feature" matching method and a passenger "dynamic feature" recognition model based on skeleton features. Through text feature matching, the passenger feature information based on speech recognition in S3 and the passenger feature information based on visual detection in S4 are matched.

[0012] S6: Based on the matching result of S5, determine the position information of the passenger; obtain the direction of the passenger with the vehicle as the reference origin through position estimation in the panoramic image;

[0013] S7: According to the direction estimation result of S6, the projector in the vehicle is controlled to project the passenger information onto the screen in the corresponding direction in the vehicle, and a prompt is given using voice information at the same time.

[0014] Preferably, the specific steps of S2 are as follows:

[0015] First, the spectrogram is extracted from the input audio and then divided into sub-blocks, denoted as X∈R T×F , where the time domain window t = {1, 2, ... T}, the frequency domain f = {1, 2, ... F}; the sub-block is first passed through a linear projection matrix W0∈R in the frequency domain f×d Map to a high-dimensional space d, and then add a learnable category embedding in the time domain order Used to learn the global information of the spectrogram, and finally add a learnable position matrix As shown in formula (1), the processed X1 is fed into the Transformer encoder;

[0016] X1=[X class ;XW1]+X pos #(1)

[0017] Then, the obtained embedded features are input into a sequential Transformer encoder consisting of L multi-head attention and multi-layer perceptron blocks; in the lth Transformer, Q = X is calculated respectively. I W Q , K = X l W K and V = X l W K ;in, d h is the dimension of each attention head; the calculation of self-attention is shown in formula (2):

[0018]

[0019] MSA by Matrix The calculation is as follows:

[0020] MSA(X l )=[SA1(X l );SA2(X l );…;SA k (X l );]W p #(3)

[0021] The output of the lth Transformer is:

[0022]

[0023] Finally, during streaming inference, a sliding window is used to cut the audio stream into sub-blocks for inference; the sliding window is a rectangular window that slides along the time axis with some overlap; Transformer detects each sub-block and scores it y i ; Once a y appears i >γ, indicating that the wake-up word is triggered, that is, the driver wakes up the system; the threshold γ∈(0,1).

[0024] Preferably, the specific steps of S3 are as follows:

[0025] First, the real-time input audio file can be a local file or a file recorded using the recording function, and the voice signal is segmented into frames through VAD;

[0026] Then, the Mel-frequency cepstral coefficient features are extracted from each frame, and a classification model is trained using a set of data frames with known speech and silence signal regions; the trained model will be used to classify unknown audio signals into spoken parts, unspoken parts, and silence parts;

[0027] The detected breakpoints, i.e. the start and end time of each pronunciation part, will be saved in a List collection; finally, this List value and the original voice file will be used as the input of speech recognition for subsequent recognition operations;

[0028] Then, the DFCNN acoustic model is used to convert the speech signal into the corresponding pinyin information. First, the obtained time domain audio signal is converted into a spectrogram, and the signal needs to be subjected to a short-time discrete Fourier transform, as shown in formula (6):

[0029]

[0030] The window length is represented by N, w(n) is the Hamming window function, f s represents the sampling frequency; as in formula (7), the spectrogram is generated by logarithmic amplitude operation;

[0031] S(k,t)=20×log 10 |X t (k)| #(7)

[0032] The generated spectrogram is used as the input of the model. The DFCNN model passes through a large number of convolutional layers, and uses a small 3×3 convolution kernel in each convolution. A pooling layer is added after multiple layers of convolution to complete the model construction.

[0033] Through the combination of a large number of convolutional layers and pooling layers, the entire sentence is modeled, and the output unit directly corresponds to the final recognition result;

[0034] Finally, the Transformer language model converts the pinyin into the corresponding text content, and the pinyin sequence output by the DFCNN acoustic model is used as input. It first passes through the text embedding layer to convert the digital representation of the words in the text into a vector representation, and captures the relationship between words through a high-dimensional space; then passes through the position encoding layer, and adds the position encoding to the word embedding vector. The results of the two layers are added as the input of the encoder;

[0035] The encoder consists of 6 encoding units, each of which has a multi-head attention mechanism, a residual connection layer, a normalization layer, and a feedforward neural network layer; the multi-head attention mechanism is expressed as:

[0036]

[0037] Among them, Q, K, and V are query vectors, key vectors, and value vectors, respectively, which are composed of the input vector and their respective weights W Q , W K , W V The weights are obtained by training the model after initialization. dk is the dimension of the K vector.

[0038]

[0039] MH(Q,K,V)=Concat(head1,...,head n )W Q #(10)

[0040] Among them, MH stands for MultiHead,

[0041] The residual connection layer transfers the gradient by adding the input directly to the output of the intermediate layer of the network;

[0042] The normalization layer adopts a batch normalization algorithm;

[0043] The encoder sublayer also includes a feed-forward neural network layer, which can be expressed as:

[0044] FFN(x)=max(0,xW1+b1)W2+b2 #(11)

[0045] Finally, the text with punctuation is saved to complete the speech-to-text recognition.

[0046] Preferably, the specific steps of S4 are as follows:

[0047] First, we build a YOLOv8n target detection network under the pytorch framework. The YOLOv8n target detection network consists of five modules: Backbone module, Neck module, Detection Head module, AnchorBoxes module, and LossFunction module.

[0048] Then, we made our own feature detection dataset. We specially added data of passengers with different color depths and different body shapes to the dataset. The dataset was divided into training set, validation set and test set in a ratio of 7:2:1, and 2,000 negative samples were added.

[0049] Next, use the labelimg tool to manually label the training set, and then perform 100 rounds of training to generate the best.pt model. The model effect generated by the training is verified by the validation set and test set data.

[0050] Finally, the “sound-image” multi-level cross-modal detection model is used to obtain the target ID information, coordinate information, and target key point information.

[0051] Preferably, the Backbone module in the YOLOv8n target detection network is composed of a convolution layer, an activation function and a pooling layer, and extracts feature representation from the input image through multi-scale feature extraction and feature fusion to achieve detection of targets of different scales; its working process includes convolution feature extraction, nonlinearity introduced by activation function, pooling dimensionality reduction, and parameter sharing, and ultimately provides necessary semantic understanding and information expression for subsequent target detection tasks;

[0052] The Neck module processes features of different scales through a feature pyramid structure, and improves the expressiveness of features through multi-scale feature fusion, semantic enhancement, etc., so as to accurately detect objects of different scales. The Neck module also realizes cross-scale feature alignment through downsampling and upsampling operations of the feature pyramid; finally, the Neck module captures more local information through a fine-grained feature fusion mechanism;

[0053] The Detection Head module predicts the location and category of the target box through feature map processing and Anchor Boxes module definition, and outputs the target detection result after screening and correction, thereby achieving accurate detection and positioning of the target;

[0054] The AnchorBoxes module is a series of predefined bounding boxes distributed at different positions and scales of the feature map, which are used to generate candidate boxes in the image. These AnchorBoxes cover various possible target shapes and scales by defining different aspect ratios and sizes. During the target detection process, each Anchor Box module corresponds to the pixel position on the feature map, and the model predicts the position and category information of the possible targets in each AnchorBox module. By comparing with the position of the actual target box and combining the category probability for screening and correction, accurate target detection results are finally obtained.

[0055] The Loss Function module includes the coordinate loss of the target box position, the classification loss of the target category, and the matching loss between the target box and the AnchorBoxes; the working process of the loss function involves calculating the difference between the predicted target box and the true target box, and weighting it in combination with the category probability to minimize the error between the predicted box and the true box; in addition, the loss function also considers the weights of positive and negative samples and the confidence threshold to ensure effective training and adjustment of samples of different difficulties; by minimizing the overall loss function, the model can more accurately predict the position and category of the target.

[0056] Preferably, the multi-level cross-modal detection model of "sound-image" in S4 comprises the following steps:

[0057] First, for the text of speech recognition in S3, the noun phrases are obtained by n-gram algorithm, and then the text embedding is extracted using the Transformer text encoder Text Encoder based on CLIP pre-training. Where T is the given text, C is the number of nouns, and D is the embedding dimension;

[0058] Next, multiple cameras at different locations on the vehicle are used to simultaneously obtain multiple real-time video streams and pass the video streams through the YOLOv8 Backbone to obtain multi-scale image features.

[0059] Then, W and X obtained in the above steps are l Feature fusion is performed through a re-parameterized visual-language path aggregation network, where the fusion process mainly includes a text-guided cross-block bias layer from text to image and an image pooling attention mechanism from image to text;

[0060] Finally, the fused target encoding is output through the Text Contrastive Head, and a decoupled head with two 3×3 convolutions is used to regress the bounding box. and object embedding And by introducing the text comparison center, the object-text similarity is obtained, so as to output the coordinates of the object with the greatest similarity; the target code e and the text code w are represented by L2-Norm;

[0061]

[0062] Where L2-Norm() represents L2 normalization, W j ∈W is the jth text embedding, α is the scaling factor, and β is the offset factor.

[0063] Preferably, the n-gram algorithm comprises the following steps: performing a sliding window operation of size N on the content of the text according to bytes to form a sequence of byte segments of length N; each byte segment is called a gram, and the occurrence frequencies of all the grams are counted, filtered according to a preset threshold, and a key gram list is formed, thereby realizing the extraction of noun phrases;

[0064] The Transformer text encoder Text Encoder based on CLIP pre-training consists of two parts, namely, a vision transformer responsible for images and a transformer responsible for texts;

[0065] In the vision transformer, the image first passes through a patch convolution and a continuous residual attention structure, and the output is a tensor of (1, 50, 768). Finally, the mapping I1, I2, ..., IN to (1, 512) is completed through matrix multiplication.

[0066] In the Transformer, the text is tokenized to get a new tensor, and then passes through the embedding layer to get a (3, 77, 512) tensor. After adding the positional encoding (nn.Parameter), it is fed into the model. Finally, the model outputs the (1, 512) features T1, T2, ..., T N ;

[0067] By calculating I i and T j The cosine similarity between them is used to measure the correspondence between the corresponding text and the image. The larger the cosine similarity, the stronger the correspondence.

[0068]

[0069] The visual-language path aggregation network, for images, multi-scale features X l Through downsampling and upsampling operations, combined with the embedding set W of the detection category vocabulary, T-CSPLayer is used to allocate attention. For text, the embedding set W of the detection category vocabulary is combined with 3 multi-scale feature maps for max pooling to perform 3x3 area block division, and 27 feature maps are obtained. Based on multiHead attention, I-Pooling Attention is further used to allocate attention;

[0070] Among them, T-CSPLayer is used to analyze the multi-scale image features X l Use the embedding W of the detection category vocabulary to allocate attention and obtain the multi-scale feature output X' l as follows:

[0071]

[0072] Where δ represents the sigmoid function.

[0073] Among them, I-Pooling Attention uses multi-scale image features X to embed the word W of the detection category. l By allocating attention, the feature output W' is obtained as follows:

[0074]

[0075] Preferably, in S5, a passenger multi-attribute "static feature" matching method is innovated for different appearance features of different passengers. First, the target information features obtained by S4 include the id value and its corresponding two-point coordinates, combined with the passenger feature text information input by the driver's voice, first construct a target label set through the input text information, loop read the target detection processing results of YOLOv8 in the video stream, and match the identified labels with the target labels, and find all possible label combinations through permutations and combinations:

[0076]

[0077] Enlarge the coordinates of the target box, and then calculate the intersection area of ​​every two bounding boxes;

[0078] interArea=max(0,min(xmaxA,xmaxB)-max(xminA,xminB))×max(0,min(ymaxA,ymaxB)-max(yminA,yminB)) #(17)

[0079] Among them, min(xmaxA,xmaxB), max(xminA,xminB), min(ymaxA,ymaxB), max(yminA,yminB) are the coordinates of the intersection of the two boxes, and interArea is the area of ​​the intersection.

[0080] Then, the corresponding IOU value is calculated. If the IOU value is greater than the set threshold of 0.3, it means that they are the same person. Then the characteristic passenger is found and the actual position of the characteristic passenger is announced according to the position of the average center point.

[0081] Thus, the passenger feature information based on voice recognition in S3 is matched with the passenger feature information based on visual detection in S4, and the feature passenger is found;

[0082]

[0083] Among them, interArea is the area of ​​the intersection area, and unionArea is the area of ​​the union area.

[0084] Next, the acquired data is filtered through the coordinates of the key points of the human body obtained by S4, and then the numpy array of the coordinates of all the key points is obtained, that is, the (x, y) coordinate information corresponding to each key point of the human body;

[0085] In view of the different gestures of passengers waving to taxis in real life and the possible false triggering of pedestrians, a passenger "dynamic feature" recognition model based on bone features is established, and strict authentication indicators for hand-raising gestures are established - the wrist key point is higher than the shoulder key point and the height between the wrist, elbow and shoulder on the same side is within a certain threshold;

[0086] Set up the angle_between_points function to calculate the angles between key points. Use the key point coordinates obtained in the above process - shoulder point[5], point[6] elbow point[7], point[8] wrist point[9], point

[10] , pass them into the angle_between_points function, and calculate the angles between the three points on the left and right sides respectively.

[0087] The coordinates of the three points on the left and right are marked as [x1, y1], [x2, y2], [x3, y3]; then calculate the AB vector and the BC vector, which are AB = (x2-x1, y2-y1) and BC = (x3-x2, y3-y2) respectively; then use the dot product and length formula of the vectors to calculate the angle between the two vectors,

[0088]

[0089] AB·BC=x1x2+y1y2 #(21)

[0090] Then we can get the cosine of the angle

[0091]

[0092]

[0093] Finally, if the angles between the three points are between 90° and 160° and the wrist y value on the same side (wrist point[9], point

[10] shoulder point[5], point[6]) is greater than the shoulder y value, it is judged as a potential hand-raising action.

[0094] Preferably, in said S6, the average center coordinates of the judged passenger characteristics obtained in S5 are used to establish a passenger position prediction scheme centered on the vehicle, and the actual position of the passenger from the driver centered on the vehicle is determined by calculating the position of the average center point of the passengers;

[0095] The specific location of the passenger is obtained based on the judgment result, and the pyttsx3 module is used for text-to-speech function to make a voice announcement to inform the driver that he should slow down, that a potential passenger has appeared, and to inform the driver of his specific location;

[0096] Determine the position of the area to be captured according to the coordinates of the key points, and capture it in the original image according to the determined area range, obtain the regional image where the key part is located, and use the cv2.resize function to scale it to the appropriate size, and flexibly adjust the placement of the screen according to the actual position, and in the video stream processing loop, update the small screen content according to the real-time recognition results, and draw the updated small screen on the original image to achieve dynamic display.

[0097] Preferably, in S7:

[0098] When the passenger is not located, the image containing the visual blind spot is projected to achieve the perspective effect of the blind spot. The image content is updated in real time in the video stream processing loop to achieve dynamic display. At the same time, the specific location of the passenger is determined based on the judgment result, and the pyttsx3 module is used for text-to-speech function to make a voice broadcast to inform the driver that he should slow down, the presence of characteristic passengers and potential passengers, and inform the driver of their specific location.

[0099] When executing system functions, the vehicle-mounted camera and projector are combined with the system to achieve a visual perspective. Multiple vehicle-mounted cameras are placed next to the door to collect 360° image data, and the visually processed results are transmitted to the projector using OpenCV. The projector is placed on the opposite side of the projection cloth, and the image is projected onto the cloth close to the inside of the car body. It can display multiple features of passengers specially marked by OpenCV, and use small screens placed at different positions of the image to specifically display different features, conduct visual warnings, and realize rapid positioning of passengers.

[0100] The above-mentioned method for quickly locating the position of online car-hailing passengers based on voice interaction and visual perspective uses multiple on-board cameras to perceive the scene outside the car to achieve car perspective, avoid accidents in blind spots and collect data in real time. The system is awakened by the "one word wake-up" function, and a scenario-based voice recognition model is built to recognize the driver's voice information. Then the data is processed using the "sound-image" multi-level cross-modal detection method, the passenger dynamic feature recognition model based on skeletal features, and the passenger multi-attribute "static feature" matching method. Finally, the results are fed back to the projector for display at the door and voice broadcast. The system can realize driver voice operation, reduce the phenomenon of "people looking for cars, cars looking for people" in online car-hailing, and alleviate congestion problems.

[0101] Compared with the prior art, this application has the following beneficial effects:

[0102] (1) A multi-level cross-modal target recognition model based on voice and image is proposed to achieve rapid detection of target passengers based on voice interaction and image recognition;

[0103] (2) The combination of the external camera and the internal projector enables the visualization of blind spot passenger positioning based on visual perspective;

[0104] (3) Use voice to indicate the relative position of passengers and assist the driver in quickly locating them. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] Figure 1 A schematic diagram of the steps of a method for quickly locating the position of a passenger of an online car-hailing vehicle based on voice interaction and visual perspective in one embodiment of the present invention;

[0106] Figure 2 It is a general framework diagram of a method in one embodiment of the present invention;

[0107] Figure 3 This is a schematic diagram of a real scene of a vehicle-mounted camera in one embodiment of the present invention;

[0108] Figure 4 This is a schematic diagram of a real scene of the screen setting in one embodiment of the present invention;

[0109] Figure 5 A schematic diagram of a real scene of a projector setting in one embodiment of the present invention;

[0110] Figure 6 This is a diagram of a voice wake-up network structure in one embodiment of the present invention;

[0111] Figure 7 A diagram showing the generation process of a spectrogram in one embodiment of the present invention;

[0112] Figure 8 This is a DFCNN network structure diagram in one embodiment of the present invention;

[0113] Fig. 9 A YOLOv8 image recognition database in one embodiment of the present invention;

[0114] Fig.10 This is a network structure diagram of a multi-level cross-modal detection model of "sound-image" in one embodiment of the present invention;

[0115] Fig.11 A diagram showing a re-parameterized VL-PAN structure in one embodiment of the present invention;

[0116] Fig.12 This is a passenger multi-attribute "static feature" matching algorithm diagram in one embodiment of the present invention;

[0117] Fig.13 This is a diagram of a passenger dynamic feature recognition algorithm based on skeleton features in one embodiment of the present invention;

[0118] Fig.14 It is a perspective schematic diagram of a specific embodiment of the present invention;

[0119] Fig.15This is a real scene test diagram in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0120] The present invention is further described in detail below in conjunction with specific embodiments.

[0121] See also Figure 1 , a method for quickly locating the position of online car-hailing passengers based on voice interaction and visual perspective, the framework structure of the method is as follows Figure 2 As shown, the following steps are included:

[0122] S1: Deploy sensors, cameras, metal reflective screens and projectors on cars;

[0123] Five cameras are installed in the blind spots of the vehicle, metal reflective screens are laid on the front doors, rear doors and rear seat backrests, and three projectors are installed diagonally above the screens;

[0124] In one embodiment, the specific steps of S1 are as follows:

[0125] S1-1: Please refer to Figure 3 , sensors are placed on the car, and five cameras are installed in the blind spots of the vehicle. The five cameras are located at the right front door, right rear door, front roof, rear roof and rear of the car. Among them, the front roof and rear roof are two wide-angle cameras for detecting target passengers; the other three are high-definition cameras for viewing the surrounding environment.

[0126] S1-2: Please refer to Figure 4 Metal reflective curtains are laid in the blind spots of the driver's field of vision, and the reflective curtains are respectively located at the front door, rear door and rear seat backrest.

[0127] S1-3: Please refer to Figure 5 , install the projector obliquely above the reflective screen to ensure the consistency of the projected image with the real external environment to the greatest extent;

[0128] When the target passenger is detected in the panoramic image, the detected target is cut and projected onto the screen in the corresponding direction inside the car. By selecting and controlling the projector to achieve perspective, the problem of the target passenger appearing in the blind spot of the vehicle can be solved. In addition, the projection image of the blind spot helps the driver observe the surrounding environment and reduce the occurrence of collision accidents.

[0129] S2: Use the vehicle microphone to continuously collect the driver's voice information in the background, use Transformer as the voice wake-up word detection model and extract voice features to design an intelligent voice wake-up model to realize the "one-word wake-up" function of voice interaction;

[0130] For details, please refer to Figure 6In one embodiment, a background sub-thread is started, audio is continuously recorded in the background, and the audio is passed to the model.

[0131] First, the spectrogram is extracted from the input audio and then divided into sub-blocks, denoted as X∈R T×F , where the time domain window t = {1, 2, ... T}, the frequency domain f = {1, 2, ... F}. The sub-block is first passed through a linear projection matrix W0∈R in the frequency domain f×d Map to a high-dimensional space d, and then add a learnable category embedding in the time domain order Used to learn the global information of the spectrogram, and finally add a learnable position matrix As shown in formula (1), the processed X1 is fed into the Transformer encoder.

[0132] X1=[X class ;XW1]+X pos #(1)

[0133] Then, the obtained embedded features are fed into a sequential Transformer encoder consisting of L multi-head attention (MSA) and multi-layer perceptron (MLP) blocks. In the lth Transformer, Q = X is calculated respectively. I W Q , K = X l W K and V = X l W K .in, d h is the dimension of each attention head. The calculation of self-attention is shown in formula (2),

[0134]

[0135] MSA by Matrix Calculated, such as,

[0136] MSA(X l )=[SA1(X l );SA2(X l );…;SA k (X l );]W p #(3)

[0137] The output of the lth Transformer is:

[0138]

[0139]

[0140] Finally, during streaming inference, a sliding window is used to cut the audio stream into sub-blocks for inference. The sliding window is a rectangular window that slides along the time axis with some overlap. Transformer detects each sub-block and scores it y i Once a y appears i >γ, indicating that the wake-up word is triggered, that is, the driver wakes up the system. Threshold γ∈(0,1).

[0141] S3: Build a scenario-based speech recognition model to automatically identify the appearance and movement features of passengers by recognizing the driver's speech information, and save the text information of the above features;

[0142] In one embodiment, first, the audio file input in real time can be selected as a local file or a file recorded using a recording function, and the voice signal is first segmented in frames through endpoint detection (VAD). Then, the Mel frequency cepstral coefficient feature is extracted from each frame, and a classification model is trained using a data frame set of known voice and silent signal areas. The trained model will be used to classify unknown audio signals, dividing them into a pronunciation part, an unpronounced part, and a silent part. The breakpoints detected, i.e., the start and end time of each pronunciation part, will be saved in a List collection. Finally, this List value and the original voice file are used as the input of speech recognition for subsequent recognition operations.

[0143] Then, the DFCNN acoustic model is used to convert the speech signal into the corresponding pinyin information. First, the obtained time domain audio signal is converted into a spectrogram, and the signal needs to be subjected to a short-time discrete Fourier transform, as shown in formula (6):

[0144]

[0145] The window length is represented by N, w(n) is the window function, and the Hamming window is used in this paper. s represents the sampling frequency. The spectrogram is generated by logarithmic amplitude calculation as shown in formula (7).

[0146] S(k,t)=20×log 10 |X t (k)| #(7)

[0147] The generated spectrogram (the spectrogram generation process is as follows Figure 7As shown in the figure, the DFCNN model takes the 3×3 small convolution kernel as the input of the model. The DFCNN model passes through a large number of convolution layers, and uses a 3×3 small convolution kernel in each convolution. A pooling layer is added after multiple layers of convolution to complete the construction of the model. Through the combination of a large number of convolution layers and pooling layers, the entire sentence of speech is modeled, and the output unit directly corresponds to the final recognition result. The advantage of using the spectrogram as the input is that it avoids the obvious loss of high-frequency parts caused by artificially designed filters during feature extraction in traditional speech recognition. By inputting the model in the form of a spectrogram, information in both the frequency domain and the time domain is preserved. Due to the accumulation of a large number of convolution layers and pooling layers in this model, its control over future and historical information has also been improved, making the model more relevant when processing speech. The DFCNN network structure is shown in the figure. Figure 8 shown.

[0148] Finally, the Transformer language model is used to convert the pinyin into corresponding text content. First, the pinyin sequence output by the DFCNN acoustic model is used as input, and then passes through the text embedding layer to convert the digital representation of the words in the text into a vector representation, capturing the relationship between words through high-dimensional space; then passes through the position encoding layer, the different positions of words may produce different semantic information, and the position encoding is added to the word embedding vector to make up for the lack of position information, and the results of the two layers are added as the input of the encoder.

[0149] The encoder consists of 6 encoding units, each of which has a multi-head attention mechanism, a residual connection layer, a normalization layer, and a feedforward neural network layer. The multi-head attention mechanism is expressed as:

[0150]

[0151] Among them, Q, K, and V are query vectors, key vectors, and value vectors, respectively, which are composed of the input vector and their respective weights W Q , W K , W V The weights are obtained by training the model after initialization. dk is the dimension of the K vector.

[0152]

[0153] MH(Q,K,V)=Concat(head1,...,head n )W Q #(10)

[0154] Among them, MH stands for MultiHead,

[0155] The residual connection layer transfers the gradient by adding the input directly to the output of the middle layer of the network, thereby alleviating the problem of the gradient gradually decreasing in the deep network. The normalization layer uses the batch normalization (BN) algorithm to normalize each batch of data in each layer so that the input data of each layer has zero mean and unit variance, which helps to avoid the problem of data distribution offset and gradient explosion during training. The encoder sublayer also contains a feedforward neural network layer. The self-attention mechanism may be insufficient in fitting complex processes, so two layers of linear networks are added to improve the expressiveness of the model. This layer of network can be expressed as:

[0156] FFN(x)=max(0,xW1+b1)W2+b2 #(11)

[0157] Finally, the text with punctuation is saved to complete the speech-to-text recognition.

[0158] S4: Use the 360-degree perspective camera outside the vehicle (composed of two wide-angle cameras and three high-definition cameras in S1) to obtain a panoramic image outside the vehicle, use the "sound-image" multi-level cross-modal detection model to detect pedestrian targets in the panoramic image, and obtain the appearance and posture feature detection information of pedestrian targets;

[0159] Specifically, in one embodiment, a YOLOv8n target detection network under a pytorch framework is first constructed, and the YOLOv8n target detection network consists of five modules: a Backbone module, a Neck module, a Detection Head module, an AnchorBoxes module, and a Loss Function module;

[0160] Among them, the Backbone module in the YOLOv8n target detection network is an important part of the target detection model. It is composed of convolutional layers, activation functions and pooling layers. Through multi-scale feature extraction and feature fusion, feature representation is extracted from the input image to achieve detection of targets of different scales. Its working process includes steps such as convolutional feature extraction, nonlinearity introduced by activation functions, pooling dimensionality reduction, parameter sharing, etc., which ultimately provides the necessary semantic understanding and information expression for subsequent target detection tasks.

[0161] The Neck module processes features of different scales through a feature pyramid structure, and improves the expressiveness of features through multi-scale feature fusion and semantic enhancement to achieve accurate detection of objects of different scales. Neck also achieves cross-scale feature alignment through downsampling and upsampling operations of the feature pyramid. Finally, Neck captures more local information through a fine-grained feature fusion mechanism.

[0162] The Detection Head module predicts the location and category of the target box through feature map processing and AnchorBoxes definition, and outputs the target detection result after screening and correction, thereby achieving accurate detection and positioning of the target.

[0163] The Anchor Boxes module is a series of predefined bounding boxes distributed at different positions and scales in the feature map, which are used to generate candidate boxes in the image. These AnchorBoxes cover various possible target shapes and scales by defining different aspect ratios and sizes. During the target detection process, each AnchorBox will correspond to the pixel position on the feature map, and the model will predict the position and category information of the possible targets in each AnchorBox. By comparing with the position of the actual target box, and combining the category probability for screening and correction, accurate target detection results are finally obtained.

[0164] The Loss Function module consists of multiple parts, including the coordinate loss of the target box location, the classification loss of the target category, and the matching loss between the target box and AnchorBoxes. The working process of the loss function involves calculating the difference between the predicted target box and the true target box, and weighting it with the category probability to minimize the error between the predicted box and the true box. In addition, the loss function also takes into account the weights of positive and negative samples and the confidence threshold to ensure effective training and adjustment of samples of different difficulties. By minimizing the overall loss function, the model can more accurately predict the location and category of the target.

[0165] Then, we made our own feature detection dataset, which has 21 types of labels and nearly 20,000 labels in total. Considering that the characteristics of passengers in real life may vary greatly, we added passenger data of different color depths and different body shapes to enhance the robustness of the model in actual use, so that it can better cope with passenger recognition tasks in various situations. The dataset is divided into training set, validation set and test set in a ratio of 7:2:1, and 2,000 negative samples are added.

[0166] Next, the labelimg tool is used to manually label the training set, and then 100 rounds of training are performed to generate the best.pt model. The training model effect is verified by the validation set and test set data. Fig. 9 shown.

[0167] Finally, the “sound-image” multi-level cross-modal detection model is used to obtain the target ID information, coordinate information, and target key point information.

[0168] In one embodiment, see Fig.10 The multi-level cross-modal detection model of "sound-image" includes the following steps:

[0169] First, for the text of speech recognition in S3, the noun phrases are obtained by n-gram algorithm, and then the text embedding is extracted using the Transformer text encoder Text Encoder based on CLIP pre-training. Where T is the given text, C is the number of nouns, and D is the embedding dimension.

[0170] Next, multiple cameras at different locations on the vehicle are used to simultaneously obtain multiple real-time video streams and pass the video streams through the YOLOv8 Backbone to obtain multi-scale image features.

[0171] Then, W and X obtained in the above steps are l Feature fusion is performed through a re-parameterizable Vision-Language Path Aggregation Network (RepVL-PAN), where the fusion process mainly includes text-guided cross stage partial layers (T-CSPLayer) from text to image and image pooling attention mechanism (I-Pooling Attention) from image to text.

[0172] Finally, the fused target encoding is output through the Text Contrastive Head, and a decoupled head with two 3×3 convolutions is used to regress the bounding box. and object embedding By introducing the text comparison center, the object-text similarity is obtained, and the coordinates of the object with the greatest similarity are output. The target encoding e and the text encoding w are represented by L2-Norm.

[0173]

[0174] Where L2-Norm() represents L2 normalization, W j ∈W is the jth text embedding, α is the scaling factor, and β is the offset factor.

[0175] The n-gram algorithm includes the following steps: the content in the text is operated with a sliding window of size N according to bytes to form a sequence of byte segments of length N. Each byte segment is called a gram, and the occurrence frequency of all the grams is counted, filtered according to a pre-set threshold, and a key gram list is formed, thereby realizing the extraction of noun phrases;

[0176] The Transformer text encoder Text Encoder based on CLIP pre-training consists of two parts, namely, a transformer responsible for image vision transformer and a transformer responsible for text;

[0177] In the vision transformer, the image first passes through a patch convolution and a continuous residual attention structure, and the output is a (1,50,768) tensor, and finally the mapping I1,I2,…,IN to (1,512) is completed through matrix multiplication.

[0178] In the Transformer, the text is tokenized to get a new tensor, and then passes through the embedding layer to get a (3, 77, 512) tensor. After adding the positional encoding (nn.Parameter), it is fed into the model. Finally, the model outputs the (1, 512) features T1, T2, ..., T N .

[0179] By calculating I i and T j The cosine similarity between the two is used to measure the correspondence between the corresponding text and the image. The larger the cosine similarity, the stronger the correspondence.

[0180]

[0181] The visual-language path convergence network described in Fig.11 , for an image, the multi-scale feature X l Through downsampling and upsampling operations, combined with the embedding set W of the detection category vocabulary, T-CSPLayer is used to allocate attention. For text, the embedding set W of the detection category vocabulary is combined with 3 multi-scale feature maps for max pooling to perform 3x3 area block division, and 27 feature map patches are obtained. Based on multiHead attention, I-Pooling Attention is further used to allocate attention.

[0182] Among them, T-CSPLayer is used to analyze the multi-scale image features X l Use the embeddingW of the detection category vocabulary to allocate attention and obtain the multi-scale feature output X' l as follows:

[0183]

[0184] Where δ represents the sigmoid function.

[0185] Among them, I-PoolingAttention uses multi-scale image features X to detect the embedding W of category vocabulary l By allocating attention, the feature output W' is obtained as follows:

[0186]

[0187] S5: Design a passenger multi-attribute "static feature" matching method and a passenger "dynamic feature" recognition model based on skeleton features. Through text feature matching, the passenger feature information based on speech recognition in S3 and the passenger feature information based on visual detection in S4 are matched.

[0188] In one embodiment, first, for the passenger multi-attribute "static feature" matching method, please refer to Fig.12 The target information features obtained by S4 include the id value and its corresponding two-point coordinates. Combined with the passenger feature text information input by the driver’s voice, the target label set is first constructed through the input text information, the target detection processing results of YOLOv8 in the video stream are read cyclically, and the identified labels are matched with the target labels. All possible label combinations are found through permutations and combinations.

[0189]

[0190] However, since the combination may come from different pedestrians, it is necessary to determine whether each possible combination belongs to the same object - enlarge the coordinates of the target box (actual tests have determined that the appearance features are enlarged by 2 times and the building features are enlarged by 1.5 times), and then calculate the area of ​​the intersection of every two bounding boxes.

[0191] interArea=max(0,min(xmaxA,xmaxB)-max(xminA,xminB))×max(0,min(ymaxA,ymaxB)-max(yminA,yminB)) #(17)

[0192] in

[0193] min(xmaxA,xmaxB), max(xminA,xminB), min(ymaxA,ymaxB),

[0194] max(yminA,yminB) is the coordinate of the intersection of the two boxes, and interArea is the area of ​​the intersection.

[0195] Then, the corresponding IOU value is calculated. If the IOU value is greater than the set threshold of 0.3, it means that they are the same person. Then the characteristic passenger is found, and the actual position of the characteristic passenger is announced by voice according to the position of the average center point. In this way, the passenger characteristic information based on voice recognition in S3 and the passenger characteristic information based on visual detection in S4 are matched to find the characteristic passenger.

[0196]

[0197] Among them, interArea is the area of ​​the intersection area, and unionArea is the area of ​​the union area.

[0198] Next, for the passenger "dynamic feature" recognition model based on bone features, please refer to Fig.13 , through the coordinates of the key points of the human body obtained by S4, the acquired data is filtered to obtain the numpy array of the coordinates of all key points, that is, the (x, y) coordinate information corresponding to each key point of the human body.

[0199] Among them, in view of the different waving postures of passengers to taxis in real life and the possible false triggering phenomenon of pedestrians (non-potential passengers), a passenger "dynamic feature" recognition model based on skeletal features is established, and strict authentication indicators for hand-raising posture are established - the wrist key point is higher than the shoulder key point and the height between the wrist, elbow and shoulder on the same side is within a certain threshold.

[0200] Set up the angle_between_points function to calculate the angles between key points. Use the key point coordinates obtained in the above process - shoulder point[5], point[6] elbow point[7], point[8] wrist point[9], point

[10] , pass them into the angle_between_points function, and calculate the angles between the three points on the left and right sides respectively.

[0201] The coordinates of the three points on the left and right are marked as [x1, y1], [x2, y2], [x3, y3]. Then calculate the AB vector and the BC vector, AB = (x2-x1, y2-y1) and BC = (x3-x2, y3-y2). Then use the dot product and length formula of the vectors to calculate the angle between the two vectors.

[0202]

[0203] AB·BC=x1x2+y1y2 #(21)

[0204] Then we can get the cosine of the angle

[0205]

[0206]

[0207] Finally, if the angles between the three points are between 90° and 160° and the wrist y value on the same side (wrist point[9], point

[10] shoulder point[5], point[6]) is greater than the shoulder y value, it is considered a potential hand-waving action. Hand-waving action detection can be used to identify characteristic passengers as well as potential passengers.

[0208] S6: Based on the matching result of S5, determine the position information of the passenger; obtain the direction of the passenger with the vehicle as the reference origin through position estimation in the panoramic image;

[0209] First, the average center coordinates of the judged passenger characteristics are obtained through S5, and a passenger position prediction scheme centered on the vehicle is established. The actual position of the passenger from the driver centered on the vehicle is determined by calculating the position of the average center point of the passengers.

[0210] Then, the specific location of the passenger is obtained based on the judgment result, and the pyttsx3 module is used for text-to-speech function to make a voice announcement to inform the driver that he should slow down, that a potential passenger has appeared, and to inform the driver of his specific location.

[0211] Finally, based on the judgment results, determine the range of the area to be intercepted - determine the location of the area to be intercepted based on the coordinates of the key points, and intercept it in the original image based on the determined area range, obtain the area image where the key part is located, and use the cv2.resize function to scale it to the appropriate size, and flexibly adjust the placement of the screen based on the actual position, and in the video stream processing loop, update the small screen content based on the real-time recognition results, draw the updated small screen on the original image, and realize dynamic display, which is convenient for the driver to promptly confirm whether it is a characteristic passenger.

[0212] S7: According to the direction estimation result of S6, the projector in the vehicle is controlled to project the passenger information onto the screen in the corresponding direction in the vehicle, and a prompt is given using voice information at the same time.

[0213] Specifically, when the passenger is not located, the image containing the visual blind spot is projected to achieve the perspective effect of the blind spot, which is beneficial for the driver to observe the surroundings when there is a large flow of people and reduce the occurrence of collision accidents. The image content is updated in real time in the video stream processing loop to achieve dynamic display. At the same time, the specific location of the passenger is determined based on the judgment result, and the pyttsx3 module is used for text-to-speech function to make a voice broadcast to inform the driver that he should slow down, the presence of characteristic passengers and potential passengers, and inform the driver of their specific location.

[0214] Finally, when executing the system function, the vehicle-mounted camera and projector are combined with the system to achieve visual perspective. Use multiple vehicle-mounted cameras placed next to the car door to collect 360° image data, and use OpenCV to pass the visual processing results to the projector. The projector is placed on the opposite side of the projection cloth, and the image is projected onto the screen close to the inside of the car body. It can display multiple features of passengers specially marked by OpenCV, and use small screens placed at different positions of the image to specifically display different features, perform visual warnings, and realize rapid positioning of passengers. The perspective diagram is shown below. Fig.14 shown.

[0215] For different usage scenarios, please refer to Fig.15 , Fig.15 This is an example diagram of the test results of a method for quickly locating the position of online ride-hailing passengers based on voice interaction and visual perspective proposed in the present invention.

[0216] This application also provides a rapid positioning model for online car-hailing passengers based on voice interaction and visual perspective, which is constructed using the positioning method described above.

[0217] This application provides a method for quickly locating the position of online car-hailing passengers based on voice interaction and visual perspective. It uses multiple on-board cameras to perceive the scene outside the car to achieve car perspective, avoid blind spot accidents and collect data in real time. Then, the system is awakened by the "one word wake-up" function, and a scenario-based speech recognition model is built to recognize the driver's voice information. Then, the data is processed using a multi-level cross-modal detection method of "sound-image", a passenger dynamic feature recognition model based on skeletal features, and a passenger multi-attribute "static feature" matching method. Finally, the results are fed back to the projector for display at the door and voice broadcast. The system can realize driver voice operation, reduce the phenomenon of "people looking for cars, cars looking for people" in online car-hailing, and alleviate congestion problems.

Claims

1. A method for quickly locating the position of a passenger in an online car-hailing vehicle based on voice interaction and visual perspective, characterized in that: The following steps are involved: S1: Deploy sensors, cameras, metal reflective screens and projectors on the online car-hailing vehicle, wherein there are five cameras, which are respectively located at the right front door, right rear door, front roof, rear roof and rear of the vehicle; the metal reflective screens are located at the front door, rear door and rear seat backrest, and the projector is located obliquely above the reflective screens; S2: Use the vehicle microphone to continuously collect the driver's voice information in the background, use Transformer as the voice wake-up word detection model and extract voice features to design an intelligent voice wake-up model to realize the "one-word wake-up" function of voice interaction; S3: Build a scenario-based speech recognition model to automatically identify the appearance and movement features of passengers by recognizing the driver's speech information, and save the text information of the above features; S4: Use the camera to obtain a panoramic image outside the vehicle, use the "sound-image" multi-level cross-modal detection model to detect pedestrian targets in the panoramic image, and obtain the appearance and posture feature detection information of pedestrian targets; S5: Design a passenger multi-attribute "static feature" matching method and a passenger "dynamic feature" recognition model based on skeleton features. Through text feature matching, the passenger feature information based on speech recognition in S3 and the passenger feature information based on visual detection in S4 are matched. S6: Based on the matching result of S5, determine the position information of the passenger; obtain the direction of the passenger with the vehicle as the reference origin through position estimation in the panoramic image; S7: According to the direction estimation result of S6, the projector in the vehicle is controlled to project the passenger information onto the screen in the corresponding direction in the vehicle, and a prompt is given using voice information at the same time.

2. According to claim 1, a method for quickly locating the position of a passenger in an online car-hailing vehicle based on voice interaction and visual perspective, characterized in that: The specific steps of S2 are as follows: First, the spectrogram is extracted from the input audio and then divided into sub-blocks, denoted as X∈R T×F , where the time domain window t = {1, 2, ... T}, the frequency domain f = {1, 2, ... F}; the sub-block is first passed through a linear projection matrix W0∈R in the frequency domain f×d Map to a high-dimensional space d, and then add a learnable category embedding in the time domain order Used to learn the global information of the spectrogram, and finally add a learnable position matrix As shown in formula (1), the processed X1 is fed into the Transformer encoder; X1=[X class ;XW1]+X pos #(1) Then, the obtained embedded features are input into a sequential Transformer encoder consisting of L multi-head attention and multi-layer perceptron blocks; in the lth Transformer, Q = X is calculated respectively. l W Q , K = X l W K and V = X l W K ;in, d h is the dimension of each attention head; the calculation of self-attention is shown in formula (2): MSA by Matrix The calculation is as follows: MSA(X l )=[SA1(X l );SA2(X l );...;SA k (X l );]W p #(3) The output of the first Transformer is: Finally, during streaming inference, a sliding window is used to cut the audio stream into sub-blocks for inference; the sliding window is a rectangular window that slides along the time axis with some overlap; Transformer detects each sub-block and scores it y i ; Once a y appears i >γ, indicating that the wake-up word is triggered, that is, the driver wakes up the system; threshold γ∈(0,1).

3. According to claim 1, a method for quickly locating the position of a passenger in an online car-hailing vehicle based on voice interaction and visual perspective, characterized in that: The specific steps of S3 are as follows: First, the real-time input audio file can be a local file or a file recorded using the recording function, and the voice signal is segmented into frames through VAD; Then, the Mel-frequency cepstral coefficient features are extracted from each frame, and a classification model is trained using a set of data frames with known speech and silence signal regions; the trained model will be used to classify unknown audio signals into spoken parts, unspoken parts, and silence parts; The detected breakpoints, i.e. the start and end time of each pronunciation part, will be saved in a List collection; finally, this List value and the original voice file will be used as the input of speech recognition for subsequent recognition operations; Then, the DFCNN acoustic model is used to convert the speech signal into the corresponding pinyin information. First, the obtained time domain audio signal is converted into a spectrogram, and the signal needs to be subjected to a short-time discrete Fourier transform, as shown in formula (6): The window length is represented by N, w(n) is the Hamming window function, f s represents the sampling frequency; as in formula (7), the spectrogram is generated by logarithmic amplitude operation; S(k,t)=20×log10|X t (k)|#(7) The generated spectrogram is used as the input of the model. The DFCNN model passes through a large number of convolutional layers, and uses a small 3×3 convolution kernel in each convolution. A pooling layer is added after multiple layers of convolution to complete the model construction. Through the combination of a large number of convolutional layers and pooling layers, the entire sentence is modeled, and the output unit directly corresponds to the final recognition result; Finally, the Transformer language model converts the pinyin into the corresponding text content, and the pinyin sequence output by the DFCNN acoustic model is used as input. It first passes through the text embedding layer to convert the digital representation of the words in the text into a vector representation, and captures the relationship between words through a high-dimensional space; then passes through the position encoding layer, and adds the position encoding to the word embedding vector. The results of the two layers are added as the input of the encoder; The encoder consists of 6 encoding units, each of which has a multi-head attention mechanism, a residual connection layer, a normalization layer, and a feedforward neural network layer; the multi-head attention mechanism is expressed as: Among them, Q, K, and V are query vectors, key vectors, and value vectors, respectively, which are composed of the input vector and their respective weights W Q , W K , W V The weights are obtained by training the model after initialization. dk is the dimension of the K vector. MH(Q,K,V)=Concat(head1,...,head n )W Q #(10) Among them, MH stands for MultiHead, The residual connection layer transfers the gradient by adding the input directly to the output of the intermediate layer of the network; The normalization layer adopts a batch normalization algorithm; The encoder sublayer also includes a feed-forward neural network layer, which can be expressed as: FFN(x)=max(0,xW1+b1)W2+b2 #(11) Finally, the text with punctuation is saved to complete the speech-to-text recognition.

4. According to claim 1, a method for quickly locating the position of a passenger in an online car-hailing vehicle based on voice interaction and visual perspective, characterized in that: The specific steps of S4 are as follows: First, we build a YOLOv8n target detection network under the pytorch framework. The YOLOv8n target detection network consists of five modules: Backbone module, Neck module, Detection Head module, Anchor Boxes module, and LossFunction module. Then, we made our own feature detection dataset. We specially added data of passengers with different color depths and different body shapes. The dataset was divided into training set, validation set and test set in a ratio of 7:2:1, and 2000 negative samples were added. Next, use the labelimg tool to manually label the training set, and then perform 100 rounds of training to generate the best.pt model. The model effect generated by the training is verified by the validation set and test set data. Finally, the "sound-image" multi-level cross-modal detection model is used to obtain the target ID information, coordinate information, and target key point information.

5. According to claim 4, a method for quickly locating the position of a passenger in an online car-hailing vehicle based on voice interaction and visual perspective is characterized in that: The Backbone module in the YOLOv8n target detection network is composed of a convolutional layer, an activation function, and a pooling layer. Through multi-scale feature extraction and feature fusion, feature representation is extracted from the input image to achieve detection of targets of different scales. Its working process includes convolution feature extraction, nonlinearity introduced by activation function, dimension reduction by pooling, and parameter sharing, which ultimately provides the necessary semantic understanding and information expression for subsequent target detection tasks. The Neck module processes features of different scales through a feature pyramid structure, and improves the expressiveness of features through multi-scale feature fusion, semantic enhancement, etc., so as to accurately detect objects of different scales. The Neck module also realizes cross-scale feature alignment through downsampling and upsampling operations of the feature pyramid; finally, the Neck module captures more local information through a fine-grained feature fusion mechanism; The Detection Head module predicts the location and category of the target box through feature map processing and Anchor Boxes module definition, and outputs the target detection result after screening and correction, thereby achieving accurate detection and positioning of the target; The Anchor Boxes module is a series of predefined bounding boxes distributed at different positions and scales of the feature map, which are used to generate candidate boxes in the image. These Anchor Boxes cover various possible object shapes and scales by defining different aspect ratios and sizes. During the object detection process, each Anchor Box module corresponds to the pixel position on the feature map, and the model predicts the position and category information of the possible objects in each Anchor Box module. By comparing the position of the actual target frame and combining it with the category probability for screening and correction, we can finally get accurate target detection results. The Loss Function module includes the coordinate loss of the target box position, the classification loss of the target category, and the matching loss between the target box and the Anchor Boxes; the working process of the loss function involves calculating the difference between the predicted target box and the true target box, and weighting it in combination with the category probability to minimize the error between the predicted box and the true box; in addition, the loss function also considers the weights of positive and negative samples and the confidence threshold to ensure effective training and adjustment of samples of different difficulties; by minimizing the overall loss function, the model can more accurately predict the position and category of the target.

6. According to claim 4, a method for quickly locating the position of a passenger in an online car-hailing vehicle based on voice interaction and visual perspective is characterized in that: The multi-level cross-modal detection model of "sound-image" in S4 includes the following steps: First, for the text of speech recognition in S3, the noun phrases are obtained by n-gram algorithm, and then the text embedding is extracted using the Transformer text encoder Text Encoder based on CLIP pre-training. Where T is the given text, C is the number of nouns, and D is the embedding dimension; Next, multiple cameras at different locations on the vehicle are used to simultaneously obtain multiple real-time video streams and pass the video streams through the YOLOv8 Backbone to obtain multi-scale image features. Then, W and X obtained in the above steps are l Feature fusion is performed through a re-parameterized visual-language path aggregation network, where the fusion process mainly includes a text-guided cross-block bias layer from text to image and an image pooling attention mechanism from image to text; Finally, the fused target encoding is output through the Text Contrastive Head, and a decoupled head with two 3×3 convolutions is used to regress the bounding box. and object embedding And by introducing the text comparison center, the object-text similarity is obtained, so as to output the coordinates of the object with the greatest similarity; the target code e and the text code w are represented by L2-Norm; Where L2-Norm() represents L2 normalization, W j ∈W is the j-th text embedding, α is the scaling factor, and β is the offset factor.

7. According to claim 1, a method for quickly locating the position of a passenger in an online car-hailing vehicle based on voice interaction and visual perspective, characterized in that: The n-gram algorithm comprises the following steps: the content in the text is subjected to a sliding window operation of size N according to bytes, forming a sequence of byte segments of length N; each byte segment is called a gram, and the occurrence frequency of all the grams is counted, filtered according to a pre-set threshold, and a key gram list is formed, thereby realizing the extraction of noun phrases; The Transformer text encoder Text Encoder based on CLIP pre-training consists of two parts, namely, a vision transformer responsible for images and a transformer responsible for texts; In the vision transformer, the image first passes through a patch convolution and a continuous residual attention structure, and the output is a tensor of (1, 50, 768). Finally, the mapping I1, I2, ..., I2 to (1, 512) is completed through matrix multiplication. N ; In the Transformer, the text is tokenized to get a new tensor, and then passes through the embedding layer to get a (3, 77, 512) tensor. After adding the positional encoding (nn.Parameter), it is fed into the model. Finally, the model outputs the (1, 512) features T1, T2, ..., T N ; By calculating I i and T j The cosine similarity between them is used to measure the correspondence between the corresponding text and the image. The larger the cosine similarity, the stronger the correspondence. The visual-language path aggregation network, for images, multi-scale features X l Through downsampling and upsampling operations, combined with the embedding set W of the detection category vocabulary, T-CSPLayer is used to allocate attention. For text, the embedding set W of the detection category vocabulary is combined with 3 multi-scale feature maps for max pooling to perform 3x3 area block division, and 27 feature map patches are obtained. Based on multiHead attention, I-Pooling Attention is further used to allocate attention; Among them, T-CSPLayer is used to analyze the multi-scale image features X l Use the embedding W of the detection category vocabulary to allocate attention and obtain the multi-scale feature output X′ l as follows: Where δ represents the sigmoid function. Among them, I-Pooling Attention uses multi-scale image features X to embed the word W of the detection category. l By allocating attention, the feature output W′ is obtained as follows:

8. According to claim 1, a method for quickly locating the position of a passenger in an online car-hailing vehicle based on voice interaction and visual perspective, characterized in that: In S5, a passenger multi-attribute "static feature" matching method is innovated for different appearance features of different passengers. First, the target information features obtained by S4 include the id value and its corresponding two-point coordinates. Combined with the passenger feature text information input by the driver's voice, the target label set is first constructed through the input text information, the target detection processing results of YOLOv8 in the video stream are read cyclically, and the identified labels are matched with the target labels. All possible label combinations are found through permutations and combinations: {set}({v i |(v i ,b i )∈c})={set}(box ids }#(16)# Enlarge the coordinates of the target box, and then calculate the intersection area of ​​every two bounding boxes; interArea=max(0,min(xmaxA,xmaxB)-max(xminA,xminB))×max(0,min(ymaxA,ymaxB)-max(yminA,yminB)) ##(17) in min(xmaxA,xmaxB), max(xminA,xminB), min(ymaxA,ymaxB), max(yminA, yminB) is the coordinate of the intersection of the two boxes, and interArea is the area of ​​the intersection. Then, the corresponding IOU value is calculated. If the IOU value is greater than the set threshold of 0.3, it means that they are the same person. Then the characteristic passenger is found and the actual position of the characteristic passenger is announced according to the position of the average center point. Thus, the passenger feature information based on voice recognition in S3 is matched with the passenger feature information based on visual detection in S4, and the feature passenger is found; Among them, interArea is the area of ​​the intersection area, and unionArea is the area of ​​the union area. Next, the acquired data is filtered through the coordinates of the key points of the human body obtained by S4, and then the numpy array of the coordinates of all the key points is obtained, that is, the (x, y) coordinate information corresponding to each key point of the human body; In view of the different gestures of passengers waving to taxis in real life and the possible false triggering of pedestrians, a passenger "dynamic feature" recognition model based on bone features is established, and strict authentication indicators for hand-raising gestures are established - the wrist key point is higher than the shoulder key point and the height between the wrist, elbow and shoulder on the same side is within a certain threshold; Set up the angle_between_points function to calculate the angles between key points. Use the key point coordinates obtained in the above process, namely shoulder point[5], point[6] elbow point[7], point[8] wrist point[9], point[10], to pass them into the angle_between_points function, and calculate the angles between the three points on the left and right sides respectively. The coordinates of the three points on the left and right are marked as [x1, y1], [x2, y2], [x3, y3]; then calculate the AB vector and the BC vector, which are AB = (x2-x1, y2-y1) and BC = (x3-x2, y3-y2) respectively; then use the dot product and length formula of the vectors to calculate the angle between the two vectors, AB·BC=x1x2+y1y2 #(21) Then we can get the cosine of the angle By using the inverse cosine Convert radians to degrees Finally, if the angles between the three points are between 90° and 160° and the wrist y value on the same side (wrist point[9], point[10] shoulder point[5], point[6]) is greater than the shoulder y value, it is judged as a potential hand-raising action.

9. The method for quickly locating the passenger position of an online car-hailing vehicle based on voice interaction and visual perspective according to claim 1, characterized in that: In said S6, the average center coordinates of the determined passenger characteristics obtained in S5 are used to establish a passenger position prediction scheme centered on the vehicle, and the actual position of the passenger from the driver centered on the vehicle is determined by calculating the position of the average center of the passenger; The specific location of the passenger is obtained based on the judgment result, and the pyttsx3 module is used for text-to-speech function to make a voice announcement to inform the driver that he should slow down, that a potential passenger has appeared, and to inform the driver of his specific location; Determine the position of the area to be captured according to the coordinates of the key points, and capture it in the original image according to the determined area range, obtain the regional image where the key part is located, and use the cv2.resize function to scale it to the appropriate size, and flexibly adjust the placement of the screen according to the actual position, and in the video stream processing loop, update the small screen content according to the real-time recognition results, and draw the updated small screen on the original image to achieve dynamic display.

10. The method for quickly locating the passenger position of an online car-hailing vehicle based on voice interaction and visual perspective according to claim 1, characterized in that: In the S7: When the passenger is not located, the image containing the visual blind spot is projected to achieve the perspective effect of the blind spot. The image content is updated in real time in the video stream processing loop to achieve dynamic display. At the same time, the specific location of the passenger is determined based on the judgment result, and the pyttsx3 module is used for text-to-speech function to make a voice broadcast to inform the driver that he should slow down, the presence of characteristic passengers and potential passengers, and inform the driver of their specific location. When executing system functions, the vehicle-mounted camera and projector are combined with the system to achieve visual perspective. Multiple vehicle-mounted cameras are placed next to the car door to collect 360° image data, and the visual processing results are transmitted to the projector using OpenCV. The projector is placed on the opposite side of the projection cloth, and projects the image onto a screen close to the inside of the car body. It can display multiple features of passengers specially marked by OpenCV, and use small screens placed at different positions of the image to specifically display different features, provide visual warnings, and realize rapid positioning of passengers.

Images