A method for rapid positioning of online ride-hailing passengers based on voice interaction and visual perspective

Abstract

The present invention provides a method for quickly locating the position of online car-hailing passengers based on voice interaction and visual perspective, which relates to the field of intelligent transportation technology. The method comprises the following steps: placing sensors, cameras, metal reflective screens and projectors on a car; collecting the driver's voice information and designing an intelligent voice wake-up model; building a scenario-based voice recognition model; using a multi-level cross-modal detection model of "sound-image" to detect pedestrian targets in panoramic images and obtain appearance and posture feature detection information of pedestrian targets; designing a "static feature" matching method and a "dynamic feature" recognition model to achieve passenger feature information matching; determining the passenger's position information based on the matching results; obtaining the passenger's direction with the vehicle as the reference origin through position estimation in the panoramic image; controlling the in-car projector according to the direction estimation result to project the passenger information onto the screen in the corresponding direction in the car, and simultaneously giving prompts using voice information.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a method for quickly locating the position of a ride-hailing passenger based on voice interaction and visual perspective. Background Art

[0002] With the rapid development and improvement of online ride-hailing platforms, ride-hailing has become a common means of transportation. However, due to inaccurate positioning or passengers' unfamiliarity with the surroundings, ride-hailing often encounters a "passenger-to-ride" situation when picking up passengers. This not only wastes the time of both drivers and passengers, but also hinders the flow of oncoming traffic and causes congestion on surrounding roads. Large ride-hailing systems also have multiple blind spots, making it difficult to quickly locate passengers. Current systems have significant shortcomings in intelligent interaction, unable to respond to passenger needs in real time or effectively monitor potential traffic risks. Therefore, in increasingly congested traffic, how can drivers find their passengers amidst the vast crowds and better understand the situation outside the vehicle to reduce the risk of accidents? Current in-vehicle systems are not effective in addressing these issues. Therefore, developing a method that can achieve intelligent monitoring and passenger services through voice interaction is of great significance.

[0003] Currently, the common methods for locating passengers for online ride-hailing services can be divided into two types: telephone communication positioning and GPS positioning. These methods have obvious drawbacks: (1) Telephone communication positioning: The online ride-hailing driver communicates with the passenger over the phone about the passenger's waiting location. However, this method is not suitable for daily life because the passenger is unfamiliar with the surrounding situation and finds it difficult to describe the specific location, or because the surrounding crowd is too large to distinguish. (2) GPS positioning: The online ride-hailing driver obtains the passenger's actual location information through the passenger's mobile phone positioning. However, this method has inaccurate positioning information and unstable signals, making it difficult for the driver to determine the passenger's actual location and difficult to apply in practice. Summary of the Invention

[0004] The purpose of the present invention is to solve the technical problems existing in the existing online car-hailing passenger positioning method.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for quickly locating a ride-hailing passenger's position based on voice interaction and visual perspective includes the following steps:

[0007] S1: Deploy sensors, cameras, metal reflective screens, and projectors on the online ride-hailing vehicle. Five cameras are provided, located at the right front door, right rear door, front roof, rear roof, and rear of the vehicle, respectively. The metal reflective screens are located at the front door, rear door, and rear seatback, and the projector is located diagonally above the reflective screens.

[0008] S2: Utilize the vehicle's microphone to continuously collect the driver's voice information in the background. Use Transformer as the voice wake-up word detection model and extract voice features to design an intelligent voice wake-up model to implement the "one-word wake-up" function for voice interaction.

[0009] S3: Build a scenario-based speech recognition model that automatically identifies the appearance and movement characteristics of passengers by recognizing the driver's voice information and saves the text information of these characteristics;

[0010] S4: Use the camera to obtain a panoramic image outside the vehicle, and use the "sound-image" multi-level cross-modal detection model to detect pedestrian targets in the panoramic image, obtaining the appearance and posture feature detection information of the pedestrian targets;

[0011] S5: Design a passenger multi-attribute "static feature" matching method and a passenger "dynamic feature" recognition model based on skeleton features. Through text feature matching, match the passenger feature information based on speech recognition in S3 with the passenger feature information based on visual detection in S4.

[0012] S6: Based on the matching results of S5, determine the passenger's position information; obtain the passenger's direction with the vehicle as the reference origin through position estimation in the panoramic image;

[0013] S7: Based on the direction estimation result of S6, control the projector in the vehicle to project the passenger information onto the screen in the corresponding direction in the vehicle, and give a prompt using voice information at the same time.

[0014] Preferably, the specific steps of S2 are as follows:

[0015] First, the spectrogram is extracted from the input audio and then divided into sub-blocks, represented as X∈R T×F , where the time domain window t={1,2,…T}, the frequency domain f={1,2,…F}; the sub-block first passes through a linear projection matrix W0∈R in the frequency domain f×d Map to a high-dimensional space d, and then add a learnable category embedding in the time domain order Used to learn the global information of the spectrogram, and finally add a learnable position matrix As shown in formula (1), the processed X1 is fed into the Transformer encoder;

[0016] X1=[X class ;XW1]+X pos #(1)

[0017] Then, the obtained embedding features are input into a sequential Transformer encoder consisting of L multi-head attention and multi-layer perceptron blocks; in the lth Transformer, Q = X is calculated respectively. I W Q , K=X l W K and V = X l W K ;in, d h is the dimension of each attention head; the calculation of self-attention is shown in formula (2):

[0018]

[0019] MSA through the matrix The calculation is as follows:

[0020] MSA(X l )=[SA1(X l );SA2(X l );…;SAk(X l );]W p #(3)

[0021] The output of the lth Transformer is:

[0022]

[0023]

[0024] Finally, during streaming inference, a sliding window is used to cut the audio stream into sub-blocks for inference; the sliding window is a rectangular window that slides along the time axis with some overlap; Transformer detects each sub-block and scores it y i ; Once a y appears i >γ, indicating that the wake-up word is triggered, that is, the driver wakes up the system; the threshold γ∈(0,1).

[0025] Preferably, the specific steps of S3 are as follows:

[0026] First, the real-time input audio file can be a local file or a file recorded using the recording function, and the voice signal is segmented into frames through VAD;

[0027] Mel-frequency cepstral coefficient features are then extracted from each frame, and a classification model is trained using a set of data frames with known speech and silence signal regions. The trained model is then used to classify unknown audio signals into spoken, unspoken, and silent parts.

[0028] The detected breakpoints, that is, the start and end time of each pronunciation segment, will be saved in a List collection; ultimately, this List value and the original voice file are used as the input of speech recognition for subsequent recognition operations;

[0029] Then, the DFCNN acoustic model is used to convert the speech signal into the corresponding pinyin information. First, the obtained time domain audio signal is converted into a spectrogram. The signal needs to be subjected to a short-time discrete Fourier transform, as shown in formula (6):

[0030]

[0031] The window length is expressed as N, w(n) is the Hamming window function, f s represents the sampling frequency; as in formula (7), the spectrogram is generated by logarithmic amplitude operation;

[0032] S(k,t)=20×log 10 |X t (k)| #(7)

[0033] The generated spectrogram is used as the input of the model. The DFCNN model passes through a large number of convolutional layers, and uses a small 3×3 convolution kernel in each convolution. A pooling layer is added after multiple layers of convolution to complete the model construction.

[0034] Through the combination of a large number of convolutional layers and pooling layers, the entire sentence is modeled, and the output unit directly corresponds to the final recognition result;

[0035] Finally, the Transformer language model converts the pinyin into the corresponding text content, and the pinyin sequence output by the DFCNN acoustic model is used as input. It first passes through the text embedding layer to convert the digital representation of the words in the text into a vector representation, and captures the relationship between words through high-dimensional space; then passes through the position encoding layer, and adds the position encoding to the word embedding vector. The results of the two layers are added as the input of the encoder;

[0036] The encoder consists of 6 encoding units, each of which has a multi-head attention mechanism, a residual connection layer, a normalization layer, and a feedforward neural network layer; the multi-head attention mechanism is expressed as:

[0037]

[0038] Among them, Q, K, and V are query vectors, key vectors, and value vectors respectively, which are composed of the input vector and their respective weights W. Q 、W K 、W V The weights are obtained by dot product and the model training after initialization. dk is the dimension of the K vector.

[0039]

[0040] MH(Q,K,V)=Concat(head1,...,head n )W Q #(10)

[0041] Among them, MH stands for MultiHead,

[0042] The residual connection layer transfers the gradient by adding the input directly to the output of the intermediate layer of the network;

[0043] The normalization layer adopts a batch normalization algorithm;

[0044] The encoder sublayer also contains a feedforward neural network layer, which can be expressed as:

[0045] FFN(x)=max(0,xW1+b1)W2+b2 #(11)

[0046] Finally, the text with punctuation is saved to complete the speech-to-text recognition.

[0047] Preferably, the specific steps of S4 are as follows:

[0048] First, we build a YOLOv8n target detection network under the pytorch framework. The YOLOv8n target detection network consists of five modules: Backbone module, Neck module, Detection Head module, AnchorBoxes module, and LossFunction module.

[0049] Then, we created a custom feature detection dataset. We specifically added data on passengers of different color depths and body types to the dataset. The dataset was divided into training, validation, and test sets in a ratio of 7:2:1, and 2,000 additional negative samples were added.

[0050] Next, we manually labeled the training set using the labelimg tool, and then performed 100 rounds of training to generate the best.pt model. We then used validation and test set data to verify the effectiveness of the trained model.

[0051] Finally, a multi-level cross-modal detection model of "sound-image" is used to obtain target ID information, coordinate information, and target key point information.

[0052] Preferably, the Backbone module in the YOLOv8n target detection network is composed of a convolution layer, an activation function, and a pooling layer. It extracts feature representations from the input image through multi-scale feature extraction and feature fusion to achieve detection of targets of different scales. Its working process includes convolution feature extraction, nonlinearity introduced by activation function, pooling dimensionality reduction, and parameter sharing, ultimately providing the necessary semantic understanding and information expression for subsequent target detection tasks.

[0053] The Neck module processes features of different scales through a feature pyramid structure and improves the expressiveness of features through multi-scale feature fusion and semantic enhancement to achieve accurate detection of objects of different scales. The Neck module also achieves cross-scale feature alignment through downsampling and upsampling operations of the feature pyramid. Finally, the Neck module captures more local information through a fine-grained feature fusion mechanism.

[0054] The Detection Head module predicts the location and category of the target box through feature map processing and Anchor Boxes module definition, and outputs the target detection result after screening and correction, thereby achieving accurate detection and positioning of the target;

[0055] The AnchorBoxes module is a series of predefined bounding boxes distributed at different locations and scales in the feature map, used to generate candidate boxes in the image. These AnchorBoxes cover a variety of possible object shapes and scales by defining different aspect ratios and sizes. During the object detection process, each AnchorBox module corresponds to a pixel position on the feature map, and the model predicts the location and category information of the possible objects within each AnchorBox module. By comparing with the actual target box position and combining the category probability for screening and correction, accurate object detection results are ultimately obtained.

[0056] The Loss Function module includes the coordinate loss of the target box position, the classification loss of the target category, and the matching loss between the target box and AnchorBoxes; the working process of the loss function involves calculating the difference between the predicted target box and the true target box, and weighting it with the category probability to minimize the error between the predicted box and the true box; in addition, the loss function also considers the weights of positive and negative samples and the confidence threshold to ensure effective training and adjustment of samples of different difficulty levels; by minimizing the overall loss function, the model can more accurately predict the location and category of the target.

[0057] Preferably, the multi-level cross-modal detection model of "sound-image" in S4 comprises the following steps:

[0058] First, for the text of speech recognition in S3, the noun phrases are obtained by n-gram algorithm, and then the text embedding is extracted using the Transformer text encoder based on CLIP pre-training. Where T is the given text, C is the number of nouns, and D is the embedding dimension;

[0059] Then, multiple cameras at different locations on the vehicle are used to simultaneously obtain multiple real-time video streams and pass the video streams through the YOLOv8n Backbone to obtain multi-scale image features.

[0060] Then, W and X obtained in the above steps are l Feature fusion is performed through a reparameterized vision-language path aggregation network, where the fusion process mainly includes a text-guided cross-block bias layer from text to image and an image pooling attention mechanism from image to text;

[0061] Finally, the fused target encoding is output through the Text Contrastive Head, and a decoupling head with two 3×3 convolutions is used to regress the bounding box. and object embedding And by introducing the text comparison center, we can obtain the object-text similarity, and output the coordinates of the object with the greatest similarity; the target code e and the text code w are represented by L2-Norm;

[0062] s k,j =α·L2-Norm(e k )·L2-Norm(w j ) T +β #(12)

[0063] Where L2-Norm() represents L2 normalization, W j ∈W is the j-th text embedding, α is the scaling factor, and β is the offset factor.

[0064] Preferably, the n-gram algorithm comprises the following steps: performing a sliding window operation of size N on the content of the text according to bytes to form a sequence of byte segments of length N; each byte segment is called a gram, and the occurrence frequency of all the grams is counted and filtered according to a pre-set threshold to form a key gram list, thereby realizing the extraction of noun phrases;

[0065] The Transformer text encoder TextEncoder based on CLIP pre-training consists of two parts: the vision transformer responsible for images and the transformer responsible for text;

[0066] In the vision transformer, the image first passes through a patch convolution and a continuous residual attention structure, outputting a tensor of (1, 50, 768), and finally completing the mapping I1, I2, ..., IN to (1, 512) through matrix multiplication;

[0067] In the Transformer, the text is tokenized to get a new tensor, and then passes through the embedding layer to get a (3, 77, 512) tensor. After adding the position code (nn.Parameter), it is fed into the model. Finally, the model outputs the (1, 512) features T1, T2, ..., T N ;

[0068] By calculating I i and T j The cosine similarity between them is used to measure the correspondence between the corresponding text and the image. The larger the cosine similarity, the stronger the correspondence.

[0069]

[0070] The visual-language path aggregation network, for images, multi-scale features X l Through downsampling and upsampling operations, combined with the embedding set W of the detection category vocabulary, T-CSPLayer is used to allocate attention. For text, the embedding set W of the detection category vocabulary is combined with three multi-scale feature maps to do max pooling for 3x3 area block division, and 27 feature map patches are obtained. And based on multiHead attention, I-PoolingAttention is further used to distribute attention;

[0071] Among them, T-CSPLayer is used to calculate the multi-scale image features X l Use the embeddingW of the detection category vocabulary to distribute attention and obtain the multi-scale feature output X' l as follows:

[0072]

[0073] Where δ represents the sigmoid function.

[0074] Among them, I-PoolingAttention uses multi-scale image features X to detect the embeddingW of category vocabulary lBy allocating attention, the feature output W' is obtained as follows:

[0075]

[0076] Preferably, in S5, a novel passenger multi-attribute "static feature" matching method is developed based on the different appearance features of different passengers. First, the target information features obtained in S4 include the ID value and its corresponding two-point coordinates. Combined with the passenger feature text information input by the driver's voice, a target label set is first constructed based on the input text information. The target detection processing results of YOLOv8n in the video stream are read cyclically, and the identified labels are matched with the target labels. All possible label combinations are found through permutations and combinations:

[0077]

[0078] {set}({v i |(v i ,b i )∈c})={set}(box ids}#(16)##

[0079] Enlarge the coordinates of the target box and then calculate the intersection area of ​​every two bounding boxes;

[0080] interArea=max(0,min(xmaxA,xmaxB)-max(xminA,xminB))×max(0,min(ymaxA,ymaxB-max(yminA,yminB)) #(17)

[0081] Among them, min(xmaxA,xmaxB), max(xminA,xminB), min(ymaxA,ymaxB), max(yminA,yminB) are the coordinates of the intersection of the two boxes, and interArea is the area of ​​the intersection.

[0082] Then, the corresponding IOU value is calculated. If the IOU value is greater than the set threshold of 0.3, it means that they are the same person. Then the characteristic passenger is found and the actual position of the characteristic passenger is announced according to the position of the average center point.

[0083] Thus, the passenger feature information based on voice recognition in S3 is matched with the passenger feature information based on visual detection in S4 to find the characteristic passenger;

[0084]

[0085] Where interArea is the intersection area, and unionArea is the union area.

[0086] Next, the acquired data is filtered using the coordinates of the key points of the human body obtained by S4 to obtain the numpy array of the coordinates of all key points, that is, the (x, y) coordinate information corresponding to each key point of the human body;

[0087] In view of the different gestures of passengers waving taxis in real life and the possible false triggering of pedestrians, a passenger "dynamic feature" recognition model based on skeletal features was established. Strict authentication indicators for hand-raising gestures were established: the wrist key point must be higher than the shoulder key point, and the height between the wrist, elbow and shoulder on the same side must be within a certain threshold.

[0088] Set up the angle_between_points function to calculate the angles between key points; use the key point coordinates obtained in the above process - shoulder point[5], point[6] elbow point[7], point[8] wrist point[9], point

[10] , pass them into the angle_between_points function, and calculate the angles between the three points on the left and right respectively;

[0089] The coordinates of the three points on the left and right are marked as [x1, y1], [x2, y2], [x3, y3]; then calculate the AB vector and the BC vector, which are AB = (x2-x1, y2-y1) and BC = (x3-x2, y3-y2) respectively; then use the dot product of the vectors and the length formula to calculate the angle between the two vectors.

[0090]

[0091]

[0092] AB·BC=x1x2+y1y2 #(21)

[0093] Then we can get the cosine of the angle

[0094] Taking the arc cosine, we get

[0095] Convert radians to degrees

[0096]

[0097] Finally, if the angles between the three points are between 90° and 160° and the wrist y value on the same side (wrist point[9], point

[10] shoulder point[5], point[6]) is greater than the shoulder y value, it is judged as a potential hand-raising action to hail a taxi.

[0098] Preferably, in said S6, the average center coordinates of the determined passenger characteristics obtained in S5 are used to establish a passenger position prediction scheme centered on the vehicle, and the actual position of the passenger from the driver centered on the vehicle is determined by calculating the position of the average center coordinates of the passengers;

[0099] The specific location of the passenger is determined based on the judgment result, and the pyttsx3 module is used for text-to-speech function to make a voice announcement to inform the driver that he should slow down, that a potential passenger has appeared, and inform the driver of the specific location;

[0100] Determine the position of the area to be captured based on the coordinates of the key points, and capture it in the original image based on the determined area range. Get the regional image where the key part is located, and use the cv2.resize function to scale it to the appropriate size. According to the actual position, flexibly adjust the placement of the screen. In the video stream processing loop, update the small screen content according to the real-time recognition results, and draw the updated small screen on the original image to achieve dynamic display.

[0101] Preferably, in S7:

[0102] When the passenger is not located, the image including the blind spot is projected to achieve the perspective effect of the blind spot. The image content is updated in real time during the video stream processing loop to achieve dynamic display. At the same time, the specific location of the passenger is determined based on the judgment result, and the pyttsx3 module is used for text-to-speech function to inform the driver to slow down, the presence of characteristic passengers and potential passengers, and inform the driver of their specific location.

[0103] When executing system functions, the on-board camera and projector are combined with the system to achieve visual perspective; multiple on-board cameras are placed next to the car door to collect 360° image data, and the visual processing results are transmitted to the projector using OpenCV; the projector is placed on the opposite side of the projection cloth, and the image is projected onto the screen close to the inside of the car body. It can display multiple features of passengers specially marked by OpenCV, and use small screens placed at different positions of the image to specifically display different features, provide visual warnings, and realize rapid positioning of passengers.

[0104] The aforementioned method for rapidly locating online ride-hailing passengers based on voice interaction and visual clairvoyance utilizes multiple onboard cameras to perceive the exterior scene, achieving vehicle clairvoyance, avoiding blind spot accidents, and collecting data in real time. A "wake-up with one word" feature wakes the system, and a scenario-based speech recognition model is constructed to identify the driver's voice. The data is then processed using a multi-level cross-modal "sound-image" detection method, a skeletal-based passenger dynamic feature recognition model, and a multi-attribute "static feature" matching method for passengers. The results are ultimately fed back to a projector for display at the vehicle door and announced by voice. This system enables driver voice control, reducing the phenomenon of "people looking for cars, cars looking for people" in online ride-hailing services, and alleviating congestion.

[0105] Compared with the prior art, this application has the following beneficial effects:

[0106] (1) A multi-level cross-modal target recognition model based on voice and image is proposed, which enables rapid detection of target passengers based on voice interaction and image recognition;

[0107] (2) The combination of the external camera and the internal projector realizes the visualization of blind spot passenger positioning based on visual perspective;

[0108] (3) Use voice to indicate the relative position of passengers and assist the driver in quickly locating them. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] Figure 1 This is a schematic diagram of the steps of a method for quickly locating a passenger's position in an online ride-hailing vehicle based on voice interaction and visual perspective in one embodiment of the present invention;

[0110] Figure 2 This is a general framework diagram of a method in one embodiment of the present invention;

[0111] Figure 3 This is a schematic diagram of a real scene of a vehicle-mounted camera in one embodiment of the present invention;

[0112] Figure 4 This is a schematic diagram of a real scene of the screen setting in one embodiment of the present invention;

[0113] Figure 5 A schematic diagram of a real-life projector configuration in accordance with one embodiment of the present invention;

[0114] Figure 6 This is a diagram of a voice wake-up network structure in one embodiment of the present invention;

[0115] Figure 7 1 is a diagram of the generation process of a spectrogram in one embodiment of the present invention;

[0116] Figure 8 This is a DFCNN network structure diagram in one embodiment of the present invention;

[0117] Figure 9 A YOLOv8n image recognition database in one embodiment of the present invention;

[0118] Figure 10 This is a network structure diagram of a multi-level cross-modal detection model of "sound-image" in one embodiment of the present invention;

[0119] Figure 11 This is a diagram of a re-parameterized VL-PAN structure in one embodiment of the present invention;

[0120] Figure 12 This is a diagram of a passenger multi-attribute "static feature" matching algorithm in one embodiment of the present invention;

[0121] Figure 13 This is a diagram of a passenger dynamic feature recognition algorithm based on skeleton features in one embodiment of the present invention;

[0122] Figure 14 It is a perspective schematic diagram of a specific embodiment of the present invention;

[0123] Figure 15 This is a real-scene test diagram in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0124] The present invention is further described in detail below with reference to specific embodiments.

[0125] See also Figure 1 , a method for quickly locating the position of online car-hailing passengers based on voice interaction and visual perspective, the framework structure of the method is as follows Figure 2 As shown, the following steps are included:

[0126] S1: Deploy sensors, cameras, metal reflective screens and projectors on cars;

[0127] Five cameras are installed in the vehicle's blind spots, and metal reflective screens are laid on the front doors, rear doors, and rear seat backs. Three projectors are installed diagonally above the screens.

[0128] In one embodiment, the specific steps of S1 are as follows:

[0129] S1-1: Please refer to Figure 3 Sensors are deployed throughout the vehicle, with five cameras installed in blind spots: the right front door, right rear door, front roof, rear roof, and rear. Two wide-angle cameras, located on the front and rear roofs, detect the target passenger; the remaining three are high-definition cameras that provide a visual understanding of the surrounding environment.

[0130] S1-2: Please refer to Figure 4Metal reflective curtains are laid in the blind spots of the driver's field of view, and the reflective curtains are located at the front doors, rear doors and rear seat backrests respectively.

[0131] S1-3: Please refer to Figure 5 , install the projector obliquely above the reflective screen to ensure the consistency of the projected image with the real external environment to the greatest extent;

[0132] When a target passenger is detected in the panoramic image, the detected object is cut out and projected onto a screen in the corresponding direction inside the vehicle. This selective control of the projector allows for perspective viewing, eliminating the problem of the target passenger appearing in the vehicle's blind spot. Furthermore, the projected image in the blind spot helps the driver observe the surrounding environment, reducing the risk of collisions.

[0133] S2: Utilize the vehicle's microphone to continuously collect the driver's voice information in the background. Use Transformer as the voice wake-up word detection model and extract voice features to design an intelligent voice wake-up model to implement the "one-word wake-up" function for voice interaction.

[0134] For details, please refer to Figure 6 In one embodiment, a background sub-thread is started to continuously record audio in the background and pass the audio into the model.

[0135] First, the spectrogram is extracted from the input audio and then divided into sub-blocks, represented as X∈R T×F , where the time domain window t={1,2,…T} and the frequency domain f={1,2,…F}. The sub-block is first passed through a linear projection matrix W0∈R in the frequency domain. f×d Map to a high-dimensional space d, and then add a learnable category embedding in the time domain order Used to learn the global information of the spectrogram, and finally add a learnable position matrix As shown in formula (1), the processed X1 is fed into the Transformer encoder.

[0136] X1=[X class ;XW1]+X pos #(1)

[0137] Then, the obtained embedding features are fed into a sequential Transformer encoder consisting of L multi-head attention (MSA) and multi-layer perceptron (MLP) blocks. In the lth Transformer, Q = X is calculated respectively. I W Q , K=X l W K and V = X l W K .in, dh is the dimension of each attention head. The calculation of self-attention is shown in formula (2),

[0138]

[0139] MSA through the matrix Calculated, such as,

[0140] MSA(X l )=[SA1(X l );SA2(X l );…;SAk(X l );]W p #(3)

[0141] The output of the lth Transformer is:

[0142]

[0143]

[0144] Finally, during streaming inference, a sliding window is used to cut the audio stream into sub-blocks for inference. The sliding window is a rectangular window that slides along the time axis with some overlap. Transformer detects each sub-block and scores it y i Once a y appears i If >γ, it indicates that the wake-up word is triggered, that is, the driver wakes up the system. The threshold γ∈(0,1).

[0145] S3: Build a scenario-based speech recognition model that automatically identifies the appearance and movement characteristics of passengers by recognizing the driver's voice information and saves the text information of these characteristics;

[0146] In one embodiment, first, the audio file input in real time can be selected as a local file or a file recorded using the recording function, and the voice signal is first segmented into frames through endpoint detection (VAD). Then, the Mel-frequency cepstral coefficient features are extracted from each frame, and a classification model is trained using a data frame set of known speech and silence signal areas. The trained model will be used to classify unknown audio signals, dividing them into pronunciation parts, unpronounced parts, and silence parts. The breakpoints detected, that is, the start and end times of each pronunciation part, will be saved in a List set. Finally, this List value and the original voice file are used as input for speech recognition for subsequent recognition operations.

[0147] Then, the DFCNN acoustic model is used to convert the speech signal into the corresponding pinyin information. First, the obtained time domain audio signal is converted into a spectrogram. The signal needs to be subjected to a short-time discrete Fourier transform, as shown in formula (6):

[0148]

[0149] The window length is expressed as N, w(n) is the window function, and the Hamming window is used in this paper. s Represents the sampling frequency. As shown in formula (7), the spectrogram is generated by logarithmic amplitude operation.

[0150] S(k,t)=20×log 10 |X t (k)| #(7)

[0151] The generated spectrogram (the spectrogram generation process is as follows Figure 7 As shown in the figure, the DFCNN model takes the input of the model as the input. The DFCNN model passes through a large number of convolutional layers, and uses a small 3×3 convolution kernel in each convolution. A pooling layer is added after multiple layers of convolution to complete the construction of the model. Through the combination of a large number of convolutional layers and pooling layers, the entire sentence of speech is modeled, and the output unit directly corresponds to the final recognition result. The advantage of using the spectrogram as the input is that it avoids the obvious loss of high-frequency parts caused by artificially designed filters during feature extraction in traditional speech recognition. By inputting the model in the form of a spectrogram, information in both the frequency domain and the time domain is preserved. Due to the accumulation of a large number of convolutional layers and pooling layers in this model, its control over future and historical information has also been improved, making the model more relevant when processing speech. The DFCNN network structure is as follows Figure 8 shown.

[0152] Finally, the Transformer language model is used to convert the pinyin into corresponding text content. First, the pinyin sequence output by the DFCNN acoustic model is used as input. It passes through the text embedding layer to convert the digital representation of the words in the text into a vector representation, and capture the relationship between words through high-dimensional space; then it passes through the position encoding layer. Different positions of words may produce different semantic information. The position encoding is added to the word embedding vector to make up for the lack of position information. The results of the two layers are added as the input of the encoder.

[0153] The encoder consists of 6 encoding units, each of which has a multi-head attention mechanism, a residual connection layer, a normalization layer, and a feedforward neural network layer. The multi-head attention mechanism is expressed as:

[0154]

[0155] Among them, Q, K, and V are query vectors, key vectors, and value vectors respectively, which are composed of the input vector and their respective weights W. Q 、W K 、W VThe weights are obtained by dot product and the model training after initialization. dk is the dimension of the K vector.

[0156]

[0157] MH(Q,K,V)=Concat(head1,...,head n )W Q #(10)

[0158] Among them, MH stands for MultiHead,

[0159] The residual connection layer transfers the gradient by adding the input directly to the output of the middle layer of the network, thereby alleviating the problem of the gradient gradually decreasing in deep networks. The normalization layer uses the batch normalization (BN) algorithm to normalize each batch of data in each layer, so that the input data of each layer has zero mean and unit variance, which helps to avoid the problem of data distribution offset and gradient explosion during training. The encoder sublayer also includes a feedforward neural network layer. The self-attention mechanism may have shortcomings in fitting complex processes, so two layers of linear networks are added to improve the expressive power of the model. This layer of network can be expressed as:

[0160] FFN(x)=max(0,xW1+b1)W2+b2 #(11)

[0161] Finally, the text with punctuation is saved to complete the speech-to-text recognition.

[0162] S4: Utilizes a 360-degree panoramic camera (composed of two wide-angle cameras and three high-definition cameras in S1) to capture panoramic images of the vehicle. Using a multi-stage cross-modal sound-image detection model, it detects pedestrians in the panoramic images and acquires their appearance and posture features.

[0163] Specifically, in one embodiment, a YOLOv8n target detection network is first constructed under the pytorch framework. The YOLOv8n target detection network consists of five modules: Backbone module, Neck module, Detection Head module, AnchorBoxes module, and Loss Function module.

[0164] The Backbone module in the YOLOv8n object detection network is a crucial component of the object detection model. Composed of convolutional layers, activation functions, and pooling layers, it extracts feature representations from input images through multi-scale feature extraction and fusion, enabling detection of objects of varying scales. Its working process includes steps such as convolutional feature extraction, the introduction of nonlinearity through activation functions, dimensionality reduction through pooling, and parameter sharing, ultimately providing the necessary semantic understanding and information representation for subsequent object detection tasks.

[0165] The Neck module processes features of different scales through a feature pyramid structure. It improves the expressiveness of features through multi-scale feature fusion and semantic enhancement, enabling accurate detection of objects of varying scales. Neck also achieves cross-scale feature alignment through downsampling and upsampling operations within the feature pyramid. Ultimately, Neck captures more local information through a fine-grained feature fusion mechanism.

[0166] The Detection Head module predicts the location and category of the target box through feature map processing and AnchorBoxes definition, and outputs the target detection result after screening and correction, thereby achieving accurate detection and positioning of the target.

[0167] The Anchor Boxes module is a set of predefined bounding boxes distributed across feature maps at different locations and scales, used to generate candidate boxes within an image. These Anchor Boxes, defined in different aspect ratios and sizes, cover a wide range of possible object shapes and scales. During object detection, each Anchor Box corresponds to a pixel location on the feature map, and the model predicts the location and category of the possible objects within each Anchor Box. This is compared with the actual location of the target box and, combined with the category probability, is filtered and corrected to achieve accurate object detection results.

[0168] The Loss Function module consists of multiple parts, including the coordinate loss of the target box location, the classification loss of the target category, and the matching loss between the target box and AnchorBoxes. The loss function works by calculating the difference between the predicted target box and the true target box, and weighting it with the category probability to minimize the error between the predicted box and the true box. In addition, the loss function also considers the weights of positive and negative samples and the confidence threshold to ensure effective training and adjustment for samples of different difficulty levels. By minimizing the overall loss function, the model is able to more accurately predict the location and category of the target.

[0169] Next, we created a custom feature detection dataset with 21 different label types, totaling nearly 20,000 labels. Considering that real-life passenger characteristics can vary significantly, we specifically added data on passengers of varying color depths and body types to enhance the model's robustness in real-world applications, enabling it to better handle passenger recognition tasks in various scenarios. The dataset is divided into training, validation, and test sets in a 7:2:1 ratio, and includes 2,000 additional negative samples.

[0170] Next, we use the labelimg tool to manually label the training set, and then perform 100 rounds of training to generate the best.pt model. The training model is then validated using the validation set and test set data. Figure 9 shown.

[0171] Finally, a multi-level cross-modal detection model of "sound-image" is used to obtain target ID information, coordinate information, and target key point information.

[0172] In one embodiment, see Figure 10 The multi-level cross-modal detection model of "sound-image" includes the following steps:

[0173] First, for the text of speech recognition in S3, the noun phrases are obtained by n-gram algorithm, and then the text embedding is extracted using the Transformer text encoder based on CLIP pre-training. Where T is the given text, C is the number of nouns, and D is the embedding dimension.

[0174] Then, multiple cameras at different locations on the vehicle are used to simultaneously obtain multiple real-time video streams and pass the video streams through the YOLOv8n Backbone to obtain multi-scale image features.

[0175] Then, W and X obtained in the above steps are l Feature fusion is performed through the Re-parameterizable Vision-Language Path Aggregation Network (RepVL-PAN), where the fusion process mainly includes text-guided cross-stage partial layers (T-CSPLayer) from text to image and image pooling attention mechanism (I-PoolingAttention) from image to text.

[0176] Finally, the fused target encoding is output through the Text Contrastive Head, and a decoupling head with two 3×3 convolutions is used to regress the bounding box. and object embedding By introducing the text comparison center, we can obtain the object-text similarity and output the coordinates of the object with the greatest similarity. The object encoding e and the text encoding w are represented using L2-Norm.

[0177] s k,j =α·L2-Norm(e k )·L2-Norm(wj ) T +β #(12)

[0178] Where L2-Norm() represents L2 normalization, W j ∈W is the j-th text embedding, α is the scaling factor, and β is the offset factor.

[0179] The n-gram algorithm includes the following steps: A sliding window of size N is applied to the text content, forming a sequence of byte segments of length N. Each byte segment is called a gram, and the frequency of occurrence of all the grams is counted and filtered according to a pre-set threshold to form a key gram list, thereby extracting noun phrases.

[0180] The CLIP-pretrained Transformer text encoder consists of two parts: the image vision transformer and the text transformer.

[0181] In the vision transformer, the image first passes through a patch convolution and a continuous residual attention structure, outputting a tensor of (1, 50, 768), and finally completing the mapping I1, I2, …, IN to (1, 512) through matrix multiplication.

[0182] In the Transformer, the text is tokenized to get a new tensor, and then passes through the embedding layer to get a (3, 77, 512) tensor. After adding the position code (nn.Parameter), it is fed into the model. Finally, the model outputs the (1, 512) features T1, T2, ..., T N .

[0183] By calculating I i and T jThe cosine similarity between the two is used to measure the correspondence between the corresponding text and the image. The larger the cosine similarity, the stronger the correspondence.

[0184]

[0185] The visual-language path convergence network described in Figure 11 , for images, multi-scale features X l Through downsampling and upsampling operations, combined with the embedding set W of the detection category vocabulary, T-CSPLayer is used to distribute attention. For text, the embedding set W of the detection category vocabulary is combined with 3 multi-scale feature maps to do max pooling and perform 3x3 area block division to obtain 27 feature maps. Based on multiHeadatattention, I-Pooling Attention is further used to distribute attention.

[0186] Among them, T-CSPLayer is used to calculate the multi-scale image features X l Use the embedding W of the detection category vocabulary to distribute attention and obtain the multi-scale feature output X' l as follows:

[0187]

[0188] Where δ represents the sigmoid function.

[0189] Among them, I-PoolingAttention uses multi-scale image features X to detect the embeddingW of category vocabulary l By allocating attention, the feature output W' is obtained as follows:

[0190]

[0191] S5: Design a passenger multi-attribute "static feature" matching method and a passenger "dynamic feature" recognition model based on skeleton features. Through text feature matching, match the passenger feature information based on speech recognition in S3 with the passenger feature information based on visual detection in S4.

[0192] In one embodiment, first, for the passenger multi-attribute "static feature" matching method, please refer to Figure 12The target information features obtained by S4 include the ID value and its corresponding two-point coordinates. Combined with the passenger feature text information input by the driver's voice, the target label set is first constructed through the input text information, and the target detection processing results of YOLOv8n in the video stream are read cyclically. The identified labels are matched with the target labels, and all possible label combinations are found through permutations and combinations.

[0193]

[0194] {set}({v i |(v i ,bi)∈c})={set}(box ids}#(16)

[0195] However, since the combination may come from different pedestrians, it is necessary to determine whether each possible combination belongs to the same object - the coordinates of the target box are enlarged (actual tests have determined that the appearance features are enlarged by 2 times and the building features are enlarged by 1.5 times), and then the area of ​​the intersection of each two bounding boxes is calculated.

[0196] interArea=max(0,min(xmaxA,xmaxB)-max(xminA,xminB))×max(0,min(ymaxA,ymaxB-max(yminA,yminB)) #(17)

[0197] in

[0198] min(xmaxA,xmaxB), max(xminA,xminB), min(ymaxA,ymaxB),

[0199] max(yminA,yminB) is the coordinate of the intersection of the two boxes, and interArea is the area of ​​the intersection.

[0200] Next, the corresponding IOU values ​​are calculated. If the IOU value is greater than a set threshold of 0.3, it indicates that they are the same person. The characteristic passenger is then found, and the actual location of the characteristic passenger is announced by voice based on the location of the average center point. This allows the passenger characteristic information based on voice recognition in S3 to be matched with the passenger characteristic information based on visual detection in S4, thus finding the characteristic passenger.

[0201]

[0202] Where interArea is the intersection area, and unionArea is the union area.

[0203] Next, for the passenger "dynamic feature" recognition model based on skeletal features, please refer to Figure 13 , the acquired data is filtered through the coordinates of the key points of the human body obtained by S4, and then the numpy array of the coordinates of all key points is obtained, that is, the (x, y) coordinate information corresponding to each key point of the human body.

[0204] Among them, in view of the different gestures of passengers waving to taxis in real life and the possible false triggering of pedestrians (non-potential passengers), a passenger "dynamic feature" recognition model based on skeletal features is established, and strict authentication indicators for hand-raising gestures are established - the wrist key point is higher than the shoulder key point and the height between the wrist, elbow and shoulder on the same side is within a certain threshold.

[0205] Set up the angle_between_points function to calculate the angles between key points. Use the key point coordinates obtained in the above process - shoulder point[5], point[6] elbow point[7], point[8] wrist point[9], point

[10] , pass them into the angle_between_points function, and calculate the angles between the three points on the left and right sides respectively.

[0206] The coordinates of the three points on the left and right are marked as [x1, y1], [x2, y2], [x3, y3]. Next, calculate the AB vector and the BC vector, which are AB = (x2-x1, y2-y1) and BC = (x3-x2, y3-y2). Then use the dot product of the vectors and the length formula to calculate the angle between the two vectors.

[0207]

[0208]

[0209] AB·BC=x1x2+y1y2 #(21)

[0210] Then we can get the cosine of the angle

[0211] Taking the arc cosine, we get

[0212] Convert radians to degrees

[0213]

[0214] Finally, if the angles between the three points are between 90° and 160° and the wrist y-value on the same side (wrist point[9], point

[10] shoulder point[5], point[6]) is greater than the shoulder y-value, it is considered a potential hand-waving gesture. Hand-waving gesture detection can be used to identify both characteristic passengers and potential passengers.

[0215] S6: Based on the matching results of S5, determine the passenger's position information; obtain the passenger's direction with the vehicle as the reference origin through position estimation in the panoramic image;

[0216] First, the average center coordinates of the judged passenger characteristics are obtained through S5, and a passenger position prediction scheme centered on the vehicle is established. The actual position of the passenger from the driver centered on the vehicle is determined by calculating the position of the average center point of the passengers.

[0217] Then, the specific location of the passenger is determined based on the judgment result, and the pyttsx3 module is used for text-to-speech function to make a voice announcement to inform the driver that he should slow down, that a potential passenger has appeared, and to inform the driver of his specific location.

[0218] Finally, based on the judgment results, the scope of the area to be intercepted is determined - the position of the area to be intercepted is determined according to the coordinates of the key points, and the area is intercepted in the original image according to the determined area range, and the area image where the key part is located is obtained. The cv2.resize function is used to scale and adjust the appropriate size, and the placement of the screen is flexibly adjusted according to the actual position. In the video stream processing loop, the content of the small screen is updated according to the real-time recognition results, and the updated small screen is drawn on the original image to achieve dynamic display, which is convenient for the driver to promptly confirm whether it is a characteristic passenger.

[0219] S7: Based on the direction estimation result of S6, control the projector in the vehicle to project the passenger information onto the screen in the corresponding direction in the vehicle, and give a prompt using voice information at the same time.

[0220] Specifically, when passenger location is not being determined, an image encompassing blind spots is projected, creating a perspective effect. This helps drivers observe their surroundings during high-traffic situations and reduces the likelihood of collisions. The video stream processing loop updates the image content in real time, enabling dynamic display. The passenger's specific location is determined based on the results, and text-to-speech conversion using the pyttsx3 module is used to announce the driver's need to slow down, identify the presence of characteristic and potential passengers, and provide their specific location.

[0221] Finally, when executing the system function, the vehicle-mounted camera and projector are combined with the system to achieve visual perspective. Use multiple vehicle-mounted cameras placed next to the car door to collect 360° image data, and use OpenCV to transmit the visual processing results to the projector. The projector is placed on the opposite side of the projection cloth, and the image is projected onto the screen close to the inside of the car body. It can display multiple features of passengers specially marked by OpenCV, and use small screens placed at different positions of the image to specifically display different features, perform visual warnings, and realize rapid positioning of passengers. The perspective diagram is as follows Figure 14 shown.

[0222] For different usage scenarios, please refer to Figure 15 , Figure 15 This is an example diagram of the test results of a method for quickly locating the position of online ride-hailing passengers based on voice interaction and visual perspective proposed in the present invention.

[0223] This application also provides a rapid positioning model for online ride-hailing passengers based on voice interaction and visual perspective, which is constructed using the positioning method described above.

[0224] This application provides a method for quickly locating the position of online car-hailing passengers based on voice interaction and visual perspective. It uses multiple on-board cameras to perceive the scene outside the car to achieve car perspective, avoid blind spot accidents and collect data in real time. Then, the system is awakened by the "one-word wake-up" function, and a scenario-based voice recognition model is built to recognize the driver's voice information. The data is then processed using a multi-level cross-modal detection method of "sound-image", a passenger dynamic feature recognition model based on skeletal features, and a passenger multi-attribute "static feature" matching method. Finally, the results are fed back to the projector for display at the car door and voice broadcast. The system can realize driver voice operation, reduce the phenomenon of "people looking for cars and cars looking for people" in online car-hailing, and alleviate congestion problems.

Claims

1. A method for quickly locating a ride-hailing passenger's position based on voice interaction and visual perspective, characterized by: The following steps are involved: S1: Deploy sensors, cameras, metal reflective screens, and projectors on the online ride-hailing vehicle. Five cameras are provided, located at the right front door, right rear door, front roof, rear roof, and rear of the vehicle, respectively. The metal reflective screens are located at the front door, rear door, and rear seatback, and the projector is located diagonally above the reflective screens. S2: Utilize the vehicle's microphone to continuously collect the driver's voice information in the background. Use Transformer as the voice wake-up word detection model and extract voice features to design an intelligent voice wake-up model to implement the "one-word wake-up" function for voice interaction. S3: Build a scenario-based speech recognition model to automatically identify the appearance and movement characteristics of passengers by recognizing the driver's voice information and save the text information of these characteristics. S4: Use the camera to obtain a panoramic image outside the vehicle. Use the "sound-image" multi-level cross-modal detection model to detect pedestrian targets in the panoramic image and obtain detection information of the appearance and posture characteristics of pedestrian targets. The "sound-image" multi-level cross-modal detection model includes the following steps: First, for the text of speech recognition in S3, the noun phrases are obtained by n-gram algorithm, and then the text embedding is extracted using the Transformer text encoder based on CLIP pre-training. Where T is the given text, C is the number of nouns, and D is the embedding dimension; Then, multiple cameras at different locations on the vehicle are used to simultaneously obtain multiple real-time video streams and pass the video streams through the YOLOv8n Backbone to obtain multi-scale image features. Then, W and X obtained in the above steps are l Feature fusion is performed through a reparameterized vision-language path aggregation network, where the fusion process mainly includes a text-guided cross-block bias layer from text to image and an image pooling attention mechanism from image to text; Finally, the fused target encoding is output through the Text Contrastive Head, and a decoupling head with two 3×3 convolutions is used to regress the bounding box. and object embedding And by introducing the text comparison center, we can obtain the object-text similarity, and output the coordinates of the object with the greatest similarity; the target code e and the text code w are represented by L2-Norm; Where L2-Norm() represents L2 normalization, W j ∈W is the jth text embedding, α is the scaling factor, and β is the offset factor; S5: Design a passenger multi-attribute "static feature" matching method and a passenger "dynamic feature" recognition model based on skeleton features. Through text feature matching, match the passenger feature information based on voice recognition in S3 with the passenger feature information based on visual detection in S4. The multi-attribute static features include static features from visual recognition and appearance features from voice recognition. S6: Based on the matching results of S5, determine the passenger's position information; obtain the passenger's direction with the vehicle as the reference origin through position estimation in the panoramic image; S7: Based on the direction estimation result of S6, control the projector in the vehicle to project the passenger information onto the screen in the corresponding direction in the vehicle, and give a prompt using voice information at the same time.

2. The method for quickly locating a ride-hailing passenger based on voice interaction and visual perspective according to claim 1, characterized in that: The specific steps of S2 are as follows: First, the spectrogram is extracted from the input audio and then divided into sub-blocks, represented as X∈R T×F , where the time domain window t={1,2,…T}, the frequency domain f={1,2,…F}; the sub-block first passes through a linear projection matrix W0∈R in the frequency domain f×d Map to a high-dimensional space d, and then add a learnable category embedding in the time domain order Used to learn the global information of the spectrogram, and finally add a learnable position matrix As shown in formula (1), the processed X1 is fed into the Transformer encoder; X1=[X class ;XW1]+X pos #(1) Then, the obtained embedding features are input into a sequential Transformer encoder consisting of L multi-head attention and multi-layer perceptron blocks; in the lth Transformer, Q = X is calculated respectively. l W Q , K=X l W K and V = X l W K ;in, d h is the dimension of each attention head; the calculation of self-attention is shown in formula (2): MSA through the matrix The calculation is as follows: MSA(X l )=[SA1(X l );SA2(X l );…;SA k (X l )]W p #(3) The output of the lth Transformer is: Finally, during streaming inference, a sliding window is used to cut the audio stream into sub-blocks for inference; the sliding window is a rectangular window that slides along the time axis with some overlap; Transformer detects each sub-block and scores it y i ; Once a y appears i >γ, indicating that the wake-up word is triggered, that is, the driver wakes up the system; the threshold γ∈(0,1).

3. The method for quickly locating a ride-hailing passenger based on voice interaction and visual perspective according to claim 1, characterized in that: The specific steps of S3 are as follows: First, the real-time input audio file can be a local file or a file recorded using the recording function, and the voice signal is segmented into frames through VAD; Mel-frequency cepstral coefficient features are then extracted from each frame, and a classification model is trained using a set of data frames with known speech and silence signal regions. The trained model is then used to classify unknown audio signals into spoken, unspoken, and silent parts. The detected breakpoints, that is, the start and end time of each pronunciation segment, will be saved in a List collection; ultimately, this List value and the original voice file are used as the input of speech recognition for subsequent recognition operations; Then, the DFCNN acoustic model converts the speech signal into the corresponding pinyin information. The specific steps are as follows: A1: First, convert the obtained time-domain audio signal into a spectrogram. The signal needs to be subjected to a short-time discrete Fourier transform, as shown in formula (6): The window length is expressed as N, w(n) is the Hamming window function, f s represents the sampling frequency; as in formula (7), the spectrogram is generated by logarithmic amplitude operation; S(k,t)=20×log 10 |X t (k)| #(7) A2: The generated spectrogram is used as the input of the DFCNN model. The DFCNN model consists of two groups of "convolutional layer-convolutional layer-pooling layer" units connected in series with a fully connected layer. Among them, the convolutional layer of each unit uses a small 3×3 convolution kernel, and a pooling layer is added after two layers of convolution to complete the construction of the DFCNN model. The entire sentence of speech is modeled through the combination of two groups of "convolutional layer-convolutional layer-pooling layer", and the output unit directly corresponds to the final recognition result. A3: The pinyin is converted into the corresponding text content through the Transformer language model, and the pinyin sequence output by the DFCNN acoustic model is used as input. It first passes through the text embedding layer to convert the digital representation of the words in the text into a vector representation, and captures the relationship between words through high-dimensional space. Then it passes through the position encoding layer to add the position encoding to the word embedding vector, and the two layers of results are added together as the input of the encoder. The encoder consists of 6 encoding units, each of which has a multi-head attention mechanism, a residual connection layer, a normalization layer, and a feedforward neural network layer; the multi-head attention mechanism is expressed as: Among them, Q, K, and V are query vectors, key vectors, and value vectors respectively, which are composed of the input vector and their respective weights W. Q 、W K 、W V The weights are obtained by point multiplication after initialization and model training; dk is the dimension of the K vector; MH(Q,K,V)=Concat(head1,…,head n )W Q #(10) Among them, MH stands for MultiHead, The residual connection layer transfers the gradient by adding the input directly to the output of the intermediate layer of the network; The normalization layer adopts a batch normalization algorithm; The encoder sublayer also contains a feedforward neural network layer, which can be expressed as: FFN(x)=max(0,xW1+b1)W2+b2 #(11) Finally, the text with punctuation is saved to complete the speech-to-text recognition.

4. The method for quickly locating a ride-hailing passenger based on voice interaction and visual perspective according to claim 1, characterized in that: The specific steps of S4 are as follows: First, we build a YOLOv8n target detection network under the pytorch framework. The YOLOv8n target detection network consists of five modules: Backbone module, Neck module, Detection Head module, AnchorBoxes module, and LossFunction module. Then, we created a custom feature detection dataset. We specifically added data on passengers of different color depths and body types to the dataset. The dataset was divided into training, validation, and test sets in a ratio of 7:2:1, and 2,000 additional negative samples were added. Next, we manually labeled the training set using the labelimg tool, and then performed 100 rounds of training to generate the best.pt model. We then used validation and test set data to verify the effectiveness of the trained model. Finally, a multi-level cross-modal detection model of "sound-image" is used to obtain target ID information, coordinate information, and target key point information.

5. The method for quickly locating a ride-hailing passenger based on voice interaction and visual perspective according to claim 4, characterized in that: The Backbone module in the YOLOv8n target detection network consists of a convolutional layer, an activation function, and a pooling layer. It extracts feature representations from the input image through multi-scale feature extraction and feature fusion to detect targets of different scales. Its working process includes convolutional feature extraction, nonlinearity introduced by activation function, dimensionality reduction by pooling, and parameter sharing, ultimately providing the necessary semantic understanding and information expression for subsequent target detection tasks; The Neck module processes features of different scales through a feature pyramid structure and improves the expressiveness of features through multi-scale feature fusion and semantic enhancement to achieve accurate detection of objects of different scales. The Neck module also achieves cross-scale feature alignment through downsampling and upsampling operations of the feature pyramid. Finally, the Neck module captures more local information through a fine-grained feature fusion mechanism. The Detection Head module predicts the location and category of the target box through feature map processing and AnchorBoxes module definition, and outputs the target detection result after screening and correction, thus achieving accurate detection and positioning of the target; The AnchorBoxes module is a series of predefined bounding boxes distributed at different positions and scales in the feature map, used to generate candidate boxes in the image. These AnchorBoxes cover a variety of possible object shapes and scales by defining different aspect ratios and sizes. During the object detection process, each AnchorBox module corresponds to a pixel position on the feature map, and the model predicts the position and category information of the possible objects within each AnchorBox module. By comparing the position of the actual target frame and combining it with the category probability for screening and correction, we can finally get accurate target detection results. The Loss Function module includes the coordinate loss of the target box position, the classification loss of the target category, and the matching loss between the target box and the Anchor Boxes. The working process of the loss function involves calculating the difference between the predicted target box and the true target box, and weighting it with the category probability to minimize the error between the predicted box and the true box. In addition, the loss function also considers the weights of positive and negative samples and the confidence threshold to ensure effective training and adjustment of samples of different difficulty levels. By minimizing the overall loss function, the model can more accurately predict the location and category of the target.

6. The method for quickly locating a ride-hailing passenger based on voice interaction and visual perspective according to claim 1, characterized in that: The n-gram algorithm includes the following steps: performing a sliding window operation of size N on the byte content of the text to form a sequence of byte segments of length N; each byte segment is called a gram, and the occurrence frequency of all grams is counted and filtered according to a pre-set threshold to form a key gram list, thereby realizing the extraction of noun phrases; The Transformer text encoder based on CLIP pre-training consists of two parts: the vision transformer responsible for images and the transformer responsible for text; In the vision transformer, the image first undergoes a patch convolution and a continuous residual attention structure, outputting a tensor of (1, 50, 768), and finally completing the mapping I1, I2, ..., IN to (1, 512) through matrix multiplication; In the Transformer, the text is tokenized to get a new tensor, and then passes through the embedding layer to get a (3, 77, 512) tensor. After adding the position code (nn.Parameter), it is fed into the model. Finally, the model outputs the (1, 512) features T1, T2, ..., T N ; By calculating I i and T j The cosine similarity between them is used to measure the correspondence between the corresponding text and the image. The larger the cosine similarity, the stronger the correspondence. The visual-language path aggregation network, for images, multi-scale features X l Through downsampling and upsampling operations, combined with the embedding set W of the detection category vocabulary, T-CSPLayer is used to allocate attention. For text, the embedding set W of the detection category vocabulary is combined with three multi-scale feature maps to do max pooling for 3x3 area block division, and 27 feature map patches are obtained. And based on multiHead attention, I-Pooling Attention is further used to distribute attention; Among them, T-CSPLayer is used to calculate the multi-scale image features X l Use the embedding W of the detection category vocabulary to distribute attention and obtain the multi-scale feature output X' l as follows: Where δ represents the sigmoid function; Among them, I-PoolingAttention uses multi-scale image features X to detect the embeddingW of category vocabulary l By allocating attention, the feature output W' is obtained as follows: 。 7. The method for quickly locating a ride-hailing passenger based on voice interaction and visual perspective according to claim 1, characterized in that: In S5, a novel passenger multi-attribute "static feature" matching method is developed based on the different appearance features of different passengers: First, the target information features obtained in S4, including the ID value and its corresponding two-point coordinates, are combined with the passenger feature text information input by the driver's voice. First, a target label set is constructed based on the input text information. The target detection processing results of YOLOv8n in the video stream are read cyclically, and the recognized labels are matched with the target labels. All possible label combinations are found through permutations and combinations: Enlarge the coordinates of the target box and then calculate the intersection area of ​​every two bounding boxes; interArea=max(0,min(xmaxA,xmaxB)-max(xminA,xminB))×max(0,min(ymaxA,ymaxB-max(yminA,yminB)) ##(17)where min(xmaxA,xmaxB), max(xminA,xminB), min(ymaxA,ymaxB), max(yminA,yminB) is the coordinate of the intersection of the two boxes, and interArea is the area of ​​the intersection; Then, the corresponding IOU value is calculated. If the IOU value is greater than the set threshold of 0.3, it means that they are the same person. Then the characteristic passenger is found and the actual position of the characteristic passenger is announced according to the position of the average center point. Thus, the passenger feature information based on voice recognition in S3 is matched with the passenger feature information based on visual detection in S4 to find the characteristic passenger; Among them, interArea is the area of ​​the intersection area, and unionArea is the area of ​​the union; Next, the acquired data is filtered using the coordinates of the key points of the human body obtained by S4 to obtain the numpy array of the coordinates of all key points, that is, the (x, y) coordinate information corresponding to each key point of the human body; In particular, to address the varying gestures passengers use when waving taxis in real life and the potential for false triggering by pedestrians, a passenger "dynamic signature" recognition model based on skeletal features was established. Strict hand-raising gesture authentication criteria were established: the wrist keypoint must be higher than the shoulder keypoint, and the height between the wrist, elbow, and shoulder on the same side must be within a certain threshold. Set up the angle_between_points function to calculate the angles between key points; use the key point coordinates obtained in the above process - shoulder point[5], point[6] elbow point[7], point[8] wrist point[9], point[10], pass them into the angle_between_points function, and calculate the angles between the three points on the left and right respectively; The coordinates of the three points on the left and right are marked as [x1, y1], [x2, y2], [x3, y3]; then calculate the AB vector and the BC vector, which are AB = (x2-x1, y2-y1) and BC = (x3-x2, y3-y2) respectively; then use the dot product of the vectors and the length formula to calculate the angle between the two vectors. AB·BC=x1x2+y1y2 #(21) Then we can get the cosine of the angle Taking the arc cosine, we get Convert radians to degrees Finally, if the angles between the three points are between 90° and 160° and the wrist y value on the same side (wrist point[9], point[10] shoulder point[5], point[6]) is greater than the shoulder y value, it is judged as a potential hand-raising action to hail a taxi.

8. The method for quickly locating a ride-hailing passenger based on voice interaction and visual perspective according to claim 1, characterized in that: In said S6, the average center coordinates of the determined passenger characteristics obtained in S5 are used to establish a passenger position prediction scheme centered on the vehicle, and the actual position of the passenger from the driver centered on the vehicle is determined by calculating the position of the average center coordinates of the passengers; The specific location of the passenger is determined based on the judgment result, and the pyttsx3 module is used for text-to-speech function to make a voice announcement to inform the driver that he should slow down, that a potential passenger has appeared, and inform the driver of the specific location; Determine the position of the area to be captured based on the coordinates of the key points, and capture it in the original image based on the determined area range. Get the regional image where the key part is located, and use the cv2.resize function to scale it to the appropriate size. According to the actual position, flexibly adjust the placement of the screen. In the video stream processing loop, update the small screen content according to the real-time recognition results, and draw the updated small screen on the original image to achieve dynamic display.

9. The method for quickly locating a ride-hailing passenger based on voice interaction and visual perspective according to claim 1, characterized in that: In the S7: When the passenger is not located, the image including the blind spot is projected to achieve the perspective effect of the blind spot. The image content is updated in real time during the video stream processing loop to achieve dynamic display. At the same time, the specific location of the passenger is determined based on the judgment result, and the pyttsx3 module is used for text-to-speech function to inform the driver to slow down, the presence of characteristic passengers and potential passengers, and inform the driver of their specific location. When executing system functions, the vehicle-mounted camera and projector are combined with the system to achieve visual perspective. Multiple vehicle-mounted cameras are placed next to the vehicle door to collect 360-degree image data, and the visual processing results are transmitted to the projector using OpenCV. The projector is placed on the opposite side of the projection cloth, and projects the image onto a screen close to the inside of the car body. It can display multiple features of passengers specially marked by OpenCV, and use small screens placed at different positions of the image to specifically display different features, provide visual warnings, and realize rapid positioning of passengers.

Images