Safe travel navigation method and device, computer equipment and storage medium

Abstract

The invention discloses a safe travel navigation method and device, computer equipment and a storage medium, and relates to the technical field of artificial intelligence and safe travel, the method comprises the following steps: obtaining multi-modal information, and preprocessing the multi-modal information, the multi-modal information comprising image data, voice data and position data; feature extraction is carried out on the preprocessed multi-modal information, and feature information of corresponding modals is obtained; performing multi-modal information fusion processing on the extracted feature information to obtain fused feature information; performing output processing on the fused feature information to obtain a security risk type, a risk assessment score, a current position of the user and predicted route time consumption; and providing real-time early warning information according to the safety risk type and the risk assessment score, and providing a safety route according to the current position of the user and the predicted route time consumption. According to the method, the accuracy of safety detection is improved, and it is ensured that an optimal travel path and safety early warning are provided for a user in a complex environment.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence and safe travel, and particularly to a safe travel navigation method, device, computer device and storage medium. Background Art

[0002] In order to ensure the safe travel of users based on augmented reality (AR) technology and combine the relevant needs in the field of medical and health (such as the elderly care field), developing an intelligent navigation and safety warning system using artificial intelligence is an effective strategy. Given the scarcity of abnormal behavior or safety risk labels, few-shot learning artificial intelligence models are particularly important in such cross-domain applications in multiple fields. It not only widely meets the needs of users in travel safety monitoring but also deeply touches the key areas of medical and health management and elderly care. By applying few-shot learning technology, the safety risk detection process during users' travel can be simplified, thereby ensuring their travel safety and enhancing their trust in technology.

[0003] Currently, few-shot learning artificial intelligence has achieved certain technological breakthroughs in the fields of speech recognition, image processing, and position tracking. These technologies are relatively intuitive and mainly achieve their goals through the analysis and processing of speech, video, and position data, without complex reasoning and judgment, thus simplifying the implementation steps. However, speech recognition and image analysis also face challenges such as environmental noise interference, visual occlusion, and misrecognition.

[0004] Since users' travel scenarios are often more complex, involving multiple modal information such as images, speech, and position data, and at the same time, multiple factors such as real-time traffic conditions and road environment changes need to be considered. For artificial intelligence models using few-shot learning, how to effectively process this multi-modal data and cope with complex real-world environments has become a major problem. Therefore, it is particularly important to introduce models that can process multi-modal information, have context connection and deduction capabilities to ensure that users can obtain the safest route selection and real-time safety warnings when traveling. Summary of the Invention

[0005] The purpose of the present invention is to provide a safe travel navigation method, device, computer device and storage medium, aiming to solve the problem that the artificial intelligence models using few-shot learning in the prior art cannot effectively process multi-modal data and cope with complex real-world environments.

[0006] In a first aspect, an embodiment of the present invention provides a safe travel navigation method based on multi-modal information fusion, including:

[0007] Obtain multi-modal information and preprocess the multi-modal information, where the multi-modal information includes image data, voice data, and position data;

[0008] Extract features from the preprocessed multi-modal information to obtain feature information corresponding to each modality;

[0009] Perform multi-modal information fusion processing on the extracted feature information to obtain fused feature information;

[0010] Perform output processing on the fused feature information to obtain the type of security risk, risk assessment score, user's current location, and estimated route time;

[0011] Provide real-time warning information according to the type of security risk and risk assessment score, and provide a safe route according to the user's current location and estimated route time.

[0012] In a second aspect, an embodiment of the present invention further provides a safe travel navigation device based on multi-modal information fusion, which includes:

[0013] An acquisition unit for acquiring multi-modal information and preprocessing the multi-modal information, where the multi-modal information includes image data, voice data, and location data;

[0014] A feature extraction unit for extracting features from the preprocessed multi-modal information to obtain feature information corresponding to each modality;

[0015] A fusion unit for performing multi-modal information fusion processing on the extracted feature information to obtain fused feature information;

[0016] An output unit for performing output processing on the fused feature information to obtain the type of security risk, risk assessment score, user's current location, and estimated route time;

[0017] A safe route providing unit for providing real-time warning information according to the type of security risk and risk assessment score, and providing a safe route according to the user's current location and estimated route time.

[0018] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the safe travel navigation method based on multi-modal information fusion described in the first aspect above.

[0019] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the safe travel navigation method based on multi-modal information fusion described in the first aspect above.

[0020] An embodiment of the present invention provides a safe travel navigation method, device, computer device, and storage medium. By adopting multi-modal information fusion technology, this method combines different types of data including images, voices, positional sounds, etc., and can more accurately extract complex feature representations. It can not only analyze the user's travel environment and behavior more comprehensively, but also capture the details and potential safety risks during the user's travel, improving the accuracy of safety warnings. In addition, the method based on the multi-modal large model can provide strong mathematical deduction capabilities, further improving the accuracy of safety detection and ensuring the provision of the optimal travel route and safety warning for users in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for the description of the embodiments. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0022] Figure 1 It is a schematic diagram of an application environment of the safe travel navigation method based on multi-modal information fusion provided in the embodiment of the present invention;

[0023] Figure 2 It is a schematic flowchart of the safe travel navigation method based on multi-modal information fusion provided in the embodiment of the present invention;

[0024] Figure 3 is Figure 2 a schematic flowchart of a specific implementation manner of step S101 in;

[0025] Figure 4 is Figure 2 a schematic flowchart of a specific implementation manner of step S102 in;

[0026] Figure 5 It is a schematic diagram of the framework structure of the multi-modal large model based on few-shot learning provided in the embodiment of the present invention;

[0027] Figure 6 is Figure 2 another schematic flowchart of a specific implementation manner of step S105 in;

[0028] Figure 7 It is a schematic diagram of the structure of the safe travel navigation device based on multi-modal information fusion provided in the embodiment of the present invention;

[0029] Figure 8 It is a schematic diagram of the structure of a computer device provided in the embodiment of the present invention;

[0030] Figure 9It is another schematic structural diagram of the computer device provided in the embodiment of the present invention. Detailed implementation manners

[0031] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0032] It should be understood that when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0033] It should also be understood that the terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0034] It should be further understood that the term "and / or" used in the specification of the present invention and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0035] The safe travel navigation method based on multimodal information fusion provided by the embodiment of the present invention can be applied to an application environment such as Figure 1 , where the client communicates with the program end through the network. The program end can obtain multimodal information through the client and preprocess the multimodal information. The multimodal information includes image data, voice data, and location data; feature extraction is performed on the preprocessed multimodal information to obtain feature information corresponding to each modality; the extracted feature information is subjected to multimodal information fusion processing to obtain fusion feature information; the fusion feature information is subjected to output processing to obtain a safety risk type, a risk assessment score, the current location of the user, and the estimated route time; according to the safety risk type and the risk assessment score, real-time warning information is provided, and a safe route is provided according to the current location of the user and the estimated route time, etc.

[0036] In the present invention, for complex insurance entities in the insurance business, images of the surrounding environment are captured by a camera installed on a user's wearable device (such as a smart watch or smart glasses), including road conditions, pedestrian density, traffic signal status, etc. The preprocessing stage includes denoising, enhancing contrast, edge detection, etc., to facilitate subsequent feature extraction. The built-in microphone of the device is used to collect the user's voice commands or conversation content in real time. The preprocessing includes speech recognition, noise filtering, and conversion to text format to facilitate the analysis of the user's emotional state or emergency needs. Through the GPS or Beidou positioning system, the user's real-time location information is obtained, and the road conditions, weather, etc. of the location where the user is located are determined in combination with map data. After preprocessing these multi-modal information (such as image denoising, speech noise reduction, location calibration), feature extraction is performed separately: a convolutional neural network (CNN) is used to extract features such as road type and obstacles from the image, a speech recognition model is used to extract the emotional state or distress signal from the speech, and road conditions, weather, and nearby medical facility information are extracted from the location data. Subsequently, the system fuses these features through a multi-modal fusion model (such as a model based on the attention mechanism) to generate comprehensive fused feature information, and outputs the type of safety risk (such as slippery road, physical discomfort), risk assessment score, the user's current location, and the estimated time consumption of the planned route. Based on this information, the system provides real-time warnings (such as reminding the user to pay attention to safety or contact family members) and plans a safe route (such as avoiding dangerous sections or choosing a path close to medical facilities). For example, when the user goes out shopping alone, the system recognizes construction obstacles ahead through image recognition, the user's dizzy state through speech recognition, locates the nearby hospital through location, and issues a warning and recommends a safe route after comprehensive evaluation; or when the user goes to a nursing home, the system recognizes slippery roads and traffic congestion, reminds the user to pay attention to safety and recommends an alternative route. Through multi-modal information fusion, the system can comprehensively evaluate travel risks, provide personalized safety suggestions in combination with medical and health data, and effectively improve the travel safety and quality of life of users.

[0037] For application scenarios in the medical field (such as elderly care), by adopting multi-modal information fusion technology, different types of data including images, voices, location sounds, etc. are combined, solving the problem that the artificial intelligence model using few-shot learning in the prior art cannot effectively process multi-modal data and cope with complex real-world environments. At the same time, it can analyze the user's travel environment and behavior more comprehensively, and can also capture the details and potential safety risks during the user's travel, improving the accuracy of safety warnings and further improving the accuracy of safety detection, ensuring to provide the optimal travel route and safety warning for users in complex environments. Among them, the client can be, but is not limited to, various personal computers, laptops, smart phones, tablets, and portable wearable devices. The program end can be implemented by an independent server or a server cluster composed of multiple servers. The present invention will be described in detail below through specific embodiments.

[0038] Please refer to Figure 2 , Figure 2 , which is a schematic flowchart of a safe travel navigation method based on multi-modal information fusion provided by an embodiment of the present invention. The method includes steps S101 to S105:

[0039] S101. Obtain multi-modal information and preprocess the multi-modal information. Among them, the multi-modal information includes image data, voice data, and location data;

[0040] In this embodiment, multi-modal information can be obtained through devices that can be carried at any time, such as AR devices, AR glasses, AR helmets, smart bracelets, smart watches, smart crutches, or other electronic devices. Specifically, the image information around the user is captured in real time through the camera of the AR device, the voice commands of the user or the sounds of the surrounding environment are captured by the microphone of the AR device, and the real-time location information of the user is obtained through the built-in GPS module of the AR device or in combination with other positioning software. As for the processing of multi-modal information, it can be implemented on the device side or sent to the server side for implementation, and then the processing result is returned to the device side. For example, the smart crutch collects multi-modal information and sends it to the server side for processing, and the server side then returns the processing result to the AR device. Subsequently, these original multi-modal information are preprocessed to improve the data quality and the accuracy of subsequent processing.

[0041] Among them, as Figure 3 shown, in step S101, the preprocessing of the multi-modal information includes the following steps S201 to S203:

[0042] S201. Perform a scaling operation on the image data and adjust its contrast to obtain preprocessed image data i pre ;

[0043] S202. Perform noise reduction processing on the voice data to obtain preprocessed voice data s pre ;

[0044] S203. Perform noise filtering processing on the location data and convert the coordinates in the location data into local coordinates to obtain preprocessed location information p pre .

[0045] For steps S201 - S203, the purpose of scaling the image data is to unify the image size to facilitate subsequent processing or analysis requirements. Scaling involves proportional scaling while maintaining the aspect ratio of the image, or non - proportional scaling according to specific needs. Adjusting the contrast of the image is to enhance the visual effect of the image and make the details in the image clearer. Specifically, the contrast adjustment can be achieved by stretching or compressing the bright and dark regions of the image to obtain the desired visual effect, or the bilinear interpolation algorithm can be used for image scaling, and the histogram equalization method can be used to adjust the image contrast.

[0046] Furthermore, the purpose of denoising the voice data is to remove the background noise in the voice data and improve the clarity of the voice. Denoising methods include filtering, spectral subtraction, machine learning algorithms, etc., and the specific selection depends on the type and characteristics of the noise.

[0047] Furthermore, the position data may contain noise introduced by factors such as sensor errors and environmental interference. The noise filtering process aims to remove this noise and improve the accuracy of the position data. Converting the coordinates in the position data to local coordinates is to unify the position data into a specific reference system. Specifically, convert the Global Positioning System (GPS) coordinates to local map coordinates, or convert the coordinates between different coordinate systems. It should be noted that the coordinate information in the position data can also be obtained through other navigation software such as the Beidou Navigation System.

[0048] In summary, by pre - processing the image data, voice data, and position data, the quality and usability of the data can be improved. In addition, it can provide a more accurate and reliable data basis for subsequent analysis, recognition, or applications.

[0049] S102. Extract features from the pre - processed multi - modal information to obtain feature information corresponding to each modality;

[0050] In this embodiment, features are extracted from the pre - processed multi - modal information respectively. For image data, visual features such as edges, textures, and colors need to be extracted. For voice data, audio features such as pitch, speech rate, and volume need to be extracted. For position data, motion features such as speed, acceleration, and direction need to be extracted.

[0051] Among them, as Figure 4 shown, in step S102, extracting features from the pre - processed multi - modal information includes the following steps S301 - S302:

[0052] S301. Process the pre - processed image data through an image feature extraction module to obtain image feature information F i :

[0053] F i = Φ i (i pre )

[0054] where Φ i is the image feature extraction module, and i pre is the preprocessed image data;

[0055] S302. Process the preprocessed speech data through the speech feature extraction module, and obtain the speech feature information F according to the following formula s :

[0056] F s = Φ s ( pre )

[0057] where Φ s is the speech feature extraction module, and s pre is the preprocessed speech data.

[0058] The image feature extraction module uses a Vision Transformer model, and the speech feature extraction module uses a Whisper model

[0059] For steps S301 - S302, by separately extracting the features of the image data and the speech data, the feature information of the image and the speech can be effectively obtained, providing a basis for subsequent multi-modal information fusion. The image feature extraction module uses a Vision Transformer model, which has high accuracy and robustness in processing image data. The speech feature extraction module uses a Whisper model, which can accurately extract speech feature information in a complex environment.

[0060] For step S301, the image feature extraction module can adopt different implementation methods. For example, a convolutional neural network (CNN) can be used to process the image data and extract the spatial features of the image. The Vision Transformer model can capture the global information in the image through the self-attention mechanism, improving the effect of feature extraction.

[0061] In one embodiment, in the Vision Transformer (ViT architecture) model, a residual attention mechanism is introduced. Specifically, by integrating the residual attention learning into the ViT architecture, the extraction of visual features can be enhanced. This residual connection can prevent the reduction of low-level visual features and, by slowing down the globalization of the attention mechanism, endow the model with the ability to utilize the previously extracted features when learning new features.

[0062] For step S302, the speech feature extraction module uses the Whisper model to extract the speech feature information F s ​​Extraction. The speech feature extraction module can also adopt traditional speech processing algorithms, such as Mel Frequency Cepstral Coefficients (MFCC) extraction, or deep learning models, such as Long Short-Term Memory Network (LSTM) or Transformer model, to improve the accuracy of speech feature extraction.

[0063] S103. Perform multi-modal information fusion processing on the extracted feature information to obtain fused feature information;

[0064] In this embodiment, the system performs multi-modal fusion processing on the extracted feature information to integrate complementary information between different modalities. The fusion process can adopt deep learning algorithms or machine learning algorithms to generate more comprehensive and accurate fused feature information.

[0065] Specifically, deep learning algorithms or machine learning algorithms are used to fuse the feature information to obtain fused feature information. Deep learning algorithms include Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) and its variants, Transformer model, and multi-modal fusion algorithms.

[0066] Specifically, CNN performs well in processing data with grid structure (such as images). It can extract features of the input data through structures such as convolutional layers, pooling layers, and fully connected layers. In CNN, features at different levels can be fused to capture low-level detailed features and high-level abstract features. For example, in image segmentation tasks, low-level detailed features and high-level semantic features can be fused to improve the accuracy of segmentation.

[0067] RNN and its variants (such as Long Short-Term Memory Network LSTM and Gated Recurrent Unit GRU) have advantages in processing sequence data. They can capture the temporal dependence in sequence data. Although RNN itself is more used for processing sequence data, in some cases, RNN can also be combined with other types of feature extractors (such as CNN) to fuse feature information from different sources or different time steps.

[0068] The Transformer model can process variable-length sequence data through self-attention mechanism and position encoding, etc., and has achieved remarkable results in fields such as natural language processing and image recognition. In terms of feature fusion, the Transformer model can perform weighted processing on features from different sources or different levels through the self-attention mechanism, thereby realizing feature fusion.

[0069] Fusion algorithms specifically designed for multimodal data, such as multimodal Transformers, multimodal attention networks, etc. These algorithms can simultaneously process data from different modalities (such as images, speech, text, etc.) and fuse the features of different modalities through specific fusion mechanisms (such as attention mechanisms, gating mechanisms, etc.).

[0070] Furthermore, algorithms such as support vector machines (SVM) or random forests can be used to further process and fuse the extracted features. Specifically, in the feature processing stage, the main task is to preprocess the extracted features to better adapt to subsequent algorithm models. This includes:

[0071] Data standardization or normalization: Convert the feature data into a standard format and perform normalization or standardization processing to eliminate the dimensional differences between different features and improve the convergence speed and performance of the algorithm.

[0072] Feature selection: Select the most useful features for classification or regression tasks from the original features to reduce the data dimension and improve the generalization ability of the model. This can be achieved through statistical methods, machine learning algorithms, or domain knowledge.

[0073] Feature dimensionality reduction: If the feature dimension is too high, it may lead to an increase in computational complexity and overfitting risk. In this case, methods such as principal component analysis (PCA), linear discriminant analysis (LDA), etc. can be used for feature dimensionality reduction.

[0074] Feature fusion: It is the process of combining multiple feature sets or data sets into a new data set to improve the accuracy and performance of the model.

[0075] Among them, in step S103, that is, performing multimodal information fusion processing on the extracted feature information to obtain fused feature information, including the following step S401:

[0076] S401. Input the image feature information, speech feature information, and preprocessed location information into the multimodal information fusion processing module, and obtain the fused feature information F according to the following formula f :

[0077] F f = Ψ(p pre , F i , F s )

[0078] where Ψ is the multimodal information fusion processing module.

[0079] In this embodiment, the multimodal information fusion processing module adopts the MMT model.

[0080] For step S401, by fusing the image feature information, voice feature information, and preprocessed location information, the problem of difficult effective processing of multi-modal information in complex travel scenarios is solved. Specifically, the multi-modal information fusion processing module can comprehensively process information from different modalities to generate more representative fusion feature information. This fusion processing not only improves the accuracy of information processing but also enhances the system's adaptability in complex environments.

[0081] To achieve effective fusion of multi-modal information, the MMT model is adopted. The MMT model can fully consider the correlation and context connection between modalities when processing different modal information, thereby generating more reliable fusion feature information. Specifically, the MMT model ensures that each modal information can play its maximum role in the fusion process through multi-level information interaction and feature extraction.

[0082] Thus, through the application of the multi-modal information fusion processing module in this embodiment, the performance of the safe travel navigation system is significantly improved. Compared with the single-modal information processing method, the multi-modal information fusion processing method in this embodiment can more comprehensively analyze and understand the user's travel environment, thereby providing more accurate safety risk assessment and real-time warning information to further ensure the user's travel safety.

[0083] S104. Perform output processing on the fusion feature information to obtain the type of safety risk, risk assessment score, user's current location, and estimated route time consumption.

[0084] In this embodiment, output processing is performed on the fusion feature information to generate key information such as the type of safety risk, risk assessment score, user's current location, and estimated route time consumption. The type of safety risk may include falls, getting lost, traffic violations, etc. The risk assessment score is used to quantify the current safety risk level. The user's current location is determined through location data and provided to the user or guardian. The estimated route time consumption is estimated based on location data and possible traffic conditions to help the user plan the itinerary.

[0085] As Figure 5 shown, it is a schematic diagram of the framework structure of a multi-modal large model based on few-shot learning, including image data, voice data, and location data, as well as an image feature extraction module, a voice feature extraction module, a multi-modal information fusion processing module, and an output module. The image data, voice data, and location data are respectively input into the corresponding feature extraction modules, and then the feature information is input into the multi-modal information fusion processing module. Then, it is input into the output module to output the type of safety risk, risk assessment score, user's current location, and estimated route time consumption.

[0086] S105. Provide real-time warning information according to the security risk type and risk assessment score, and provide a safe route according to the user's current location and the estimated route time consumption.

[0087] In this embodiment, the fused feature information is output and processed to obtain the security risk type, risk assessment score, user's current location, and the estimated route time consumption. These information can be used to provide real-time warnings and safe routes. For example, based on the security risk type and risk assessment score, the virtual object generation module is called to generate the corresponding three-dimensional risk prompt model and path navigation mark. By using the AR device to parse these models and marks, a three-dimensional digital model of the user's location scene can be constructed, and the virtual content projection position can be calibrated. Then, through the real-time rendering engine, the light and shadow processing and anti-aliasing optimization of the coordinate matching image are performed, and the rendered image is output to the display screen of the AR device to provide real-time warning information.

[0088] Further, in a specific embodiment, image data, voice data, and location data are obtained through the AR device, and the collected image data, voice data, and location data are sent to the server for processing through a wireless network (such as a 5G network, Wi-Fi, etc.). The server identifies and analyzes the received image data to detect potential safety hazards such as obstacles, pedestrians, and vehicles. The voice data is processed such as noise reduction and recognition to extract the user's voice commands or key information. The location data is parsed, combined with map information and historical data, to provide the optimal navigation route for the user. The processed data (including warning information and navigation route) is sent back to the AR device through the wireless network. The AR device generates three-dimensional warning information and navigation route in real time according to the received data, and displays them to the user through the display screen. According to the results of image recognition and analysis, voice recognition and processing, and location data analysis, the system can generate real-time warning information, such as "There is construction ahead, please detour", "Pay attention to pedestrians crossing the road", etc. Finally, combined with the user's current location, destination, and real-time traffic conditions, the system plans an optimal navigation route for the user and presents it in three-dimensional form on the AR display screen.

[0089] It should be noted that other electronic devices can also be used to provide safe navigation routes and warning information for users. For example, in order to facilitate the user's travel, an electronic cane can be used. The AR device can be a head-mounted AR glasses or a handheld AR device, etc.

[0090] Among them, as Figure 6 shown, in step S105, that is, providing real-time warning information according to the security risk type and risk assessment score, and providing a safe route according to the user's current location and the estimated route time consumption, includes the following steps S501 to S506:

[0091] S501. Call the virtual object generation module respectively based on the security risk type and the risk assessment score to generate corresponding three-dimensional risk prompt models and path navigation markers;

[0092] S502. Use the AR device to parse the three-dimensional risk prompt models and path navigation markers to obtain three-dimensional content parameters;

[0093] S503. Based on the three-dimensional content parameters, use the SLAM algorithm to construct a three-dimensional digital model of the user's current scene and calibrate the virtual content projection position;

[0094] S504. Match the coordinates of the virtual content projection position with the key areas in the user's actual environment to obtain a coordinate-matched image;

[0095] S505. Call the real-time rendering engine to perform lighting processing and anti-aliasing optimization on the coordinate-matched image, and output the rendered coordinate-matched image to the display screen of the AR device to provide real-time warning information;

[0096] S506. Provide a safe route based on the real-time warning information, using the user's current location and the estimated route time.

[0097] For steps S501 - S506, based on the identified security risk types (such as fire, slip, object fall, etc.), determine which type of risk prompt needs to be displayed. The risk assessment score is used to determine the urgency and importance of the risk prompt, which may affect the appearance of the three-dimensional model (such as color, size) or the detail level of the prompt information. According to the security risk type and the risk assessment score, this module generates corresponding three-dimensional risk prompt models and path navigation markers. These models may be three-dimensional graphics or animations for intuitively displaying risk and navigation information.

[0098] Furthermore, use augmented reality devices (such as AR glasses, tablets, etc.) to parse and process these three-dimensional models. The three-dimensional content parameter parsing process extracts parameters such as the geometric shape, color, texture, and position of the three-dimensional model. This information will be used for subsequent three-dimensional scene construction. Then, the Simultaneous Localization and Mapping (SLAM) algorithm is used to construct a three-dimensional digital model of the user's current scene in real time. The SLAM algorithm perceives the environment through sensor data (such as cameras, lidar, etc.) and constructs a map. In the constructed three-dimensional model, determine the projection positions of the virtual risk prompts and navigation markers to ensure that they can accurately appear in the correct positions in the user's field of view.

[0099] Furthermore, identify key areas in the user's actual environment (such as exits, safety passages, dangerous areas, etc.). Match the projection positions of the virtual content with these key areas to generate a coordinate-matched image that combines virtual and real elements. Use a real-time rendering engine to perform lighting processing and anti-aliasing optimization on the coordinate-matched image to enhance the visual effect and user experience. Output the rendered image to the display screen of the AR device, enabling the user to see the real environment integrated with virtual risk warnings and navigation markers in real time. Based on the real-time warning information on the display screen, the system can identify the user's current position and the estimated route time. According to this information and the preset safety guidelines, the system provides the user with the safest route to avoid potential safety risks.

[0100] It can be seen that in the above solution, for complex pension entities such as the pension business, by obtaining and preprocessing multi-modal information, feature extraction, multi-modal information fusion, and output processing, and finally obtaining steps such as safe routes and safety warning information, the problem that an artificial intelligence model with few learning times cannot effectively process multi-modal data and cope with complex real-world environments is solved. At the same time, it can analyze the user's travel environment and behavior more comprehensively, and can also capture the details and potential safety risks during the user's travel, improve the accuracy of safety warnings, and further improve the accuracy of safety detection, ensuring the optimal travel path and safety warning for users in a complex environment.

[0101] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0102] The embodiment of the present invention also provides a safe travel navigation device based on multi-modal information fusion. The safe travel navigation device based on multi-modal information fusion corresponds one-to-one with the safe travel navigation method based on multi-modal information fusion in the above embodiment. As Figure 7 shown, the safe travel navigation device based on multi-modal information fusion includes an acquisition unit 601, a feature extraction unit 602, a fusion unit 603, an output unit 604, and a safe route providing unit 605. The detailed description of each functional unit is as follows:

[0103] The acquisition unit 601 is used to acquire multi-modal information and preprocess the multi-modal information, where the multi-modal information includes image data, voice data, and position data;

[0104] The feature extraction unit 602 is used to extract features from the preprocessed multi-modal information to obtain feature information corresponding to each modality;

[0105] The fusion unit 603 is configured to perform multimodal information fusion processing on the extracted feature information to obtain fused feature information;

[0106] The output unit 604 is configured to perform output processing on the fused feature information to obtain the type of security risk, the risk assessment score, the current location of the user, and the estimated route time;

[0107] The safe route providing unit 605 is configured to provide real-time warning information according to the type of security risk and the risk assessment score, and provide a safe route according to the current location of the user and the estimated route time.

[0108] The present invention provides a safe travel navigation device based on multimodal information fusion. By adopting the multimodal information fusion technology, the device combines different types of data including images, voices, location sounds, etc., and can more accurately extract complex feature representations. It can not only more comprehensively analyze the travel environment and behavior of users, but also capture the details and potential security risks during the travel process of users, improving the accuracy of security warnings. In addition, the method based on the multimodal large model can provide strong mathematical deduction ability, further improving the accuracy of security detection and ensuring the provision of the optimal travel path and security warning for users in a complex environment.

[0109] For the specific limitations of the safe travel navigation device based on multimodal information fusion, reference can be made to the limitations of the safe travel navigation method based on multimodal information fusion in the above text, which will not be elaborated here. Each module in the above-mentioned safe travel navigation device based on multimodal information fusion can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in the processor of the computer device in hardware form or be independent of it, or be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0110] In one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 8 shown. The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage media. The network interface of the computer device is used to communicate with external clients through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a safe travel navigation method based on multimodal information fusion.

[0111] In one embodiment, a computer device is provided. The computer device can be a client, and its internal structure diagram can be as shown in Figure 9 . The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the client side of a secure travel navigation method based on multi-modal information fusion.

[0112] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the following steps are realized:

[0113] Obtain multi-modal information and preprocess the multi-modal information, where the multi-modal information includes image data, voice data, and location data;

[0114] Extract features from the preprocessed multi-modal information to obtain feature information corresponding to each modality;

[0115] Perform multi-modal information fusion processing on the extracted feature information to obtain fused feature information;

[0116] Perform output processing on the fused feature information to obtain a security risk type, a risk assessment score, the user's current location, and the estimated route time;

[0117] Provide real-time warning information according to the security risk type and the risk assessment score, and provide a safe route according to the user's current location and the estimated route time.

[0118] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, the following steps are realized:

[0119] Obtain multi-modal information and preprocess the multi-modal information, where the multi-modal information includes image data, voice data, and location data;

[0120] Extract features from the preprocessed multi-modal information to obtain feature information corresponding to each modality;

[0121] Perform multi-modal information fusion processing on the extracted feature information to obtain fused feature information;

[0122] Perform output processing on the fusion feature information to obtain the security risk type, risk assessment score, user's current location, and estimated route time consumption;

[0123] Provide real-time warning information according to the security risk type and risk assessment score, and provide a safe route according to the user's current location and estimated route time consumption.

[0124] It should be noted that for the functions or steps that can be achieved by the above computer-readable storage medium or computer device, reference can be made to the relevant descriptions on the server side and the client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0125] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above method embodiments. Among them, any reference to memory, storage, database, or other media used in the various embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0126] Those skilled in the art can clearly understand that for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0127] The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A safe travel navigation method based on multimodal information fusion, characterized in that: include: Acquiring multimodal information and preprocessing the multimodal information, wherein the multimodal information includes image data, voice data, and position data; Performing feature extraction on the preprocessed multimodal information to obtain feature information of corresponding modalities respectively; The extracted feature information is subjected to multi-modal information fusion processing to obtain fused feature information; The fused feature information is output and processed to obtain the security risk type, risk assessment score, user's current location and estimated route time; According to the security risk type and risk assessment score, real-time warning information is provided, and a safe route is provided according to the user's current location and estimated route time.

2. The safe travel navigation method based on multimodal information fusion according to claim 1 is characterized in that: The preprocessing of the multimodal information includes: The image data is scaled and its contrast is adjusted to obtain preprocessed image data i pre ; The speech data is subjected to noise reduction processing to obtain pre-processed speech data s pre ; The position data is subjected to noise filtering and the coordinates in the position data are converted into local coordinates to obtain the pre-processed position information p pre .

3. The safe travel navigation method based on multimodal information fusion according to claim 2 is characterized in that: The performing feature extraction on the preprocessed multimodal information to obtain feature information of corresponding modalities respectively includes: The preprocessed image data is processed by the image feature extraction module, and the image feature information F is obtained according to the following formula i : F i =Φ i (i pre ) Among them, Φ i is the image feature extraction module, i pre is the preprocessed image data; The pre-processed speech data is processed by the speech feature extraction module, and the speech feature information F is obtained according to the following formula s : F s =Φ s (s pre ) Among them, Φ s is the speech feature extraction module, s pre is the preprocessed speech data.

4. The safe travel navigation method based on multimodal information fusion according to claim 3 is characterized in that: The step of performing multi-modal information fusion processing on the extracted feature information to obtain fused feature information includes: The image feature information, speech feature information and pre-processed position information are input into the multimodal information fusion processing module, and the fusion feature information F is obtained according to the following formula: f : F f =Ψ(p pre ,F i ,F s ) Among them, Ψ is the multimodal information fusion processing module.

5. The safe travel navigation method based on multimodal information fusion according to claim 3 is characterized in that: The image feature extraction module adopts a visual Transformer model, and the speech feature extraction module adopts a Whisper model.

6. The safe travel navigation method based on multimodal information fusion according to claim 4 is characterized in that: The multimodal information fusion processing module adopts the MMT model.

7. The safe travel navigation method based on multimodal information fusion according to claim 1 is characterized in that: The providing of real-time warning information according to the security risk type and risk assessment score, and providing a safe route according to the user's current location and estimated route time, includes: Calling a virtual object generation module based on the security risk type and the risk assessment score to generate a corresponding three-dimensional risk warning model and a path navigation mark; Analyze the three-dimensional risk warning model and the path navigation mark using an AR device to obtain three-dimensional content parameters; Based on the three-dimensional content parameters, a SLAM algorithm is used to construct a three-dimensional digital model of the scene where the user is located, and a virtual content projection position is calibrated; Coordinate matching of the virtual content projection position with the key area in the user's actual environment to obtain a coordinate matching image; Calling a real-time rendering engine to perform light and shadow processing and anti-aliasing optimization on the coordinate matching image, and outputting the rendered coordinate matching image to a display screen of the AR device to provide real-time warning information; A safe route is provided based on the real-time warning information using the user's current location and the estimated route duration.

8. A safe travel navigation device based on multimodal information fusion, characterized in that: include: an acquisition unit, configured to acquire multimodal information and preprocess the multimodal information, wherein the multimodal information includes image data, voice data, and position data; A feature extraction unit, used to extract features from the preprocessed multimodal information to obtain feature information of corresponding modalities; A fusion unit, used for performing multi-modal information fusion processing on the extracted feature information to obtain fused feature information; An output unit, used to output the fused feature information to obtain the security risk type, risk assessment score, user's current location and estimated route time; A safe route unit is provided, which is used to provide real-time warning information according to the security risk type and risk assessment score, and to provide a safe route according to the user's current location and the estimated route time.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the safe travel navigation method based on multimodal information fusion according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to execute the safe travel navigation method based on multimodal information fusion as described in any one of claims 1 to 7.

Images