Navigation fault detection method and electronic device
Patent Information
- Application Number
- PCT/CN2024/091312
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-08
- Filing Date
- 2024-05-07
- Publication Date
- 2024-11-14
Smart Images

Figure CN2024091312_14112024_PF_FP_ABST
Abstract
Description
Navigation fault detection method and electronic equipment
[0001] This application claims priority to the Chinese patent application with application number 202310515770.9 filed with the State Intellectual Property Office of China on May 8, 2023, and priority to the Chinese patent application with the invention name “A Navigation Fault Detection Method and Electronic Device”, all contents of which are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of computer technology, and in particular to a navigation fault detection method and electronic equipment. Background Art
[0003] Global Navigation Satellite System (GNSS) jamming occurs when a jammer simulates and transmits false GNSS jamming signals through platforms such as software-defined radio (SDR). This jammer intrudes upon and takes over the target receiver, causing it to output erroneous positioning results. These erroneous positioning results can have adverse consequences. For example, in autonomous driving scenarios, a jammer can design sophisticated GNSS jamming algorithms to target the target vehicle's multi-sensor fusion framework (which outputs GNSS position information used for navigation and autonomous driving). This can cause the framework to take over, output erroneous or false GNSS position information, and cause the vehicle to deviate from its planned route, potentially leading to an accident. Currently, the industry has not developed effective measures to accurately detect navigation failures caused by GNSS jamming signals, resulting in a high rate of missed detection.
[0004] Summary of the Invention
[0005] This application discloses a navigation fault detection method and electronic equipment. Based on the different characteristics of GNSS interference changes in different time periods, different methods are used to detect navigation faults caused by GNSS interference signals according to different changes, thereby improving the accuracy of fault detection, reducing the missed detection rate of faults, and effectively improving the positioning accuracy and safety of electronic equipment during autonomous driving.
[0006] In a first aspect, an embodiment of the present application provides a navigation fault detection method, which is applied to a first device, the method comprising: obtaining first data, the first data comprising multiple observation values at a first moment respectively measured by multiple sensors in the first device; obtaining first status information of the first device at the first moment based on the first data; detecting whether the first device is interfered with by the global navigation satellite system GNSS based on the first data and the first status information; when it is detected that the first device is interfered with by the GNSS, displaying a first interface, the first interface comprising first prompt information, the first prompt information indicating that the first device is interfered with by the GNSS.
[0007] In the above method, the electronic device can obtain the first status information of the first device at the first moment based on multiple observation values, and detect whether the electronic device is subject to GNSS interference based on the multiple observation values and the first status information to monitor the starting moment of the GNSS interference. When interference is detected, the electronic device can promptly alert the user, such as displaying a first prompt message to remind the user that it is currently subject to GNSS interference, to avoid the misplaced positioning information output due to GNSS interference affecting the user's use and causing accidents, thereby improving the safety of electronic devices when driving, improving the accuracy of fault detection, and reducing the missed detection rate of faults.
[0008] In one possible implementation, detecting whether the first device is subject to GNSS interference based on the first data and the first status information includes: detecting whether the first device is subject to GNSS interference based on the first data, historical data corresponding to the first data, the first status information, and historical status information corresponding to the first status information.
[0009] In the above method, the electronic device can detect whether the first device is subject to GNSS interference based on the first data and first status information at the first moment, as well as the historical data before the first moment (i.e., the historical data corresponding to the first data and the historical status information corresponding to the first status information), thereby avoiding the Kalman filter only considering the observation value at the current moment (i.e., the first moment mentioned above) and ignoring the constraints of the data at historical moments on the data at the current moment, thereby further improving the accuracy of fault detection.
[0010] In one possible implementation, detecting whether the first device is subject to GNSS interference based on the first data, the historical data corresponding to the first data, the first status information, and the historical status information corresponding to the first status information includes: obtaining second data based on the first data, the historical data corresponding to the first data, the second status information, and the historical status information corresponding to the second status information; obtaining a first statistic of the first device at the first moment based on the second data, and determining that the first device is subject to GNSS interference when the first statistic is greater than or equal to a first threshold.
[0011] In the above method, the electronic device can timely detect slowly varying soft faults in GNSS interference through the first statistic to monitor the start time of GNSS interference, improve the accuracy of fault detection, and reduce the missed detection rate of faults.
[0012] In one possible implementation, the method further includes: when it is detected that the first device is not subject to GNSS interference, displaying a second interface, the second interface including a first position and a first route, the first position being the position of the first device at the first moment estimated based on the second state information, and the first route being the navigation route of the first device at the first moment estimated based on the second state information.
[0013] In the above method, when it is detected that the electronic device is not subject to GNSS interference, the electronic device can display the position and navigation route at the first moment obtained based on the first data, and automatically drive according to the navigation route, so that the user can view the real-time position and navigation route of the electronic device during the current trip, and the user can also monitor the road conditions and improve driving safety.
[0014] In one possible implementation, the method further includes: when it is detected that the first device is subject to GNSS interference, displaying a third interface, the third interface including a second position and a second route, the second position being the position of the first device at the first moment estimated based on the first status information, and the second route being the navigation route of the first device at the first moment estimated based on the first status information.
[0015] In the above method, when it is detected that the electronic device is subject to GNSS interference, the electronic device can display the second position and the second route at the first moment obtained based on the first status information. The second position is closer to the actual position of the electronic device when it is subject to GNSS interference, thereby avoiding the electronic device from deviating from the normally planned route due to the erroneous positioning information output by the GNSS interference, causing an accident, and improving the positioning accuracy and safety of the electronic device during automatic driving.
[0016] In a possible implementation, the first state information is obtained according to a first algorithm, and the second state information is obtained according to a second algorithm. The first algorithm is a recognition suppression algorithm, and the second algorithm is a multi-sensor fusion MSF positioning algorithm.
[0017] In the above method, the position obtained by the factor graph-based identification and suppression algorithm is closer to the actual position of the electronic device when experiencing GNSS interference, allowing the electronic device to still output correct position information even when experiencing GNSS interference. The MSF positioning algorithm only considers the observation value at the current moment, ignoring the constraints imposed by historical data on the current moment. It also has high requirements for system linearization and noise distribution. When the measurement model is nonlinear and / or the noise does not conform to a Gaussian distribution, the fusion accuracy is low and the estimated state information is inaccurate.
[0018] In a possible implementation, the first state information includes at least one of the position, posture, speed, and time of the first device.
[0019] In one possible implementation, the method further includes: receiving a first operation when displaying the first interface; and displaying a third interface in response to the first operation, the third interface including a second position and a second route, the second position being the position of the first device at the first moment estimated based on the first state information, and the second route being the navigation route of the first device at the first moment estimated based on the first state information.
[0020] In the above method, the electronic device can detect signal anomalies before a dangerous collision occurs and prompt the user that the current driving safety is about to be threatened. After receiving the user operation, the second position and second route of the first moment obtained according to the first state information can be displayed. The second position is closer to the actual position of the electronic device when it is subject to GNSS interference, thereby avoiding the electronic device from deviating from the normally planned route due to the erroneous positioning information output by the GNSS interference, causing an accident, and improving the positioning accuracy and safety of the electronic device during automatic driving.
[0021] In one possible implementation, when it is detected that the first device is subject to GNSS interference, a first interface is displayed, including: when it is detected that the first device is subject to GNSS interference, if the defense function is not turned on, the first interface is displayed; the method also includes: when it is detected that the first device is subject to GNSS interference, if the defense function is turned on, a third interface is displayed, the third interface including a second position and a second route, the second position is the position of the first device at the first moment estimated based on the first status information, and the second route is the navigation route of the first device at the first moment estimated based on the first status information.
[0022] In the above method, if the defense function is not turned on, the electronic device can display the first interface when it encounters GNSS interference to prompt the user to turn on the defense function, thereby improving the safety of driving the electronic device. If the defense function is turned on, the electronic device can output correct location information when it encounters GNSS interference, thereby improving the positioning accuracy of the electronic device when driving.
[0023] In one possible implementation, the method further includes: when the first interface is displayed, the driving mode of the first device is an automatic driving mode; after the first interface is displayed, the method further includes: receiving a second operation, and the second operation is used to switch the automatic driving mode to a manual driving mode.
[0024] In the above method, when the electronic device is subject to GNSS interference, it can switch the automatic driving mode to the manual driving mode according to the user's operation, so as to prevent the electronic device from deviating from the normal planned route due to the erroneous positioning information output by the GNSS interference and affecting the user's use, thereby further improving the safety of the electronic device during driving and ensuring the safety of the user.
[0025] In one possible implementation, the detecting that the first device is subject to GNSS interference includes: when it is detected that the offset distance is greater than or equal to an offset threshold, determining that the first device is subject to GNSS interference, the offset distance is the difference between a first position and a second position, the first position is a position estimated according to a first algorithm, the second position is a position estimated according to a second algorithm, the first algorithm is an identification suppression algorithm, and the second algorithm is an MSF positioning algorithm; when it is detected that the offset distance is less than the offset threshold, determining that the GNSS interference to the first device has ended.
[0026] In the above method, when the offset distance of the electronic device is greater than or equal to the offset threshold, the electronic device may deviate from the normally planned route, which may easily cause an accident. Therefore, when the offset distance is greater than or equal to the offset threshold, the electronic device determines that it is subject to GNSS interference to ensure safety during driving.
[0027] In one possible implementation, the method further includes: when it is detected that the first device is not subject to GNSS interference, the first device is located in a first lane; the offset distance is the distance between the first position estimated by the first algorithm and a lane dividing line, and the lane dividing line is the dividing line between the first lane and an adjacent lane.
[0028] In the above method, when the electronic device switches from the first lane in which it is traveling to an adjacent lane, the electronic device deviates from the normally planned route, and therefore, the electronic device determines that it is subject to GNSS interference.
[0029] In one possible implementation, the method further includes: after detecting that the first device is subject to GNSS interference, obtaining second data, the second data including multiple observation values at a second moment measured by the multiple sensors respectively, the second moment being later than the first moment; detecting whether the GNSS interference to the first device has ended based on the second data; and when it is detected that the GNSS interference to the first device has ended, displaying a second prompt message, the second prompt message indicating that the GNSS interference to the first device has ended.
[0030] In the above method, after the electronic device is subjected to GNSS interference, the electronic device can detect whether the GNSS interference suffered by the electronic device has ended based on multiple observation values at the second moment, so as to monitor the end time of the GNSS interference, avoid false alarms caused by untimely detection, so that normal GNSS signals cannot be used for navigation and positioning. When the end of the GNSS interference is detected, the alarm end information is displayed to prompt the user that the current driving safety threat is lifted, thereby improving the safety of the electronic device when driving, improving the accuracy of fault detection, and reducing the missed detection rate of faults.
[0031] In one possible implementation, the method further includes: after detecting that the GNSS interference to the first device has ended, displaying a fourth interface, the fourth interface including a third position and a third route, the third position being the position of the first device at the second moment estimated based on the third state information, the third route being the navigation route of the first device at the second moment estimated based on the third state information, the third state information being obtained based on a second algorithm, and the second algorithm being an MSF positioning algorithm.
[0032] In the above method, when the electronic device detects that the GNSS interference has ended, the electronic device can restore to the third position and third route estimated by the MSF positioning algorithm, avoiding the identification suppression algorithm from occupying too much running memory of the electronic device and saving memory resources.
[0033] In one possible implementation, after displaying the second interface, the method further includes: when it is detected that the first device is subject to GNSS interference, if the defense function is not turned on, displaying a fifth interface, the fifth interface including a fourth position and a fourth route, the fourth position and the fourth route being obtained according to a second algorithm, the time corresponding to the fourth position being later than the first time, the fourth position being different from the first position, the fourth route being different from the first route, and the second algorithm being an MSF positioning algorithm.
[0034] In the above method, if the electronic device continues to be subject to GNSS interference and the defense function is not enabled, the electronic device will display the fourth position and fourth route obtained by the MSF positioning algorithm. At this time, the fourth position and fourth route are inaccurate positioning information, which will cause the electronic device to deviate from the normally planned route and cause an accident.
[0035] In one possible implementation, detecting whether the GNSS interference suffered by the first device has ended based on the second data includes: obtaining third data based on the second data and a first estimated value, the first estimated value being obtained based on part of the observation values in the second data, the first estimated value being obtained based on a first algorithm, the first algorithm being an identification and suppression algorithm, the first estimated value corresponding to the second moment, and the third data corresponding to the second moment; when it is detected at a third moment that the first device is suffering from GNSS interference, obtaining a second statistic based on the second data, the third moment being a moment before the second moment, and the second statistic corresponding to the second moment; when the second statistic is greater than or equal to a first threshold, obtaining a third statistic based on the third data, the third statistic corresponding to the second moment; when the third statistic is greater than or equal to the second threshold, determining that the GNSS interference suffered by the first device has ended.
[0036] In the above method, the electronic device can use the second statistic and the third statistic to timely detect sudden hard faults in GNSS interference to monitor the end time of GNSS interference, improve the accuracy of fault detection, and reduce the missed detection rate of faults.
[0037] In a second aspect, the present application provides an electronic device comprising a transceiver, a processor and a memory, wherein the memory is used to store a computer program, and the processor calls the computer program to execute the navigation fault detection method in any possible implementation of the first aspect.
[0038] In a third aspect, the present application provides an electronic device comprising one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are configured to store computer program code, the computer program code comprising computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the navigation fault detection method of any possible implementation of the first aspect.
[0039] In a fourth aspect, the present application provides a computer storage medium storing a computer program. When the computer program is executed by a processor, the navigation fault detection method in any possible implementation of any of the above aspects is implemented.
[0040] In a fifth aspect, the present application provides a computer program product, which, when running on an electronic device, enables the electronic device to execute the navigation fault detection method in any possible implementation of the first aspect above.
[0041] In a sixth aspect, the present application provides an electronic device, the electronic device including a method or apparatus for executing any one of the implementations of the first aspect of the present application. The electronic device is, for example, a chip. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] The following is an introduction to the drawings used in this application.
[0043] FIG1 is a schematic diagram of the hardware structure of an electronic device 100 provided in this application;
[0044] FIG2 is a schematic diagram of a software architecture of an electronic device 100 provided in this application;
[0045] FIG3 is a schematic structural diagram of an electronic device 100 provided in this application;
[0046] Figures 4 to 6 are schematic diagrams of some user interfaces provided by this application;
[0047] FIG7 is a flow chart of a navigation fault detection method provided by the present application;
[0048] FIG8 is a schematic diagram of the structure of a factor graph provided by the present application;
[0049] FIG9 is a schematic diagram of the structure of a factor graph corresponding to a GNSS residual sequence provided by the present application;
[0050] FIG10 is a flow chart of another navigation fault detection method provided by the present application;
[0051] FIG11 is a schematic diagram of the structure of a factor graph corresponding to residual information provided by the present application;
[0052] FIG12 is a logic diagram of a combined first method and second method for detecting the end of GNSS interference provided by the present application;
[0053] FIG13 is a schematic diagram of a GNSS interference detection process provided by the present application. DETAILED DESCRIPTION
[0054] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " represents or, for example, A / B can represent A or B; "and / or" in the text is merely a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" refers to two or more than two.
[0055] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.
[0056] The location information provided by the Global Navigation Satellite System (GNSS) can be applied to scenarios such as autonomous driving and terminal location-based services (such as navigation services). Since the GNSS signals emitted by satellites are not strong, when the GNSS signals are weak and / or modulated, the GNSS signals are susceptible to radio frequency interference, resulting in inaccurate / degraded positioning accuracy and even interruption of positioning services. GNSS interference is a type of radio frequency interference with strong intentions. Interferers can simulate and transmit false GNSS interference signals (such as forged signals with the same parameters as GNSS satellite signals) through platforms such as software radio, causing the target receiver to be invaded and taken over by the interference signal, thereby outputting erroneous positioning results. Incorrect positioning results can lead to adverse consequences. For example, in autonomous driving and navigation scenarios, vehicles deviate from their normal planned routes, causing accidents.
[0057] For the convenience of explanation, the following embodiments are described using an autonomous driving scenario as an example.
[0058] In autonomous driving scenarios, vehicles can navigate and autonomously operate based on GNSS position information output by a multi-sensor fusion (MSF) framework. This framework outputs GNSS position information based on detection signals from multiple sensors, such as, but not limited to, GNSS receivers, inertial measurement units (IMUs), lidars (light detection and ranging), and cameras. Jammers can design sophisticated GNSS jamming algorithms targeting the target vehicle's MSF framework, causing it to take over and output erroneous or false GNSS position information, causing the vehicle to deviate from its planned route and experience navigation failures, potentially leading to accidents.
[0059] At present, the multi-sensor fusion framework is based on the Kalman filter to estimate the state information of the vehicle, such as but not limited to position, attitude, speed, time and other information. In one embodiment, the Kalman filter can include two stages: prediction and update. In the prediction stage, the Kalman filter can obtain the estimated result of the current state (hereinafter referred to as the current state) based on the estimated result of the state at the previous moment (hereinafter referred to as the previous state). In the update stage, the estimated result obtained in the prediction stage is optimized using the observed value of the current state to obtain the optimal estimated result of the current state. For example, the estimated value at time k is obtained based on the estimated value at time k-1, and the estimated value at time k is optimized using the observed value at time k to obtain the optimal estimated value at time k. The estimated value can be estimated by a measurement model, and the observed value can be measured by multiple sensors. For example, the acceleration and angular velocity of the vehicle are measured by an IMU. The measurement model can indicate the functional relationship between the observed value and the state to be estimated. Integrated navigation systems based on Kalman filters suffer from inaccurate state information estimation. For example, Kalman filters only consider the estimated value of the previous state and the observed value of the current state, ignoring the constraints imposed by state information at previous moments (referred to as historical information) on the current state information. Another example is that Kalman filters are implemented based on Gaussian noise and linear systems, placing high demands on the system's linearization and noise distribution. When the measurement model is nonlinear and / or the noise does not conform to a Gaussian distribution, the Kalman filter's fusion accuracy is low, resulting in inaccurate estimated state information.
[0060] Furthermore, GNSS interference initially varies gradually, potentially causing small, slowly varying soft faults. Near the end of the interference, it varies suddenly, potentially causing sudden hard faults. Currently, the residual chi-square method can be used to detect GNSS interference signals and faults. While this method has high sensitivity for sudden hard faults, it has low sensitivity for small, slowly varying soft faults. Therefore, the residual chi-square method cannot promptly detect slowly varying soft faults caused by GNSS interference, resulting in a high rate of missed faults and a high risk of accidents.
[0061] The present application provides a navigation fault detection method, which is applied to electronic devices. The electronic devices can use a factor graph-based identification and suppression algorithm to estimate state information to obtain the real-time position / positioning result of the electronic devices, solving the problem of inaccurate state information estimated by the multi-sensor fusion (MSF) positioning algorithm. The electronic devices can use a detection method based on exponentially decreasing sequential probability ratio to detect slowly changing soft faults in GNSS interference to monitor the start time of GNSS interference. In addition, the electronic devices can use a method combining exponentially decreasing sequential probability ratio and chi-square test to detect sudden hard faults in GNSS interference to monitor the end time of GNSS interference. The present application comprehensively considers the slowly changing and sudden cases in GNSS interference, adopts different methods to detect faults for different cases, improves the accuracy of fault detection, and reduces the missed detection rate of faults. In addition, the real-time position / positioning result obtained by the factor graph-based identification and suppression algorithm is closer to the actual position of the electronic device when it is subjected to GNSS interference, so that the electronic device can still output correct position information when it is subjected to GNSS interference, avoiding the electronic device from deviating from the normal planned route due to erroneous positioning information output by GNSS interference, causing accidents, and improving the positioning accuracy and safety of the electronic device during autonomous driving.
[0062] In addition, the electronic device can display the detection process of the GNSS interference signal to the user in a visual form. For example, after the GNSS interference begins, the electronic device can display an alarm message to remind the user that it is currently suffering from GNSS interference. When the GNSS interference ends, the electronic device can display an alarm end message to remind the user that the current GNSS interference has ended. During the GNSS interference process, the electronic device can display the real-time location / positioning results obtained by the identification and suppression algorithm to avoid the user's use being affected by the erroneous positioning information output by the GNSS interference, further improving the safety of the electronic device during automatic driving, thereby ensuring user safety.
[0063] In this application, the electronic device may be a mobile phone, a tablet computer, a handheld computer, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), smart home devices such as smart TVs, wearable devices such as smart bracelets, smart watches, and smart glasses, extended reality (XR) devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), in-vehicle devices or smart city devices, etc. This application does not impose any special restrictions on the specific types of electronic devices.
[0064] Next, an exemplary electronic device 100 according to an embodiment of the present application is introduced.
[0065] FIG1 exemplarily shows a schematic diagram of the hardware structure of an electronic device 100 .
[0066] As shown in Figure 1, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
[0067] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0068] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.
[0069] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.
[0070] Processor 110 may also include a memory for storing instructions and data. In one embodiment, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.
[0071] In one embodiment, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.
[0072] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In one embodiment, the processor 110 may include multiple I2C bus lines. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. via different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K via the I2C interface, enabling communication between the processor 110 and the touch sensor 180K via the I2C bus interface, thereby implementing the touch function of the electronic device 100.
[0073] The I2S interface can be used for audio communication. In one embodiment, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In one embodiment, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface, enabling the function of answering calls via a Bluetooth headset.
[0074] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In one embodiment, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In another embodiment, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.
[0075] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In one embodiment, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In one embodiment, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface, enabling the playback of music via Bluetooth headphones.
[0076] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In one embodiment, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the electronic device 100. The processor 110 and the display 194 communicate via the DSI interface to implement the display function of the electronic device 100.
[0077] The GPIO interface can be configured via software. It can be configured as either a control signal or a data signal. In one embodiment, the GPIO interface can be used to connect the processor 110 to the camera 193, display 194, wireless communication module 160, audio module 170, sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.
[0078] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, or to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as augmented reality devices.
[0079] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.
[0080] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.
[0081] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.
[0082] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.
[0083] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.
[0084] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In one embodiment, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In one embodiment, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.
[0085] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In one embodiment, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.
[0086] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.
[0087] In one embodiment, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with a network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).
[0088] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.
[0089] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In one embodiment, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.
[0090] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.
[0091] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and color. It can also optimize parameters such as exposure and color temperature of the captured scene. In one embodiment, the ISP can be located within camera 193.
[0092] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In one embodiment, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.
[0093] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.
[0094] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.
[0095] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.
[0096] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.
[0097] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.
[0098] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.
[0099] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In one embodiment, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.
[0100] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or listen to hands-free calls through the speaker 170A.
[0101] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the user can place the receiver 170B close to the ear to hear the voice.
[0102] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.
[0103] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.
[0104] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In one embodiment, pressure sensor 180A can be located on display screen 194. There are many types of pressure sensors 180A, such as resistive, inductive, and capacitive. A capacitive pressure sensor can include at least two parallel plates made of conductive material. When force acts on pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the intensity of the pressure based on this change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the touch intensity based on pressure sensor 180A. Electronic device 100 can also calculate the touch location based on the detection signal from pressure sensor 180A. In one embodiment, touch operations applied to the same touch location but with different touch intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, a command to view short messages is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to a short message application icon, a command to create a new short message is executed.
[0105] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In one embodiment, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.
[0106] The air pressure sensor 180C is used to measure air pressure. In one embodiment, the electronic device 100 calculates the altitude using the air pressure value measured by the air pressure sensor 180C to assist in positioning and navigation.
[0107] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip case. In one embodiment, if the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover based on the magnetic sensor 180D. Based on the detected opening and closing status of the case or flip cover, features such as automatic unlocking of the flip cover can be configured.
[0108] Accelerometer 180E can detect the magnitude of acceleration of electronic device 100 in all directions (generally three axes). It can also detect the magnitude and direction of gravity when electronic device 100 is stationary. It can also be used to identify the electronic device's posture, enabling applications such as switching between landscape and portrait modes and pedometers.
[0109] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance using infrared or laser. In one embodiment, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.
[0110] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user is holding the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.
[0111] Ambient light sensor 180L is used to sense ambient light brightness. Electronic device 100 can adaptively adjust the brightness of display screen 194 based on the perceived ambient light. Ambient light sensor 180L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 180L can also work with proximity light sensor 180G to detect whether electronic device 100 is in a pocket to prevent accidental touches.
[0112] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.
[0113] The temperature sensor 180J is used to detect temperature. In one embodiment, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to prevent the electronic device 100 from shutting down abnormally due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 boosts the output voltage of the battery 142 to prevent abnormal shutdown due to low temperature.
[0114] The touch sensor 180K is also called a "touch-sensitive device." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a location different from that of the display screen 194.
[0115] The bone conduction sensor 180M can obtain vibration signals. In one embodiment, the bone conduction sensor 180M can obtain vibration signals of the vibrating bones of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure pulse signals. In one embodiment, the bone conduction sensor 180M can also be set in headphones to form bone conduction headphones. The audio module 170 can parse out the voice signal based on the vibration signal of the vibrating bones of the vocal part obtained by the bone conduction sensor 180M to realize the voice function. The application processor can parse the heart rate information based on the blood pressure pulse signal obtained by the bone conduction sensor 180M to realize the heart rate detection function.
[0116] The buttons 190 include a power button, a volume button, and the like. The buttons 190 may be mechanical buttons or touch buttons. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100.
[0117] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.
[0118] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.
[0119] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and disconnected from the electronic device 100 by inserting it into or removing it from the SIM card interface 195. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than one. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The multiple cards can be of the same or different types. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In one embodiment, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.
[0120] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a microservice architecture, or a cloud architecture. For example, the software system with a layered architecture can be an Android system, a Harmony operating system (OS), or other software systems. The embodiment of the present application takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device 100.
[0121] FIG2 exemplarily shows a schematic diagram of a software architecture of an electronic device 100 .
[0122] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In one embodiment, the Android system is divided into four layers: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.
[0123] The application layer can include a series of application packages.
[0124] As shown in Figure 2, the application package may include applications such as camera, gallery, music, calendar, short message, call, navigation, Bluetooth, browser, and screen projection.
[0125] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.
[0126] As shown in FIG2 , the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.
[0127] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.
[0128] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.
[0129] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.
[0130] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including answering, hanging up, etc.).
[0131] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.
[0132] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.
[0133] Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for scheduling and management of the Android system.
[0134] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.
[0135] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.
[0136] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.
[0137] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.
[0138] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
[0139] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.
[0140] A 2D graphics engine is a drawing engine for 2D drawings.
[0141] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.
[0142] The following is an example of the workflow of the software and hardware of the electronic device 100, using the autonomous driving scenario of the navigation application.
[0143] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, touch operation timestamp and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is a control of the navigation application, the navigation application calls the interface of the application framework layer, and then controls the display driver by calling the kernel layer to display the main interface of the navigation application through the display screen 194.
[0144] FIG3 exemplarily shows a structural diagram of an electronic device 100 .
[0145] As shown in FIG3 , the electronic device 100 may include a physical unit 101 and a functional unit 102. The physical unit 101 may include a sensor 1011, an onboard computer 1012, and a central control display screen 1013. The functional unit 102 may include a multi-sensor fusion (MSF) positioning algorithm module 1021, an identification and suppression algorithm module 1022, and a global navigation satellite system (GNSS) interference detection module 1023. Optionally, the onboard computer 1012 may include at least one module in the functional unit 102, wherein:
[0146] The sensor 1011 may include, but is not limited to, a GNSS receiver, an IMU, an antenna, a lidar, a camera, etc. The sensor 1011 may be used to obtain multiple observation quantities / observation values of the electronic device 100, such as the position of the electronic device 100 measured by the GNSS receiver, the acceleration and angular velocity of the electronic device 100 measured by the IMU, and the distance measured by the lidar. The onboard computer 1012 may be used to receive the multiple observation quantities / observation values measured by the sensor 1011 and send the obtained multiple observation quantities / observation values to the MSF positioning algorithm module 1021, the identification and suppression algorithm module 1022, and the GNSS interference detection module 1023. The MSF positioning algorithm module 1021 can obtain first state information of the electronic device 100 based on multiple observations / observation values sent by the onboard computer 1012. The first state information includes, for example, but is not limited to, position, attitude, speed, time, and other information. In some examples, the MSF positioning algorithm module 1021 can estimate the first state information of the electronic device 100 at time k based on multiple observations / observation values at time k (which can also be referred to as the first state information being an estimated value at time k). The MSF positioning algorithm module 1021 can send the first state information to the onboard computer 1012, which can estimate the first real-time position of the electronic device 100 at time k based on the first state information. Optionally, the onboard computer 1012 can match the first real-time position with a high-precision map to determine the first lane information of the electronic device 100. The onboard computer 1012 can send the first real-time position, and optionally, the first lane information of the electronic device 100, to the central control display screen 1013. The central control display screen 1013 can be used to display the first real-time position, and optionally, the first lane information, of the electronic device 100 during navigation.
[0147] The recognition and suppression algorithm module 1022 can obtain the second state information of the electronic device 100 based on the multiple observations / observation values sent by the onboard computer 1012. The second state information includes, for example, but not limited to, position, posture, speed, time, and other information. In some examples, the recognition and suppression algorithm module 1022 can estimate the second state information of the electronic device 100 at time k based on the multiple observations / observation values at time k (which can also be referred to as the second state information being an estimated value at time k). It can be understood that for multiple observations at the same time (k), the first state information and the second state information can be estimated values obtained using different algorithms. The recognition and suppression algorithm module 1022 can send the second state information to the onboard computer 1012, and the onboard computer 1012 can estimate the second real-time position of the electronic device 100 at time k based on the second state information. Optionally, the onboard computer 1012 can match the second real-time position with a high-precision map to determine the second lane information of the electronic device 100.
[0148] The identification and suppression algorithm module 1022 can send second status information to the GNSS interference detection module 1023. The GNSS interference detection module 1023 can detect whether the electronic device 100 is currently experiencing GNSS interference based on the second status information and multiple observation quantities / observation values sent by the onboard computer 1012. When GNSS interference is detected, the GNSS interference detection module 1023 can send an alarm message to the onboard computer 1012. After receiving the alarm message, the onboard computer 1012 can display the alarm message on the central control display screen 1013 to prompt the user that the GNSS interference is currently occurring. When the GNSS interference is detected to have ended, the GNSS interference detection module 1023 can send an alarm end message to the onboard computer 1012. After receiving the alarm end message, the onboard computer 1012 can display the alarm end message on the central control display screen 1013 to prompt the user that the current GNSS interference has ended. Optionally, during the GNSS interference process, the onboard computer 1012 can display the second real-time location and second lane information of the electronic device 100 during navigation on the central control display screen 1013.
[0149] It can be understood that the electronic device 100 shown in Figure 3 may include more or fewer modules. Any module in the electronic device 100 may be a hardware module or a software module. Any of the above modules may be a separate module, and at least one of the above modules may also be integrated together. For example, the MSF positioning algorithm module 1021, the identification and suppression algorithm module 1022, and the GNSS interference detection module 1023 may be integrated in the on-board computer 1012. This application does not limit the specific modules of the electronic device 100.
[0150] The following describes an application scenario involved in an embodiment of the present application and a user interface diagram for the scenario.
[0151] Please refer to FIG4 , which exemplarily shows a user interface diagram of autonomous driving navigation.
[0152] As shown in FIG4(A), the electronic device 100 may display a user interface 410 of a navigation application. The user interface 410 includes a search box 411 and a search box 412. The search box 411 may be used to enter the starting point of the navigation route, and the search box 412 may be used to enter the end point of the navigation route. The user interface 410 also includes a map 413, on which are displayed a location marker 413A, a location marker 413B, and a route 413C. The location marker 413A is used to identify the location of the starting point on the map 413, where the starting point is the location entered in the search box 411. The location marker 413B is used to identify the location of the end point on the map 413, where the end point is the location entered in the search box 412. The route 413C is used to indicate route information from the starting point to the end point. The user interface 410 also includes navigation information 414. The navigation information 414 may include multiple route information from the above-mentioned starting point to the above-mentioned end point, such as "recommended" route information and route information of other options. The "recommended" route information is displayed with the characters "32 minutes · 11 kilometers", which can represent the time and route distance required for the recommended route, wherein the "recommended" route information has been selected, corresponding to route 413C. The navigation information 414 may also include a control 414A. The control 414A is displayed with the characters "Start navigation", which can be used to trigger the navigation interface that displays the above-mentioned selected route information. In one embodiment, the electronic device 100 can respond to a user operation on the control 414A (for example, the user operation is a touch operation), automatically drive according to the selected route information, and display the user interface of the automatic driving navigation (also referred to as navigation in automatic driving mode), specifically referring to the user interface 420 shown in (B) of Figure 4.
[0153] As shown in (B) of Figure 4, the user interface 420 may include an autonomous driving navigation interface 421 and navigation information 422. The autonomous driving navigation interface 421 may display multiple lanes, for example, lane 421A, lane 421B and lane 421C. The autonomous driving navigation interface 421 may also include a location identifier 421D and a navigation route 421E. The location identifier 421D may identify the location of the electronic device 100 on the map, specifically the lane where the electronic device 100 is located (for example, the electronic device 100 is currently in lane 421B). The navigation route 421E may represent the current automatic driving route of the electronic device 100. Navigation information 422 may include driving prompt information, road information, distance and time information. The driving prompt information may indicate the driving of the electronic device 100 (for example, "Please keep straight"), the road information may indicate the road information during the driving process of the electronic device 100 (for example, "2 kilometers to enter Taizi Lake North Road"), and the distance and time information may indicate the distance between the electronic device 100 and the destination (for example, "11 kilometers"), the remaining driving time (for example, "32 minutes") and the estimated time to reach the destination (for example, "expected to arrive at 8:32").
[0154] In one embodiment, during autonomous driving navigation, for example, after the embodiment shown in (B) of FIG. 4 , if the electronic device 100 is currently in a first preset situation, for example, the electronic device 100 is subject to GNSS interference and the electronic device 100 does not turn on the safe navigation mode, the navigation route and navigation information of the electronic device 100 change. At this time, the electronic device 100 can display a navigation interface after being subject to GNSS interference. For details, please refer to the user interface 430 shown in (C) of FIG. 4 .
[0155] As shown in (C) of Figure 4, the user interface 430 may include a location identifier 431A, a navigation route 431B, and navigation information 432. The location identifier 431A may identify the location of the electronic device 100 on the map, and the navigation route 431B may represent the changed navigation route. Assuming that the electronic device 100 is at the location identified by the location identifier 431A, the electronic device 100 is subject to GNSS interference. At this time, the navigation route of the electronic device 100 is changed from the original navigation route (for example, a route that stays straight in lane 421B) to navigation route 431B. Specifically, the lane in which the electronic device 100 is traveling may change, for example, the electronic device 100 switches from lane 421B to lane 421A, and the driving prompt information included in the navigation information 432 is displayed as "lane change is about to take place ahead" to instruct the electronic device 100 to change from lane 421B to lane 421A.
[0156] In one embodiment, after the embodiment shown in (C) of FIG. 4 , the electronic device 100 may display an interface for traveling along a changed navigation route (e.g., the navigation route 431B in the user interface 430 shown in (C) of FIG. 4 ). When the electronic device 100 detects that it is currently in the first preset situation, the electronic device 100 may display an alarm user interface to prompt the user to turn on the safe navigation mode. For details, see the user interface 440 shown in (A) of FIG. 5 .
[0157] As shown in (A) of Figure 5, the user interface 440 is similar to the user interface 430 shown in (C) of Figure 4. The difference is that the location marker 441A in the user interface 440 can indicate that the electronic device 100 is currently at the dividing line between lane 421A and lane 421B, rather than in the previous lane 421B. The location identified by the location marker 441A can be a false location on the map when the electronic device 100 is subject to GNSS interference. The navigation information 442 and the navigation route 431B in the user interface 440 can indicate that the electronic device 100 is traveling along the navigation route 431B, for example, from the location indicated by the location marker 431A on the map to the location indicated by the location marker 441A. However, at this time, the actual driving route of the electronic device 100 is the above-mentioned original navigation route (for example, a route that stays straight in lane 421B). Specifically, the electronic device 100 can travel from the location indicated by the location marker 431A to the location indicated by the location marker 441B (not displayed in the user interface 440). The location identified by the location marker 441B can be the actual location on the map when the electronic device 100 is subject to GNSS interference. The user interface 440 may also include a window 443, which may include prompt information (for example, including the characters "There is a problem with the current navigation positioning", "Please confirm whether to turn on the safe navigation mode"), a control 443A and a control 443B. The control 443A can be used to turn on the safe navigation mode, and the control 443B can be used to refuse to turn on the safe navigation mode.
[0158] In one embodiment, after the electronic device 100 turns on the safe navigation mode, it can relocate the position of the electronic device 100, update the navigation route, and automatically drive according to the updated navigation route. For example, the electronic device 100 can respond to a user operation (for example, the user operation is a touch operation) on the control 443A in the user interface 440 shown in Figure 5 (A) to display the user interface 450 shown in Figure 5 (B). As shown in Figure 5 (B), the user interface 450 may include prompt information 451 and control 452. The prompt information 451 displays the characters "Switched to safe navigation mode" and "Repositioning". The prompt information 451 and control 452 can indicate that the electronic device 100 has turned on the safe navigation mode (for example, switched from the automatic driving mode to the safe navigation mode) and relocated the position of the electronic device 100 on the map. The user interface after the electronic device 100 repositions and updates the navigation route is, for example, the user interface 460 shown in Figure 5 (C). As shown in FIG5(C), user interface 460 is similar to user interface 450 shown in FIG5(B), except that navigation route 461B in user interface 460 can represent the updated navigation route of electronic device 100 in safe navigation mode. For example, the navigation route is updated from: electronic device 100 switches from lane 421B to lane 421A and keeps going straight in lane 421A to: electronic device 100 switches from lane 421A to lane 421B and keeps going straight in lane 421B. This navigation route can represent the planned route of electronic device 100 on the actual driving road. Location identifier 461A in user interface 460 can identify the relocated position of electronic device 100, which can represent the position of electronic device 100 on the actual driving road.
[0159] In one embodiment, if the electronic device 100 detects that it is currently in a second preset situation, for example, when the GNSS interference to the electronic device 100 ends, the electronic device 100 can display a user interface indicating the end of the alarm to prompt the user to turn on the automatic driving mode. For details, please refer to the user interface 470 shown in (A) of Figure 6.
[0160] As shown in (A) of Figure 6 , the user interface 470 is similar to the user interface shown in (C) of Figure 5 , except that the user interface 470 also includes a window 471, which may include prompt information (for example, including characters such as "Current navigation positioning has been restored", "Please confirm whether to turn on the automatic driving mode"), a control 471A and a control 471B, wherein the control 471A can be used to turn on the automatic driving mode, and the control 471B can be used to refuse to turn on the automatic driving mode.
[0161] In one embodiment, the electronic device 100 may display a user interface for autonomous driving navigation in response to a user operation on the control 471A (e.g., the user operation is a touch operation), specifically, see the user interface 480 shown in FIG6(B). As shown in FIG6(B), the user interface 480 is similar to the user interface 420 shown in FIG4(B), except that the navigation information 481 in the user interface 480 is different from the navigation information 422 in the user interface 420, and the location of the electronic device 100 on the map and the current autonomous driving route of the electronic device 100 in the user interface 480 are different from those in the user interface 420.
[0162] Without being limited to the above-mentioned embodiment, in another embodiment, the user can set the permissions of the navigation application through the system setting function of the electronic device 100, or through the setting function of the navigation application, for example, allowing the navigation application to automatically turn on the safe navigation mode. For details, please refer to the user interface 490 shown in (C) of Figure 6. As shown in (C) of Figure 6, the user interface 490 may include a setting name 491 and a control 492. The setting name 491 displays the characters "Safe Navigation Mode". The control 492 can be used to turn on or off the safe navigation mode of the electronic device 100. The control 492 in the user interface 490 is in the turned-on state. In some examples, after the electronic device 100 turns on the safe navigation mode, the electronic device 100 may provide: when it is detected that the current situation is a first preset situation, the function of automatically turning on the safe navigation mode is provided. For a specific example, please refer to the embodiment shown in Figure 5.
[0163] Without being limited to the above examples, in other examples, when the electronic device 100 does not turn on the safe navigation mode in response to a user operation, the electronic device 100 can turn off the automatic navigation mode. For example, the electronic device 100 can turn off the automatic navigation mode of the current electronic device 100 in response to a user operation (e.g., a touch operation) on the control 443B in the user interface 440 shown in FIG5 (A). At this time, the electronic device 100 stops automatic driving and displays a prompt message to prompt the user to take over the electronic device 100.
[0164] Without being limited to the above examples, in other examples, when the electronic device 100 does not turn on the automatic driving mode in response to a user operation, the electronic device 100 can continue to automatically drive according to the updated navigation route. At this time, the electronic device 100 can display the user interface of the automatic driving in the safe navigation mode. For example, the electronic device 100 can respond to a user operation on the control 471B in the user interface 470 shown in (A) of FIG6 (for example, the user operation is a touch operation) and display a user interface similar to the user interface 480 shown in (B) of FIG6 , except that the navigation route of the user interface is different from the navigation route in the user interface 480 shown in (B) of FIG6 , and the navigation route is the updated navigation route in the safe navigation mode.
[0165] Without being limited to the above examples, in other examples, when the electronic device 100 does not receive a user operation for turning on or off the safe navigation mode within a first preset time period, the electronic device 100 may turn on the safe navigation mode by default. For example, after the electronic device 100 displays the user interface 440 shown in FIG5(A), if no user operation (for example, the user operation is a touch operation) on the control 443A or the control 443B is received within a first preset time period (for example, 10 seconds), the electronic device 100 may automatically turn on the safe navigation mode and display the user interface 450 shown in FIG5(B).
[0166] Next, the navigation fault detection method provided by the embodiment of the present application is introduced.
[0167] Please refer to Figure 7, which is a flowchart of a navigation fault detection method provided by an embodiment of the present application. The method can be applied to the electronic device 100 shown in Figure 1. The method can also be applied to the electronic device 100 shown in Figure 3. The method may include but is not limited to the following steps:
[0168] S101: A sensor acquires first data.
[0169] In one embodiment, the first data may include multiple observations / observation values measured by sensors at time k, where the sensors include, but are not limited to, GNSS receivers, IMUs, antennas, lidars, cameras, etc., and the multiple observations include, but are not limited to, the position of the electronic device 100 measured by the GNSS receiver, the acceleration and angular velocity of the electronic device 100 measured by the IMU, and the distance measured by the lidar, etc.
[0170] In one embodiment, the electronic device 100 can receive a user operation and, in response to the user operation, for example, in response to a user operation on the control 414A in the user interface 410 shown in (A) of FIG4 , turn on the automatic driving / navigation function, at which time, the sensor in the electronic device 100 obtains the first data.
[0171] S102: The sensor sends first data to the MSF positioning algorithm module.
[0172] In one embodiment, the MSF positioning algorithm may be an algorithm based on a Kalman filter.
[0173] S103: The sensor sends first data to the recognition and suppression algorithm module.
[0174] S104: The sensor sends first data to the GNSS interference detection module.
[0175] The order of any two steps in S102, S103 and S104 is not limited.
[0176] S105: The MSF positioning algorithm module obtains first status information according to the first data.
[0177] In one embodiment, the first state information is the state information of the electronic device 100 at the corresponding moment obtained based on the first data. The first state information and the time corresponding to the first data are the same. The first state information includes, for example but not limited to, information such as the position, posture, and speed of the electronic device 100. In some examples, the MSF positioning algorithm module can estimate the first state information at moment k based on multiple observation quantities received at moment k. It can be understood that the first state information is the state estimate value obtained using the MSF positioning algorithm at moment k.
[0178] S106: The MSF positioning algorithm module obtains first navigation information according to the first state information.
[0179] In one embodiment, the first navigation information includes, for example but not limited to, the first real-time position of the electronic device 100, the first route information, etc., wherein the first real-time position and the first route information are related. For example, the MSF positioning algorithm module can determine the first route information based on the first real-time position of the electronic device 100 at time k and the destination position.
[0180] S107: The MSF positioning algorithm module outputs first navigation information.
[0181] In one embodiment, after the MSF positioning algorithm module obtains the first navigation information, it can output the first navigation information, for example, by displaying the first navigation information through the display screen of the electronic device 100, such as displaying the user interface 420 shown in (B) of Figure 4, wherein the location identifier 421D in the user interface 420 indicates the first real-time location, and the navigation route 421E indicates the first route information.
[0182] S108: The identification and suppression algorithm module obtains second state information according to the first data.
[0183] In one embodiment, the second state information is the state information of the electronic device 100 at the corresponding moment obtained based on the first data. The second state information is the same as the time corresponding to the first data. The second state information includes, for example but not limited to, information such as the position, posture, and speed of the electronic device 100. In some examples, the recognition and suppression algorithm module can estimate the second state information at moment k based on multiple observation quantities received at moment k. It can be understood that the second state information is the state estimation value obtained using the recognition and suppression algorithm at moment k.
[0184] In one embodiment, the identification and suppression algorithm module may input multiple observations at time k into a factor graph-based multi-sensor fusion framework, and output an optimal estimate at time k, i.e., the aforementioned second state information. In some examples, the identification and suppression algorithm module may estimate the optimal estimate at the current moment based on observations at historical moments and observations at the current moment, where the historical moments may include k-1 different moments within a second preset duration before the current moment. For example, if the current moment is k, the historical moments may include 20 different moments within 20 seconds before k, where one second corresponds to one moment.
[0185] In one embodiment, a factor graph is a graphical model that can intuitively represent the information transfer relationship between state variables in the state space through nodes and edges (i.e., the connections between nodes). It can be understood that the multi-sensor fusion framework based on the factor graph can combine multiple observations (also called measurement factors) to construct a maximum likelihood estimation problem to obtain the optimal estimate value. An example of the factor graph can be seen in Figure 8.
[0186] As shown in Figure 8, node x i is the state factor to be estimated, node b i is the bias factor, node f i,imu is the IMU pre-integration factor, node f i,bias is the IMU bias factor, node f i,2D / 3D is the two-dimensional (2D) / three-dimensional (3D) visual positioning factor, node f i,gps is the GPS factor, node fprior,state is the initial state factor, and node fprior,bias is the initial bias factor, where i is a positive integer greater than or equal to 1 and less than or equal to k. Each state factor x to be estimated i It is related to the factors that have a connection relationship with it. Take the state factor x1 to be estimated as an example to illustrate. x1 is related to f 1,imu 、f 1,2D / 3D ,fprior,state related, where f 1,imuRelated to b1, b1 and f 1,bias ,fprior,bias is related.
[0187] In some examples, the factor graph model can be transformed into a nonlinear problem using maximum a posteriori probability estimation. The specific implementation process can be seen in the following equations (1), (2) and (3): x (∏ j f j (x j )) (1)
[0188] Among them, x j represents the jth state variable to be estimated, z j represents the jth observation, f j (x j ) represents the j-th factor node, X={x1,x2,…,x k} represents the set of state variables to be estimated, h j (·) represents the jth observation z j Related measurement equations, such as h j (x j ) is the state variable x to be estimated j and the observed quantity z j The observation function between j represents the measurement variance, one j corresponds to a moment, j∈[0,n], n represents the sliding window size, which can be understood as the number of historical moments, and n can be k-1. k is the first data at the kth moment, x k is the second state information at the kth moment.
[0189] Formula (1) and formula (2) can be converted into a nonlinear least squares problem. The nonlinear least squares problem can be seen in the following formula (3):
[0190] Next, solve formula (3) to get the optimal estimated value x k (ie, the second state information).
[0191] The multi-sensor fusion framework based on factor graph can take into account the observations at historical moments and continuously optimize the optimal solution of the nonlinear problem through repeated iteration and relinearization to obtain the optimal estimate.
[0192] The order of S105 and S108 is not limited.
[0193] S109: The identification suppression algorithm module obtains second navigation information according to the second state information.
[0194] In one embodiment, the second navigation information includes, for example but not limited to, the second real-time position of the electronic device 100, the second route information, etc., wherein the second real-time position and the second route information are related. For example, the recognition and suppression algorithm module can determine the second route information based on the second real-time position of the electronic device 100 at time k and the end point position.
[0195] The order of S106 and S109 is not limited, and the order of S107 and S109 is not limited.
[0196] S110: The identification and suppression algorithm module sends second status information to the GNSS interference detection module.
[0197] The order of any two steps in S110, S105 and S108 is not limited.
[0198] S111: The GNSS interference detection module obtains second data according to the first data and the second status information, and detects whether GNSS interference is caused according to the second data.
[0199] In one embodiment, the GNSS interference detection module can obtain the second data based on the first data, the historical data corresponding to the first data, the second state information, and the historical state information corresponding to the second state information, wherein the first data can be an observation value at time k, the historical data corresponding to the first data can be an observation value at a historical moment before time k (time t, where t is a positive integer greater than or equal to 1 and less than or equal to k-1, and k-1 is the number of historical moments), the second state information can be an estimated value at time k, and the historical state information corresponding to the second state information can be an estimated value at a historical moment before time k (time t). Optionally, the historical data corresponding to the first data and the historical state information corresponding to the second state information can be stored in the electronic device 100.
[0200] In one embodiment, the second data may be a residual sequence consisting of the difference between the observed value and the estimated value at the same time, for example, the difference between the first data at time k and the second state information at time k, and the difference between the first data at the same time k before time k (time t) and the second state information at time t. In some examples, it is assumed that the GNSS measurement model can refer to the following equations (4) and (5): k =H k X k +V k +ρ k (4)
[0201] Among them, z k is the observed value at time k, H k is the discrete measurement matrix, Xk is the estimated value at time k, V k is the discrete measurement noise, ρ k is a discrete measurement disturbance.
[0202] Through formula (4) and formula (5), the residual of the GNSS measurement factor can be seen in the following formula (6):
[0203] Among them, v k is the GNSS measurement residual at time k, and the GNSS measurement residuals within multiple sliding windows (i.e., multiple historical moments) in the factor graph model are selected to form the residual sequence {v j |j=1,2,3,…,k}, wherein an example of a factor graph corresponding to a GNSS residual sequence can be seen in FIG9. Referring to FIG9, FIG9 is a schematic diagram of the structure of a factor graph corresponding to a GNSS residual sequence provided in an embodiment of the present application.
[0204] As shown in Figure 9, node x i is the state factor to be estimated, node f i,imu is the IMU pre-integration factor, node f i,2D / 3D is the 2D / 3D visual positioning factor, node f i,gps is the GPS factor. Where i is a positive integer greater than or equal to 1 and less than or equal to k, the dotted box represents the sliding window size, and the residual v i Related to the IMU pre-integration factor, 2D / 3D visual positioning factor, and GPS factor.
[0205] In one embodiment, the GNSS interference detection module can analyze the second data in a first manner to obtain a first interference statistic. The first manner includes, for example but not limited to, taking the mean, variance, expectation, etc., and comparing the first interference statistic with a first threshold. When the first interference statistic is greater than or equal to the first threshold, it is determined that GNSS interference is caused. At this time, S112 is executed; when the first interference statistic is less than the first threshold, it is determined that GNSS interference is not caused. S101-S111 are executed again. At this time, the time corresponding to the first data obtained is different from the k time. Accordingly, the time corresponding to the data obtained according to the first data is also different from the k time.
[0206] In some examples, the residual sequence is composed of GNSS measurement residuals, and the first method is the statistical test method of exponentially fading sequential probability ratio. When there is no interference in the GNSS measurement model, the residual sequence should obey the normal distribution N(0, Σ). At this time, the expectation and variance of the residual can be seen in the following formula (7):
[0207] Among them, E(v k) represents the expected residual at time k, represents the variance of the residual at time k.
[0208] When interference occurs in the GNSS measurement model, the expected value and variance of the residual can be found in the following equation (8):
[0209] According to formula (7) and formula (8), the following hypothesis test is proposed: H0: ρ = 0, the GNSS measurement model is normal; H1: ρ ≠ 0, interference occurs in the GNSS measurement model. The probability density of the residual samples under the two probability assumptions can be seen in the following formulas (9) and (10):
[0210] in,
[0211] According to the probability densities of the two hypotheses in formula (9) and formula (10), the likelihood ratio under the two hypotheses can be calculated as shown in the following formula (11):
[0212] Next, to L k By calculating the logarithm, we can obtain the first interference statistic of the sequential probability ratio under the factor graph model optimization, which can be seen in the following formula (12):
[0213] The change rate of the first interference statistic at each moment can be found in the following formula (13):
[0214] Therefore, the first interference statistic of the sequential probability ratio can also be represented by an iterative form, as shown in the following formula (14): k =λ k-1 +Δλ k (14)
[0215] The residual mean at time k can be found in the following formula (15):
[0216] in, represents the fading weighting factor.
[0217] It can be seen from formula (15) that the introduction of the fading weighting factor can increase the residual v at time k k The weight of , reducing the residual mean of historical moments The weight of can be understood as increasing the weight of the residual at time k in the residual mean and decreasing the weight of the residual at previous moments in the residual mean when GNSS interference occurs. This increases the rate of change of the residual mean when GNSS interference occurs, making the residual mean more sensitive to the onset of GNSS interference, thereby determining the onset of GNSS interference. In other words, by analyzing the residual sequence, if the change in the residual mean at the current moment is greater than or equal to a first threshold, it is determined that GNSS interference has begun at the current moment. The first interference statistic can indicate the residual mean.
[0218] In one embodiment, when the electronic device 100 begins to experience GNSS interference, the first real-time position in the first navigation information estimated by the MSF positioning algorithm module cannot represent the real-time position of the electronic device 100 during navigation. Therefore, the navigation route information of the electronic device 100 may change. In some examples, the user interface of the electronic device 100 during autonomous driving is, for example, user interface 420 shown in FIG. 4(B), and the user interface of the electronic device 100 when it begins to experience GNSS interference is, for example, user interface 430 shown in FIG. 4(C).
[0219] In one embodiment, when the electronic device 100 enables the autonomous driving / navigation function, the GNSS interference detection module can be automatically enabled to enable the GNSS interference identification and suppression detection function. For example, the electronic device 100 can receive user operations and set permissions for the navigation application during the autonomous driving process based on the user operations, such as allowing the navigation application to automatically enable the GNSS interference identification and suppression detection function. Alternatively, the electronic device 100 may not enable the GNSS interference detection module. In this case, the electronic device 100 does not execute S110-S113. Optionally, the electronic device 100 can enable the GNSS interference detection module during the autonomous driving process to enable the GNSS interference identification and suppression detection function.
[0220] S112: The GNSS interference detection module outputs a first prompt message.
[0221] In one embodiment, when the GNSS interference detection module detects that the electronic device 100 is subject to GNSS interference, the GNSS interference detection module can output a first prompt message, for example, by displaying the first prompt message through the display screen of the electronic device 100. The first prompt message can be used to prompt the user that the current electronic device 100 is subject to GNSS interference, for example, by displaying the user interface 440 shown in (A) of Figure 5.
[0222] S113: When the first preset condition is met, the recognition suppression algorithm module outputs the second navigation information.
[0223] In one embodiment, the first preset condition may be that the electronic device 100 has turned on the safe navigation mode. When the first preset condition is met, the identification suppression algorithm module can output the second navigation information, for example, display the second navigation information through the display screen of the electronic device 100.
[0224] In some examples, when the electronic device 100 does not turn on the safe navigation mode, the first prompt information can also be used to prompt the user to turn on the safe navigation mode. The electronic device 100 can respond to user operations, such as a touch operation on the control 443A in the user interface 440 shown in (A) of Figure 5, to turn on the safe navigation mode. At this time, the recognition suppression algorithm module can output the second navigation information, for example, the electronic device 100 displays the user interface 460 shown in (C) of Figure 5.
[0225] In other examples, when the electronic device 100 turns on the automatic driving / navigation function, the safe navigation mode can be automatically turned on. For example, the electronic device 100 can receive user operations and set permissions for the navigation application during the automatic driving process based on the user operations, such as allowing the navigation application to automatically turn on the safe navigation mode. For a specific example, see user interface 490 shown in (C) of Figure 6. In this case, after the GNSS interference detection module outputs the first prompt information, it can directly output the second navigation information.
[0226] S114: When the first preset condition is not met, the MSF positioning algorithm module outputs first navigation information.
[0227] In one embodiment, when the first preset condition is not met, the MSF positioning algorithm module can output the first navigation information. At this time, the electronic device 100 can display a prompt message to remind the user that it is currently subject to GNSS interference, and the navigation positioning and navigation route information may be erroneous. The user can choose to manually drive the electronic device 100.
[0228] In some examples, when the electronic device 100 does not have the safe navigation mode turned on, for example, the electronic device 100 can respond to the user operation of the control 443B in the user interface 440 shown in (A) of Figure 5 to turn off the automatic driving mode and cancel the safe navigation mode. At this time, the electronic device 100 can display a prompt message to remind the user that it is currently subject to GNSS interference and ask the user to take over / manually drive the electronic device 100.
[0229] The above embodiments are described using autonomous driving as an example, but are not limited thereto. The present application can also be applied to manual driving scenarios, and the specific description is similar to the above description.
[0230] In the method shown in Figure 7, the GNSS interference detection module can use a detection method based on exponentially fading sequential probability ratios to promptly detect slowly varying soft faults caused by GNSS interference. This method monitors the onset of GNSS interference, detects signal anomalies before a dangerous collision occurs in an electronic device, and alerts the user that their current driving safety is about to be threatened, thereby improving driving safety, enhancing fault detection accuracy, and reducing the rate of missed faults. Furthermore, the factor graph-based multi-sensor fusion framework avoids the assumptions of linearization and Gaussian noise in the measurement model, improving the accuracy of multi-sensor fusion. This ensures that the real-time position obtained by the identification and suppression algorithm is closer to the actual position of the electronic device when it is experiencing GNSS interference, enhancing the robustness and stability of the multi-sensor fusion framework and improving the positioning accuracy of the electronic device during autonomous driving.
[0231] In addition, the GNSS interference detection module can distinguish between abnormal errors caused by GNSS interference and errors caused by natural factors (such as GNSS positioning errors caused by multipath, positioning errors caused by sensor failures such as IMU / lidar, etc.). It will only alarm when it is determined that the abnormal error is caused by GNSS interference, avoiding false alarms and further improving the accuracy of fault detection.
[0232] Please refer to Figure 10, which is a flowchart of another navigation fault detection method provided by an embodiment of the present application. The method can be applied to the electronic device 100 shown in Figure 1. The method can also be applied to the electronic device 100 shown in Figure 3. The method may include but is not limited to the following steps:
[0233] S201: The sensor obtains first data.
[0234] In one embodiment, before S201, S101-S114 are also included. When the electronic device 100 detects that it is subject to GNSS interference, the sensor in the electronic device 100 can continuously obtain the first data. For example, if the electronic device 100 detects that it is subject to GNSS interference at time k, the sensor in the electronic device 100 can obtain the first data at time m (m is greater than k). In this case, the first data may include multiple observation quantities / observation values measured by the sensor at time m. For detailed description, please refer to the description of S101 in Figure 7, which will not be repeated here.
[0235] S202: The sensor sends first data to the GNSS interference detection module.
[0236] S203: The sensor sends first data to the recognition and suppression algorithm module.
[0237] The order of S202 and S203 is not limited.
[0238] S204: The identification and suppression algorithm module obtains second state information according to the first data.
[0239] In one embodiment, the description of S204 is similar to the description of S108 in FIG. 7 , except that the second state information in S204 is an estimated value obtained at time m using the recognition suppression algorithm.
[0240] S205: The identification and suppression algorithm module obtains second navigation information according to the second state information.
[0241] In one implementation, the description of S204 is similar to the description of S109 in FIG. 7 , except that the second navigation information in S205 is determined based on the second state information at time m.
[0242] S206: The GNSS interference detection module obtains third data according to the first data, and detects whether the GNSS interference ends according to the third data.
[0243] In one embodiment, the GNSS interference detection module can calculate the third data based on the first data, wherein the first data includes, for example but not limited to, observation values obtained by IMU measurement, observation values obtained by lidar measurement, observation values obtained by camera measurement, etc.
[0244] In one embodiment, the third data may be residual information formed by the difference between the observed value and the estimated value at the same time. The specific description of the third data is similar to the description of the second data in S111 of Figure 7, except that the estimated value here is an estimated value estimated based on the observed value of sensors other than the GNSS receiver (such as IMU and lidar). An example of the factor graph corresponding to the residual information can be seen in Figure 11 below. Please refer to Figure 11, which is a schematic diagram of the structure of a factor graph corresponding to residual information provided in an embodiment of the present application.
[0245] As shown in Figure 11, the structure of the factor graph shown in Figure 11 is similar to that shown in Figure 9, except that Figure 11 also includes node f i,lidar LiDAR factor, residual information Z i Related to the IMU pre-integration factor, 2D / 3D visual positioning factor, GPS factor, and lidar factor, where i is a positive integer greater than or equal to 1 and less than or equal to m.
[0246] In some examples, it is assumed that the residual information of the measurement model is recalculated / reconstructed based on the state variable nodes of the IMU, lidar and / or camera in the factor graph. The residual information can be referred to in the following formula (16): m =Z gps,m -ximu / lidar,m (16)
[0247] Among them, z gps,mrepresents the observation value of the Global Positioning System (GPS) at time m, ximu / lidar,m represents the estimated value at time m, and Z m Represents the residual at time m.
[0248] In some examples, the residual is the above-mentioned residual information, and the second method is the residual chi-square detection method. According to formula (16), the chi-square detection statistic (also called the second interference statistic) is established, which can be seen in the following formula (17):
[0249] Among them, ∑ -1 represents the variance of the residual.
[0250] In one embodiment, the GNSS interference detection module can detect whether the GNSS interference at the current moment has ended based on the third data and the detection result at the previous moment. In some examples, the GNSS interference detection module can combine the first method and the second method to detect whether the GNSS interference has ended. Please refer to Figure 12, which is a logical diagram of a combined first method and second method for detecting the end of GNSS interference provided by an embodiment of the present application. As shown in Figure 12, when the detection result at the previous moment (m-1 moment) is that GNSS interference has occurred (GNSS interference has not ended), the GNSS interference detection module can first analyze the second data at the current moment (m moment) using the first method to obtain the first interference statistic at m moment. The specific implementation process can be seen in S111 of Figure 7, which will not be repeated here. When the first interference statistic at time m is greater than or equal to the first threshold, the third data is analyzed by a second method to obtain a second interference statistic, the second method including, for example but not limited to, a residual chi-square detection method, and comparing the second interference statistic with the second threshold. When the second interference statistic is less than the second threshold, it is determined that the GNSS interference has ended (the end time is time m). At this time, S207 is executed; when the second interference statistic is greater than or equal to the second threshold, it is determined that the GNSS interference has not ended (i.e., it is still subject to GNSS interference). At this time, S208 is executed, wherein the specific implementation method of the detection result at the previous moment is the same as the implementation method at the current moment (m), i.e., the first method and the second method are combined for detection. Optionally, when the second interference statistic at time m is less than the second threshold, it is determined that the GNSS interference has not ended.
[0251] The order of S204 and S206 is not limited, and the order of S205 and S206 is not limited.
[0252] S207: The GNSS interference detection module outputs a second prompt message.
[0253] In one embodiment, when the GNSS interference detection module detects that the GNSS interference has ended, the GNSS interference detection module may output a second prompt message, for example, by displaying the second prompt message on the display screen of the electronic device 100. The second prompt message may be used to prompt the user that the GNSS interference currently affecting the electronic device 100 has ended, for example, by displaying the user interface 470 shown in (A) of FIG6 .
[0254] In one embodiment, the second prompt information can also be used to prompt the user to turn on the automatic driving mode. In some examples, the electronic device 100 can switch from the safe navigation mode to the automatic driving mode in response to a user operation on the functional control for turning on the automatic driving mode, such as a touch operation on the control 471A in the user interface 470 shown in FIG6 (A). At this time, the MSF positioning algorithm module can output the first navigation information. For example, the electronic device 100 displays the user interface 480 shown in FIG6 (B). In other examples, the electronic device 100 can cancel the automatic driving mode and continue to remain in the safe navigation mode in response to the user operation on the control 471B in the user interface 470 shown in FIG6 (A). At this time, the recognition suppression algorithm module can output the second navigation information. Without limitation, in some other examples, the electronic device 100 can cancel the automatic driving mode and turn off the safe navigation mode according to the user operation. At this time, the user can choose to manually drive the electronic device 100.
[0255] S208: When the first preset condition is met, the recognition suppression algorithm module outputs the second navigation information.
[0256] In one embodiment, when the GNSS interference detection module detects that the GNSS interference has not ended, the electronic device 100 is still subject to GNSS interference. When the first preset condition is met, the identification and suppression algorithm module can continue to output the second navigation information. The specific description is similar to that of S113 in Figure 7 and will not be repeated here.
[0257] In one embodiment, after S208, the electronic device 100 may execute S201-S206 again. At this time, the time corresponding to the acquired first data is different from time m. Accordingly, the time corresponding to the data obtained according to the first data is also different from time m.
[0258] Not limited to the above embodiment, in another embodiment, after S207, if the navigation of the electronic device 100 has not ended, the electronic device 100 may execute S101-S111 of FIG. 7 again.
[0259] In one embodiment, an example diagram of the electronic device 100 detecting GNSS interference and the end of GNSS interference can be seen in Figure 13. Referring to Figure 13, Figure 13 is a schematic diagram of a GNSS interference detection process provided by an embodiment of the present application.
[0260] As shown in FIG13 , assuming that the electronic device 100 is at position 1 and is subject to GNSS interference, the navigation route of the electronic device 100 will change, for example, from a straight-line route in lane b to a route that switches from lane b to lane a. The electronic device 100 will then travel along the changed route (first navigation information). During the process of traveling from position 1 to position 2, when the electronic device 100 reaches an offset boundary, the electronic device 100 detects that the offset distance A is greater than or equal to a first offset threshold. The electronic device 100 determines that it is subject to GNSS interference and can display a warning message to inform the user that it is currently subject to GNSS interference. Assuming that the electronic device 100 is subject to GNSS interference and does not enable safe navigation mode, the electronic device 100 continues to travel along the changed route (for example, from position 2 to position 3). During the process of traveling from position 3 to position 4, when the electronic device 100 reaches an offset boundary, the electronic device 100 detects that the offset distance B is less than a second offset threshold. The electronic device 100 determines that the GNSS interference has ended and can display a warning message to inform the user that the GNSS interference has ended. Among them, the offset distance A can indicate the first interference statistic in Figure 7, the first offset threshold can indicate the first threshold in Figure 7, the offset distance B can indicate the second interference statistic in Figure 10, and the second offset threshold can indicate the second threshold in Figure 10.
[0261] Not limited to the above examples, in other examples, after the electronic device 100 detects that the offset distance A is greater than or equal to the first offset threshold, if the safe navigation mode has been turned on, it will travel according to the navigation route when it is not affected by GNSS interference (corresponding to the second navigation information above). When the electronic device 100 detects that the offset distance B is less than the second offset threshold, it is determined that the GNSS interference has ended, and the electronic device 100 can display the end of the alarm.
[0262] In the method shown in Figure 10, the GNSS interference detection module can use a combined exponentially fading sequential probability ratio and chi-square test method to promptly detect sudden hard faults in GNSS interference. This allows the module to monitor the end of the GNSS interference, avoiding false alarms caused by untimely detection, which can render normal GNSS signals unusable for navigation and positioning. Upon detecting the end of the GNSS interference, the module displays an alarm end message, notifying the user that the current driving safety threat has been resolved. Furthermore, during the GNSS interference process, the electronic device can display the real-time location obtained by the identification and suppression algorithm, preventing the user from being affected by erroneous positioning information output by the GNSS interference, further improving the safety of the electronic device during autonomous driving and ensuring user safety.
[0263] The methods provided in the various embodiments of the present application may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, they may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a digital video disc (DWD), or a semiconductor medium (e.g., a solid state drive (SSD)). As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the same. Although the present application has been described in detail with reference to the above embodiments, a person skilled in the art should understand that the technical solutions described in the above embodiments may be modified, or some of the technical features thereof may be replaced by equivalents. However, such modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A navigation fault detection method, characterized in that: Applied to a first device, the method includes: Acquire first data, where the first data includes a plurality of observation values at a first moment respectively measured by a plurality of sensors in the first device; Obtaining first state information of the first device at the first moment according to the first data; detecting, according to the first data and the first status information, whether the first device is interfered with by a global navigation satellite system GNSS; When it is detected that the first device is subject to GNSS interference, a first interface is displayed, wherein the first interface includes first prompt information, and the first prompt information indicates that the first device is subject to GNSS interference.
2. The method according to claim 1, characterized in that The detecting, according to the first data and the first status information, whether the first device is subject to GNSS interference includes: Whether the first device is subject to GNSS interference is detected according to the first data, the historical data corresponding to the first data, the first status information, and the historical status information corresponding to the first status information.
3. The method according to claim 2, characterized in that The detecting, according to the first data, the historical data corresponding to the first data, the first status information, and the historical status information corresponding to the first status information, whether the first device is subject to GNSS interference includes: Obtaining second data according to the first data, historical data corresponding to the first data, the second state information, and historical state information corresponding to the second state information; A first statistic of the first device at the first moment is obtained according to the second data, and when the first statistic is greater than or equal to a first threshold, it is determined that the first device is subject to GNSS interference.
4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: When it is detected that the first device is not affected by GNSS interference, a second interface is displayed, and the second interface includes a first position and a first route. The first position is the position of the first device at the first moment estimated based on the second state information, and the first route is the navigation route of the first device at the first moment estimated based on the second state information.
5. The method according to claim 4, characterized in that The method further comprises: When it is detected that the first device is subject to GNSS interference, a third interface is displayed, and the third interface includes a second position and a second route, the second position is the position of the first device at the first moment estimated based on the first status information, and the second route is the navigation route of the first device at the first moment estimated based on the first status information.
6. The method according to claim 5, characterized in that The first state information is obtained according to a first algorithm, and the second state information is obtained according to a second algorithm. The first algorithm is an identification suppression algorithm, and the second algorithm is a multi-sensor fusion MSF positioning algorithm.
7. The method according to claims 1 to 6, characterized in that The first state information includes at least one of the position, posture, speed, and time of the first device.
8. The method according to claim 1, characterized in that The method further comprises: When the first interface is displayed, receiving a first operation; In response to the first operation, a third interface is displayed, and the third interface includes a second position and a second route, the second position is the position of the first device at the first moment estimated based on the first state information, and the second route is the navigation route of the first device at the first moment estimated based on the first state information.
9. The method according to claim 1, characterized in that When it is detected that the first device is interfered with by the GNSS, displaying a first interface includes: When it is detected that the first device is subject to GNSS interference, if the defense function is not enabled, displaying the first interface; The method further comprises: When it is detected that the first device is interfered with by GNSS, if the defense function is turned on, a third interface is displayed, and the third interface includes The present invention includes a second position and a second route, wherein the second position is the position of the first device at the first moment estimated based on the first state information, and the second route is the navigation route of the first device at the first moment estimated based on the first state information.
10. The method according to claim 1, characterized in that The method further comprises: When the first interface is displayed, the driving mode of the first device is the automatic driving mode; After displaying the first interface, the method further includes: A second operation is received, where the second operation is used to switch the automatic driving mode to a manual driving mode.
11. The method according to claim 1, characterized in that: The detecting that the first device is interfered with by the GNSS includes: When it is detected that the offset distance is greater than or equal to the offset threshold, it is determined that the first device is subject to GNSS interference, the offset distance is a difference between a first position and a second position, the first position is a position estimated according to a first algorithm, the second position is a position estimated according to a second algorithm, the first algorithm is an identification suppression algorithm, and the second algorithm is an MSF positioning algorithm; When it is detected that the offset distance is less than the offset threshold, it is determined that the GNSS interference to the first device ends.
12. The method according to claim 11, characterized in that The method further comprises: When it is detected that the first device is not interfered with by the GNSS, the first device is located in a first lane; The offset distance is the distance between the first position estimated by the first algorithm and a lane dividing line, and the lane dividing line is a dividing line between the first lane and an adjacent lane.
13. The method according to any one of claims 1 to 12, characterized in that: The method further comprises: After detecting that the first device is subject to GNSS interference, acquiring second data, where the second data includes a plurality of observation values at a second moment respectively measured by the plurality of sensors, where the second moment is later than the first moment; detecting, according to the second data, whether the GNSS interference to the first device is ended; When it is detected that the GNSS interference suffered by the first device ends, a second prompt information is displayed, and the second prompt information indicates that the GNSS interference suffered by the first device ends.
14. The method according to claim 13, characterized in that The method further comprises: After detecting that the GNSS interference to the first device has ended, a fourth interface is displayed, wherein the fourth interface includes a third position and a third route, wherein the third position is the position of the first device at the second moment estimated based on the third state information, and the third route is the navigation route of the first device at the second moment estimated based on the third state information, and the third state information is obtained based on a second algorithm, and the second algorithm is an MSF positioning algorithm.
15. The method according to claim 4, characterized in that After displaying the second interface, the method further includes: When it is detected that the first device is subject to GNSS interference, if the defense function is not turned on, a fifth interface is displayed, and the fifth interface includes a fourth position and a fourth route. The fourth position and the fourth route are obtained according to a second algorithm. The time corresponding to the fourth position is later than the first time. The fourth position is different from the first position, and the fourth route is different from the first route. The second algorithm is an MSF positioning algorithm.
16. The method according to claim 13, characterized in that The detecting, according to the second data, whether the GNSS interference suffered by the first device is ended includes: Obtaining third data according to the second data and a first estimated value, wherein the first estimated value is obtained according to a portion of observed values in the second data, the first estimated value is obtained according to a first algorithm, the first algorithm is a recognition suppression algorithm, the first estimated value corresponds to the second moment, and the third data corresponds to the second moment; When it is detected at a third moment that the first device is subject to GNSS interference, obtaining a second statistic according to the second data, the third moment being a moment before the second moment, and the second statistic corresponding to the second moment; When the second statistic is greater than or equal to the first threshold, obtaining a third statistic according to the third data, wherein the third statistic corresponds to the second moment; When the third statistic is greater than or equal to a second threshold, it is determined that the GNSS interference to the first device ends.
17. An electronic device, characterized in that: The method comprises a transceiver, a processor and a memory, wherein the memory is used to store a computer program, and the processor calls the computer program to execute the method according to any one of claims 1 to 16.
18. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 17 is implemented.
Citation Information
Patent Citations
Anti-deception navigation method based on residual chi-square-improved sequential probability ratio
CN112902967A
Design method of UUV multi-sea-area integrated navigation system based on factor graph optimization
CN113984061A
Vehicle control method and device based on road side unit
CN114987542A
Multi-sensor fusion positioning method and system in unmanned complex scene
CN115265551A
GNSS-Based Timing Synchronization
US20200241146A1
Cited By
Fault cutting-based robust method
CN119779344A
Factor graph optimization train positioning method for resisting satellite navigation spoofing attack
CN120468909A
Compatibility verification method and system applied to Beidou positioning system
CN121432474A
