ARHUD-based perception and display methods, systems, media, and devices

Abstract

The application provides an ARHUD-based perception and display method, system, medium and equipment, and belongs to the technical field of automobile intelligent driving auxiliary display. The method comprises the following steps: acquiring collected multispectral sensor data; adopting a multisensor data fusion algorithm based on distribution transformation to fuse the multispectral sensor data to obtain fused multispectral data; constructing a deep learning model based on a convolutional neural network to perform target identification and classification on the fused multispectral data; adopting a Kalman filtering algorithm to combine deep learning feature matching to perform target tracking; and adopting a differentiated display mode and color division for different types of targets to optimize ARHUD display content and effect. The above method can solve the problem of insufficient perception ability of a single sensor in a harsh environment, realize more accurate target identification and tracking, optimize ARHUD display content and effect, provide comprehensive and accurate environmental situation awareness, and assist the driver in safe driving.

Description

Technical Field

[0001] This invention relates to the field of automotive intelligent driving assistance display technology, and in particular to a perception and display method, system, medium and device based on ARHUD. Background Technology

[0002] With the development of automotive intelligence, ARHUD, as an important driver assistance display technology, can project key driving information such as vehicle speed, navigation guidance, and collision warnings onto the windshield in augmented reality form, allowing drivers to access information without looking down, greatly improving driving safety and convenience. However, existing ARHUD systems have certain limitations in perception and display.

[0003] At the perception level, current technology primarily relies on single or a few types of sensors, such as visible light cameras. In adverse weather conditions, such as heavy fog, torrential rain, or darkness, the image quality of visible light cameras is severely affected, leading to a significant decrease in the accuracy of identifying targets such as roads, vehicles, and pedestrians. For example, in foggy weather, visible light is scattered by the fog, making the image blurry and difficult to accurately determine the distance and shape of obstacles ahead; in darkness, low light causes the loss of image details, weakening the ability to detect objects in dark areas.

[0004] At the display level, existing ARHUD displays are often based on limited perceptual information, failing to provide drivers with comprehensive and accurate environmental situational awareness. When the perception system misjudges or misses a judgment, the information displayed by the ARHUD may not match the actual road conditions, misleading the driver. Therefore, there is an urgent need to develop an ARHUD technology that can achieve reliable all-weather perception and provide more accurate and comprehensive display information based on multispectral fusion. Summary of the Invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art, and proposes a sensing and display method, system, medium and device based on ARHUD.

[0006] In a first aspect, embodiments of the present invention provide a perception and display method based on ARHUD, comprising the following steps:

[0007] S100: Acquire the collected multispectral sensor data;

[0008] S200. A multi-sensor data fusion algorithm based on distribution transformation is used to fuse multispectral sensor data to obtain fused multispectral data;

[0009] S300. Construct a deep learning model based on convolutional neural networks to perform target recognition and classification on the fused multispectral data;

[0010] S400 uses the Kalman filter algorithm combined with deep learning feature matching for target tracking;

[0011] S500 uses differentiated display methods and colors to distinguish different types of targets in order to optimize the content and effect of ARHUD display.

[0012] Furthermore, in step S100, the multispectral sensor data includes visible light images, infrared thermal imaging images, and millimeter-wave radar data.

[0013] Further, step S200 includes the following steps:

[0014] S210. Convert the format of the multispectral sensor data, unifying the grayscale values ​​and measured values ​​into vector form;

[0015] S220. Construct probability density function models for the multispectral sensor data respectively;

[0016] S230. Using the distribution transformation theory, construct a joint probability density function model for multi-source data based on the probability density function model of multispectral sensor data;

[0017] S240. Based on the parameters of the joint probability density function model, the maximum likelihood estimation method is used for calculation, and the target detection result after fusion is obtained by classification through the Bayesian criterion.

[0018] Furthermore, visible light images are modeled using Gaussian mixture models based on color and texture distribution, infrared thermal imaging images are modeled using exponential distribution models based on the statistical laws of thermal radiation intensity, and millimeter-wave radar data are modeled using normal distribution models based on the distribution characteristics of measured values.

[0019] Furthermore, for categories with insufficient training samples or excessive errors, the Adaboost compensation algorithm is used for optimization.

[0020] Further, step S400 includes the following steps:

[0021] S410. In the initial frame, the target to be tracked is determined by target recognition and its multispectral features are extracted; the target motion state is initialized using a Kalman filter, and the position, velocity, and acceleration parameters are set.

[0022] S420. In subsequent frames, the target position at the current moment is predicted based on the Kalman filter prediction formula and the target state at the previous moment. Then, a search area is defined in the current frame image with the predicted position as the center. The multispectral features of the target in this area are extracted by a deep learning model and matched with the target features in the initial frame. The similarity is calculated using methods such as Euclidean distance. The area with the highest similarity is selected as the actual position of the target in the current frame to complete the tracking.

[0023] Further, step S500 includes: acquiring vehicle acceleration and angular velocity motion attitude information in real time through vehicle inertial measurement unit sensors, and adjusting the attitude of the ARHUD display information accordingly.

[0024] Further, step S500 includes: measuring the time difference between data collected by the sensor and data displayed by the ARHUD, and using delay compensation algorithms such as linear interpolation to correct the timestamp of the displayed information.

[0025] Secondly, embodiments of the present invention provide a perception and display system based on ARHUD, comprising:

[0026] The acquisition module is used to acquire the collected multispectral sensor data;

[0027] The fusion module is used to fuse multispectral sensor data using a multi-sensor data fusion algorithm based on distribution transformation to obtain fused multispectral data;

[0028] The recognition module is used to build a deep learning model based on convolutional neural networks to perform target recognition and classification on the fused multispectral data;

[0029] The tracking module is used to track targets using a Kalman filter algorithm combined with deep learning feature matching.

[0030] The display module is used to differentiate the display methods and colors for different types of targets in order to optimize the content and effect of ARHUD display.

[0031] Thirdly, embodiments of the present invention provide an electronic device, including:

[0032] One or more processors;

[0033] Memory, used to store one or more programs;

[0034] When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described above.

[0035] Fourthly, embodiments of the present invention provide a computer-readable medium storing a computer program, which, when executed by a processor, implements the steps of the method described above.

[0036] The present invention provides an ARHUD-based perception and display method, system, medium, and device, which acquires multispectral sensor data; fuses the multispectral sensor data using a distribution transformation-based multisensor data fusion algorithm to obtain fused multispectral data; constructs a deep learning model based on a convolutional neural network to perform target recognition and classification on the fused multispectral data; uses a Kalman filter algorithm combined with deep learning feature matching for target tracking; and employs differentiated display methods and color distinctions for different types of targets to optimize the ARHUD display content and effect. It can fuse multispectral sensor data, leverage the advantages of each sensor, and solve the problem of insufficient perception capability of a single sensor in harsh environments. Through multispectral fusion and deep learning, it achieves more accurate target recognition and tracking, optimizes the ARHUD display content and effect, provides comprehensive and accurate environmental situational awareness, and assists drivers in safe driving. Attached Figure Description

[0037] Figure 1 A flowchart illustrating a perception and display method based on ARHUD provided in an embodiment of the present invention;

[0038] Figure 2 This is a flowchart of multispectral sensor data fusion provided in an embodiment of the present invention;

[0039] Figure 3 This is a diagram of a deep learning-based target recognition and tracking architecture provided in an embodiment of the present invention.

[0040] Figure 4 This is a schematic diagram of the optimized layout for ARHUD display provided in an embodiment of the present invention;

[0041] Figure 5 A structural block diagram of an ARHUD-based sensing and display system provided for an embodiment of the present invention;

[0042] Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0043] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0044] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0045] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0046] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0047] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0048] This invention provides a perception and display method based on ARHUD, see reference. Figure 1 As shown, the method includes the following steps:

[0049] S100: Acquire the collected multispectral sensor data.

[0050] In one embodiment, the multispectral sensor data includes visible light images, infrared thermal imaging images, and millimeter-wave radar data.

[0051] In this embodiment, multiple sensors are integrated, including a visible light camera, an infrared thermal imaging camera, and a millimeter-wave radar. The visible light camera is responsible for acquiring road scene images rich in texture and color details under good lighting conditions, providing basic visual information for the system. The infrared thermal imaging camera utilizes the thermal radiation characteristics of objects to accurately identify heat-generating targets, such as people and vehicles, in adverse weather conditions such as darkness, fog, rain, and snow, compensating for the imaging deficiencies of the visible light camera in low light or adverse environments. The millimeter-wave radar, using millimeter-wave electromagnetic waves, stably measures the distance, speed, and angle of target objects, possessing strong penetration and anti-interference capabilities, ensuring the reliability of perception under adverse weather conditions.

[0052] S200. A multi-sensor data fusion algorithm based on distribution transformation is used to fuse multispectral sensor data to obtain fused multispectral data.

[0053] In one embodiment, see Figure 2 As shown, step S200 includes the following steps:

[0054] S210. Convert the format of the multispectral sensor data, unifying the grayscale values ​​and measured values ​​into vector form; this facilitates subsequent processing.

[0055] S220. Construct probability density function (PDF) models for multispectral sensor data respectively. Specifically, considering the characteristics of visible light imaging data, infrared thermal imaging data, and millimeter-wave radar detection data, a Gaussian mixture model is constructed for visible light images based on color and texture distribution; an exponential distribution model is established for infrared thermal imaging images based on the statistical law of thermal radiation intensity; and a normal distribution model is constructed for millimeter-wave radar data based on the distribution characteristics of measured values.

[0056] S230. Using distribution transformation theory, a joint probability density function model of multi-source data is constructed based on the probability density function model of multispectral sensor data. In this embodiment of the invention, the correlation between multi-sensor data is analyzed, and a joint probability density function model of multi-source data is constructed using distribution transformation theory to comprehensively consider the interrelationship of the characteristics of each sensor data.

[0057] S240. Based on the parameters of the joint probability density function model, the maximum likelihood estimation method is used for calculation, and the target detection result after fusion is obtained by classification through the Bayesian criterion.

[0058] In one embodiment, the Adaboost compensation algorithm is used to optimize categories with insufficient training samples or excessive errors, thereby improving the accuracy and reliability of the fusion results.

[0059] S300. Construct a deep learning model based on convolutional neural networks to perform target recognition and classification on the fused multispectral data.

[0060] Specifically, the model is trained on a large dataset of road scene samples (including visible light images, thermal images, and corresponding radar data) covering various weather and lighting conditions, learning the unique patterns of different targets under multispectral features. After training, the model can accurately distinguish various targets such as pedestrians, cars, motorcycles, traffic signs, and traffic lights. When the fused data is input, the model outputs the predicted probability of each target category and selects the category with the highest probability as the recognition result.

[0061] The S400 uses a Kalman filter algorithm combined with deep learning feature matching for target tracking.

[0062] In one embodiment, see Figure 3 As shown, step S400 includes the following steps:

[0063] S410. In the initial frame, the target is identified through target recognition and its multispectral features are extracted; the target motion state is initialized using a Kalman filter, and the position, velocity, and acceleration parameters are set.

[0064] S420. In subsequent frames, the target position at the current moment is predicted based on the Kalman filter prediction formula and the target state at the previous moment. Then, a search area is defined in the current frame image with the predicted position as the center. The multispectral features of the target in this area are extracted by a deep learning model and matched with the target features in the initial frame. The similarity is calculated using methods such as Euclidean distance. The area with the highest similarity is selected as the actual position of the target in the current frame to complete the tracking.

[0065] This embodiment is used to deal with situations where targets occlude each other in complex scenes. By leveraging the complementarity of multispectral features, such as using infrared thermal imaging to detect the thermal radiation of the occluded target, the target trajectory can be continuously and accurately tracked.

[0066] S500 uses differentiated display methods and colors to distinguish different types of targets in order to optimize the content and effect of ARHUD display.

[0067] For details, please refer to Figure 4 As shown, pedestrians are represented by green human-shaped icons, cars by blue vehicle icons, and traffic lights are represented by icons of corresponding red, yellow, and green colors, with key information such as target distance and speed marked next to the icons. For potentially dangerous targets, such as pedestrians that are rapidly approaching vehicles or suddenly appearing, eye-catching warning methods such as flashing and color changing are used to attract the driver's attention.

[0068] In one embodiment, step S500 includes: acquiring vehicle acceleration and angular velocity motion attitude information in real time through vehicle inertial measurement unit (IMU) sensors, and adjusting the attitude of the ARHUD display information accordingly; thereby ensuring that the display information is stably aligned with the actual road scene during vehicle operation, avoiding shaking or misalignment.

[0069] In one embodiment, step S500 includes: measuring the time difference between data acquisition from the sensor and data display in the ARHUD, and using delay compensation algorithms such as linear interpolation to correct the timestamp of the displayed information; this embodiment addresses the delay caused by data transmission and processing, ensuring that the displayed information is synchronized with the actual road conditions, and providing the driver with timely and accurate driving assistance.

[0070] This invention acquires multispectral sensor data; employs a distribution-transform-based multisensor data fusion algorithm to fuse the multispectral sensor data, obtaining fused multispectral data; constructs a deep learning model based on a convolutional neural network to perform target recognition and classification on the fused multispectral data; uses a Kalman filter algorithm combined with deep learning feature matching for target tracking; and optimizes the ARHUD display content and effect by using differentiated display methods and color distinctions for different types of targets. It can fuse multispectral sensor data, leveraging the advantages of each sensor to solve the problem of insufficient perception capability of a single sensor in harsh environments. Through multispectral fusion and deep learning, it achieves more accurate target recognition and tracking, optimizes the ARHUD display content and effect, provides comprehensive and accurate environmental situational awareness, and assists drivers in safe driving.

[0071] This invention also provides an ARHUD-based sensing and display system, see reference. Figure 5 As shown, the system includes:

[0072] Acquisition module 11 is used to acquire collected multispectral sensor data;

[0073] The fusion module 12 is used to fuse multispectral sensor data using a multi-sensor data fusion algorithm based on distribution transformation to obtain fused multispectral data;

[0074] The recognition module 13 is used to construct a deep learning model based on a convolutional neural network to perform target recognition and classification on the fused multispectral data;

[0075] Tracking module 14 is used to track targets using a Kalman filter algorithm combined with deep learning feature matching;

[0076] Display module 15 is used to differentiate the display methods and colors of different types of targets in order to optimize the content and effect of ARHUD display.

[0077] This invention also provides an electronic device, see below. Figure 6 As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement any of the ARHUD-based perception and display methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0078] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0079] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0080] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0081] This invention also provides a computer-readable medium. The computer-readable medium stores a computer program, which, when executed by a processor, implements the steps of any of the ARHUD-based perception and display methods described in the above embodiments. The computer-readable storage medium may be volatile or non-volatile.

[0082] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0083] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0084] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0085] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0086] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0087] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0088] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0089] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0090] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0091] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A perception and display method based on ARHUD, characterized in that, Includes the following steps: S100: Acquire the collected multispectral sensor data; S200. A multi-sensor data fusion algorithm based on distribution transformation is used to fuse multispectral sensor data to obtain fused multispectral data; S300. Construct a deep learning model based on convolutional neural networks to perform target recognition and classification on the fused multispectral data; S400 uses the Kalman filter algorithm combined with deep learning feature matching for target tracking; S500 uses differentiated display methods and colors to distinguish different types of targets in order to optimize the content and effect of ARHUD display.

2. The method according to claim 1, characterized in that, In step S100, the multispectral sensor data includes visible light images, infrared thermal imaging images, and millimeter-wave radar data.

3. The method according to claim 2, characterized in that, Step S200 includes the following steps: S210. Convert the format of the multispectral sensor data, unifying the grayscale values ​​and measured values ​​into vector form; S220. Construct probability density function models for the multispectral sensor data respectively; S230. Using the distribution transformation theory, construct a joint probability density function model for multi-source data based on the probability density function model of multispectral sensor data; S240. Based on the parameters of the joint probability density function model, the maximum likelihood estimation method is used for calculation, and the target detection result after fusion is obtained by classification through the Bayesian criterion.

4. The method according to claim 3, characterized in that, Visible light images are modeled using Gaussian mixture models based on color and texture distribution; infrared thermal imaging images are modeled using exponential distribution models based on the statistical laws of thermal radiation intensity; and millimeter-wave radar data are modeled using normal distribution models based on the distribution characteristics of measured values.

5. The method according to claim 3, characterized in that, For categories with insufficient training samples or excessive errors, the Adaboost compensation algorithm is used for optimization.

6. The method according to claim 3, characterized in that, Step S400 includes the following steps: S410. In the initial frame, the target to be tracked is determined by target recognition and its multispectral features are extracted; the target motion state is initialized using a Kalman filter, and the position, velocity, and acceleration parameters are set. S420. In subsequent frames, the target position at the current moment is predicted based on the Kalman filter prediction formula and the target state at the previous moment. Then, a search area is defined in the current frame image with the predicted position as the center. The multispectral features of the target in this area are extracted by a deep learning model and matched with the target features in the initial frame. The similarity is calculated using methods such as Euclidean distance. The area with the highest similarity is selected as the actual position of the target in the current frame to complete the tracking.

7. The method according to claim 6, characterized in that, Step S500 includes: acquiring vehicle acceleration and angular velocity motion attitude information in real time through vehicle inertial measurement unit sensors, and adjusting the attitude of the ARHUD display information accordingly.

8. The method according to claim 7, characterized in that, Step S500 includes: measuring the time difference between data acquisition from the sensor and data display in the ARHUD, and using delay compensation algorithms such as linear interpolation to correct the timestamp of the displayed information.

9. A perception and display system based on ARHUD, characterized in that, include: The acquisition module is used to acquire the collected multispectral sensor data; The fusion module is used to fuse multispectral sensor data using a multi-sensor data fusion algorithm based on distribution transformation to obtain fused multispectral data; The recognition module is used to build a deep learning model based on convolutional neural networks to perform target recognition and classification on the fused multispectral data; The tracking module is used to track targets using a Kalman filter algorithm combined with deep learning feature matching. The display module is used to differentiate the display methods and colors for different types of targets in order to optimize the content and effect of ARHUD display.

10. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 8.

11. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 8.

Images